Hotspot

https://github.com/yoseflab/hotspot
Admin
A tool for identifying informative genes and gene modules in single-cell datasets based on cell...
Tokens:12,697
Snippets:144
Trust Score:9.7
License:MIT
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Hotspot

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/index.md

Install Hotspot directly from the git repository using pip.

```default
pip install hotspotsc
```

--------------------------------

### Install Hotspot from GitHub

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Installs the hotspot library from a specified GitHub branch, along with necessary dependencies like numba, scanpy, and mplscience. This is useful for installing directly from source or a specific development branch.

```python
import sys

#if branch is stable, will install via pypi, else will install from source
branch = "anndata"
IN_COLAB = "google.colab" in sys.modules

if IN_COLAB:
    !pip install --quiet --upgrade numba
    !pip install git+https://github.com/yoseflab/hotspot@$branch#egg=hotspot
    !pip install --quiet scanpy
    !pip install --quiet mplscience
```

--------------------------------

### Install Hotspot and Dependencies

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Installs the Hotspot package and other necessary libraries like scanpy, mplscience, and ete3. It checks if running in Google Colab for appropriate installation commands.

```python
import sys

#if branch is stable, will install via pypi, else will install from source
branch = "anndata"
IN_COLAB = "google.colab" in sys.modules

if IN_COLAB:
    !pip install --quiet --upgrade numba
    !pip install --quiet git+https://github.com/yoseflab/hotspot@$branch#egg=hotspot
    !pip install --quiet scanpy
    !pip install --quiet mplscience
    !pip install --quiet ete3
    !pip3 install --upgrade xlrd==1.2.0
```

--------------------------------

### Install Hotspot and Dependencies

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Installs the hotspot library from GitHub and other necessary packages like scanpy, muon, and mplscience. Use this in Colab environments.

```python
import sys

#if branch is stable, will install via pypi, else will install from source
branch = "master"
IN_COLAB = "google.colab" in sys.modules

if IN_COLAB:
    !pip install --quiet --upgrade jsonschema
    !pip install --quiet git+https://github.com/yoseflab/hotspot@$branch#egg=hotspot
    !pip install --quiet scanpy
    !pip install --quiet muon
    !pip install mplscience
```

--------------------------------

### Install Hotspot

Source: https://github.com/yoseflab/hotspot/blob/master/README.md

Install the latest version of Hotspot from PyPI. For older versions, specify the version number.

```bash
pip install hotspotsc
```

```bash
pip install hotspotsc==0.9.1
```

--------------------------------

### Import Hotspot

Source: https://github.com/yoseflab/hotspot/blob/master/README.md

Import the Hotspot library after installation.

```python
import hotspot
```

--------------------------------

### Get Top Genes for a Module

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Retrieves the top 10 genes for a specified module, sorted by Z-score. Requires joining results with modules and filtering by module ID.

```python
module = 1

results = hs.results.join(hs.modules)
results = results.loc[results.Module == module]
results.sort_values('Z', ascending=False).head(10)
```

--------------------------------

### Initialize Hotspot Object with Legacy Method

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Use this class method to initialize a Hotspot object for analysis using legacy input formats. Ensure that either latent space, distances, or a tree structure is provided.

```python
>>> gene_exp = pd.read_csv(path, index_col=0) # genes by cells
>>> latent = pd.read_csv(latent_path, index_col=0) # cells by dims
>>> hs = hotspot.Hotspot.legacy_init(gene_exp, model="normal", latent=latent)
```

--------------------------------

### Create Hotspot Object and KNN Graph

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Initializes the Hotspot object with AnnData, layer key, model, latent space key, and UMI counts key. Then, computes the k-nearest neighbors graph for spatial analysis.

```python
# Create the Hotspot object and the neighborhood graph
hs = hotspot.Hotspot(
    adata, 
    layer_key="csc_counts",
    model='bernoulli', 
    latent_obsm_key="spatial", 
    umi_counts_obs_key="total_counts",
)

hs.create_knn_graph(
    weighted_graph=False, n_neighbors=300,
)
```

--------------------------------

### Hotspot Initialization

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Initializes a Hotspot object for analysis. Either latent space, distances, or a tree structure is required.

```APIDOC
## class hotspot.Hotspot(adata, layer_key=None, model='danb', latent_obsm_key=None, distances_obsp_key=None, tree=None, umi_counts_obs_key=None, use_gpu=False)

### Description
Initializes a Hotspot object for analysis. Requires either latent space, distances, or a tree structure.

### Parameters
* **adata** (*anndata.AnnData*) – Count matrix (shape is cells by genes)
* **layer_key** (*str*) – Key in adata.layers with count data, uses adata.X if None.
* **model** (*string*, optional) – Specifies the null model to use for gene expression. Valid choices are: 'danb', 'bernoulli', 'normal', 'none'.
* **latent_obsm_key** (*string*, optional) – Latent space encoding cell-cell similarities with euclidean distances. Shape is (cells x dims). Input is key in adata.obsm
* **distances_obsp_key** (*pandas.DataFrame*, optional) – Distances encoding cell-cell similarities directly. Shape is (cells x cells). Input is key in adata.obsp
* **tree** (*ete3.coretype.tree.TreeNode*) – Root tree node. Can be created using ete3.Tree
* **umi_counts_obs_key** (*str*) – Total umi count per cell. Used as a size factor. If omitted, the sum over genes in the counts matrix is used.
* **use_gpu** (*bool*, optional) – Whether to use GPU acceleration via CuPy. Default is False.
```

```APIDOC
## classmethod hotspot.Hotspot.legacy_init(counts, model='danb', latent=None, distances=None, tree=None, umi_counts=None)

### Description
Initializes a Hotspot object for analysis using a legacy method. Requires either latent space, distances, or a tree structure.

### Parameters
* **counts** (*pandas.DataFrame*) – Count matrix (shape is genes x cells)
* **model** (*string*, optional) – Specifies the null model to use for gene expression. Valid choices are: 'danb', 'bernoulli', 'normal', 'none'.
* **latent** (*pandas.DataFrame*, optional) – Latent space encoding cell-cell similarities with euclidean distances. Shape is (cells x dims)
* **distances** (*pandas.DataFrame*, optional) – Distances encoding cell-cell similarities directly. Shape is (cells x cells)
* **tree** (*ete3.coretype.tree.TreeNode*) – Root tree node. Can be created using ete3.Tree
* **umi_counts** (*pandas.Series*, optional) – Total umi count per cell. Used as a size factor. If omitted, the sum over genes in the counts matrix is used.

### Request Example
```pycon
>>> import pandas as pd
>>> import hotspot
>>> gene_exp = pd.read_csv(path, index_col=0) # genes by cells
>>> latent = pd.read_csv(latent_path, index_col=0) # cells by dims
>>> hs = hotspot.Hotspot.legacy_init(gene_exp, model="normal", latent=latent)
```
```

--------------------------------

### Download Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Downloads the 10x Genomics dataset using wget. Ensure you are in a Colab environment for this command to work.

```shell
if IN_COLAB:
    !wget https://cf.10xgenomics.com/samples/cell-exp/3.1.0/5k_pbmc_protein_v3_nextgem/5k_pbmc_protein_v3_nextgem_filtered_feature_bc_matrix.h5
```

--------------------------------

### Import Libraries

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Imports necessary libraries for data analysis, including hotspot, scanpy, muon, numpy, and mplscience.

```python
import warnings; warnings.simplefilter('ignore')

import hotspot
import scanpy as sc
import muon as mu

import numpy as np
import mplscience
```

--------------------------------

### Create Hotspot Object and KNN Graph

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Initializes the Hotspot object with the AnnData object, specifying the layer key for counts, the background model, the latent space key for PCA, and the observation key for UMI counts. It then creates a k-nearest neighbors graph with binary weights and a specified number of neighbors.

```python
# Create the Hotspot object and the neighborhood graph
# hotspot works a lot faster with a csc matrix!
ada_cd4.layers["counts_csc"] = adata_cd4.layers["counts"].tocsc()
hs = hotspot.Hotspot(
    adata_cd4, 
    layer_key="counts_csc", 
    model='danb', 
    latent_obsm_key="X_pca",
    umi_counts_obs_key="total_counts"
)

hs.create_knn_graph(
    weighted_graph=False, n_neighbors=30,
)
```

--------------------------------

### Create Hotspot Object and Neighborhood Graph

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Initializes the Hotspot object with an AnnData object, a lineage tree, and a specified model. It then computes the k-nearest neighbors graph for neighborhood analysis. Use `weighted_graph=False` for binary weights when using a lineage tree.

```python
hs = hotspot.Hotspot(adata, model='normal', tree=tree)

hs.create_knn_graph(
    weighted_graph=False, n_neighbors=30,
)
```

--------------------------------

### Download Data Files

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Downloads necessary data files from NCBI GEO and Google Drive using wget and gdown. This includes count matrices, barcode and gene information, cell state kernels, and a reconstructed lineage tree.

```python
%%capture
if IN_COLAB:
    !wget https://ftp.ncbi.nlm.nih.gov/geo/samples/GSM3302nnn/GSM3302829/suppl/GSM3302829_embryo3_SeuratBarcodes.tsv.gz
    !wget https://ftp.ncbi.nlm.nih.gov/geo/samples/GSM3302nnn/GSM3302829/suppl/GSM3302829_embryo3_SeuratGenes.tsv.gz
    !wget https://ftp.ncbi.nlm.nih.gov/geo/samples/GSM3302nnn/GSM3302829/suppl/GSM3302829_embryo3_SeuratNorm.mtx.gz
    !wget https://ftp.ncbi.nlm.nih.gov/geo/series/GSE122nnn/GSE122187/suppl/GSE122187_CellStateKernels.xls.gz
```

```python
%%capture
import gdown
url = "https://drive.google.com/uc?id=1FlrryhNv9Khwo_BstUTPCBOVaMgZLZ9N"
output = 'tree.txt'
gdown.download(url, output, quiet=False)
```

--------------------------------

### Import Libraries

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Imports necessary libraries for spatial analysis, including scanpy, hotspot, numpy, and matplotlib.

```python
import scanpy as sc
import hotspot

import numpy as np
import mplscience
import matplotlib
```

--------------------------------

### create_knn_graph

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Creates the k-nearest neighbors (KNN) graph and associated weights for the hotspot analysis.

```APIDOC
## hotspot.Hotspot.create_knn_graph(weighted_graph=False, n_neighbors=30, neighborhood_factor=3, approx_neighbors=True)

### Description
Creates the k-nearest neighbors (KNN) graph and graph weights. The resulting matrices are stored in `self.neighbors` and `self.weights`.

### Parameters
* **weighted_graph** (*bool*) – Whether or not to create a weighted graph. Default is False.
* **n_neighbors** (*int*) – Neighborhood size. Default is 30.
* **neighborhood_factor** (*float*) – Used when creating a weighted graph. Sets how quickly weights decay relative to the distances within the neighborhood. The weight for a cell with a distance d will decay as exp(-d/D) where D is the distance to the n_neighbors/neighborhood_factor-th neighbor. Default is 3.
* **approx_neighbors** (*bool*) – Use approximate nearest neighbors or exact scikit-learn neighbors. Only applicable when hotspot is initialized with latent space. Default is True.
```

--------------------------------

### Create Hotspot Object

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/index.md

Initialize the Hotspot object with an AnnData object, specifying layer keys, background model, and metric space. The UMI counts per cell are used for scaling by default.

```default
import hotspot
hs = hotspot.Hotspot(
    adata,
    layer_key="counts",
    model='danb',
    latent_obsm_key="X_pca",
    umi_counts_obs_key="total_counts"
)
```

--------------------------------

### Load and Preprocess Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Loads spatial data from a .h5ad file using scanpy and calculates total counts. It also stores the sparse CSC counts in a layer.

```python
url = "https://github.com/YosefLab/scVI-data/blob/master/rodriques_slideseq.h5ad?raw=true"
adata = sc.read("rodriques_slideseq.h5ad", backup_url=url)
adata.obs["total_counts"] = np.asarray(adata.X.sum(1)).ravel()

adata.layers["csc_counts"] = adata.X.tocsc()
```

--------------------------------

### Load and Parse Lineage Tree

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Loads a reconstructed lineage tree from a text file using ete3 and extracts all leaf node names, which represent individual cells with recoverable lineage information.

```python
# Load the tree and enumerate the leaves
from ete3 import Tree
tree = Tree("tree.txt", format=1)
leaves = set()
for tn in tree.traverse('postorder'):
    if tn.is_leaf():
        leaves.add(tn.name)
len(leaves)
```

--------------------------------

### Prepare Data for PCA

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Prepares the AnnData object for PCA by copying raw counts, normalizing, log-transforming, and scaling the data. Layers are used to store intermediate states.

```python
adata_cd4.layers["counts"] = adata_cd4.X.copy()
sc.pp.normalize_total(adata_cd4)
sc.pp.log1p(adata_cd4)
adata_cd4.layers["log_normalized"] = adata_cd4.X.copy()
sc.pp.scale(adata_cd4)
```

--------------------------------

### Violin Plot for QC Metrics

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Generates violin plots for QC metrics (number of genes, total counts, mitochondrial counts percentage) using scanpy and mplscience.

```python
with mplscience.style_context():
    sc.pl.violin(adata, ['n_genes_by_counts', 'total_counts', 'pct_counts_mt'],
                jitter=0.4, multi_panel=True, log=True)
```

--------------------------------

### Calculate QC Metrics

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Calculates quality control metrics for the AnnData object, including mitochondrial gene percentage.

```python
adata.var['mt'] = adata.var_names.str.startswith('MT-')  # annotate the group of mitochondrial genes as 'mt'
sc.pp.calculate_qc_metrics(adata, qc_vars=['mt'], percent_top=None, log1p=False, inplace=True)
```

--------------------------------

### Compute UMAP and Neighbors

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Prepares the AnnData object for visualization by computing nearest neighbors and the UMAP embedding.

```python
sc.pp.neighbors(adata_cd4)
sc.tl.umap(adata_cd4)
sc.pl.umap(adata_cd4, frameon=False)
```

--------------------------------

### Compute UMAP for Visualization

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Prepares the AnnData object for UMAP visualization by scaling data, performing PCA, and computing neighbors. This is a standard preprocessing pipeline for dimensionality reduction.

```python
adata.raw = adata
```

```python
sc.pp.scale(adata)
sc.tl.pca(adata)
sc.pp.neighbors(adata)
sc.tl.umap(adata)
```

--------------------------------

### Create Gene Modules with Agglomerative Clustering

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/index.md

Use `create_modules` to group genes into modules via agglomerative clustering. This method takes thresholds for minimum gene count, core genes, and FDR to control the merging process. Genes not assigned to any module are marked with -1.

```default
modules = hs.create_modules(
    min_gene_threshold=30, core_only=True, fdr_threshold=0.05
)
```

--------------------------------

### Normalize and Log1p Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Prepares the AnnData object for spatial analysis by normalizing total counts and applying log1p transformation.

```python
sc.pp.normalize_total(adata)
sc.pp.log1p(adata)
```

--------------------------------

### Create Gene Modules

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Groups genes into modules using agglomerative clustering. Customize merging behavior with `min_gene_threshold`, `core_only`, and `fdr_threshold`.

```python
modules = hs.create_modules(
    min_gene_threshold=15, core_only=True, fdr_threshold=0.05
)
```

--------------------------------

### Assign Cells to Closest Profiles using Kernels

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Assigns each cell to its closest profile using cosine distance metric between cell expression and kernel profiles. Requires `scipy.spatial.distance.cdist` and `pandas`.

```python
from scipy.spatial.distance import cdist

common = adata.var_names & kernels.index
dd = cdist(adata.raw[:, common].X.toarray(), kernels.loc[common].T, metric='cosine')
dd = pd.DataFrame(dd, index=adata.obs_names, columns=kernels.columns)
assignment = dd.idxmin(axis=1).loc[adata.obs_names]
```

--------------------------------

### Create Spatial Modules

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Groups genes into spatial modules using agglomerative clustering based on local correlations. Parameters control merging thresholds and assignment of ambiguous genes.

```python
modules = hs.create_modules(
    min_gene_threshold=20, core_only=False, fdr_threshold=0.05
)

modules.value_counts()
```

--------------------------------

### Load and Process Count Matrix and Metadata

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Loads the raw count matrix, barcodes, and gene information. It then filters the barcodes to include only those present in the lineage tree and prepares the data for AnnData object creation.

```python
counts_raw = mmread("GSM3302829_embryo3_SeuratNorm.mtx.gz")
counts_raw = csr_matrix(counts_raw)
barcodes = pd.read_table("GSM3302829_embryo3_SeuratBarcodes.tsv.gz", header=None)[0]
barcodes = [x+'-1' for x in barcodes] # to match the newick file
genes = pd.read_table("GSM3302829_embryo3_SeuratGenes.tsv.gz", header=None)[0]
```

--------------------------------

### Violin Plot for Protein Markers

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Generates violin plots for key protein markers (CD14, CD4, CD3) to visualize their abundance.

```python
with mplscience.style_context():
    sc.pl.violin(prot_data, keys=["CD14_TotalSeqB", "CD4_TotalSeqB", "CD3_TotalSeqB"], multi_panel=True)
```

--------------------------------

### Load and Process Kernel Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Loads gene expression data from a gzipped Excel file, sets the 'Gene' column as the index, sorts columns, and creates a mapping for kernel names. This is useful for preparing gene expression matrices for further analysis.

```python
import gzip
with gzip.open("GSE122187_CellStateKernels.xls.gz") as fin:
    kernels = pd.read_excel(fin, skiprows=1).set_index('Gene')
    
kernels = kernels.loc[
        :, kernels.columns.sort_values()
    ]

kernel_name_map = {
    0: "trophoblast stem cells",
    1: "neural ectoderm anterior",
    2: "primitive streak late",
    3: "anterior primitive streak",
    4: "primitive/definitive endoderm",
    5: "allantois",
    6: "secondary heart field/splanchnic lateral plate",
    7: "gut endoderm",
    8: "ectoderm early 1",
    9: "primitive blood early",
    10: "preplacodal ectoderm",
    11: "neural ectoderm posterior",
    12: "posterior lateral plate mesoderm",
    13: "hematopoietic/endothelial progenitors",
    14: "parietal endoderm",
    15: "amnion mesoderm early",
    16: "surface ectoderm",
    17: "epiblast",
    18: "somites",
    19: "ectoderm early 2",
    20: "splanchnic lateral plate/anterior paraxial mesoderm",
    21: "primitive heart tube",
    22: "primitive blood late",
    23: "notochord",
    24: "fore/midbrain",
    25: "distal extraembryonic ectoderm",
    26: "neuromesodermal progenitor early",
    27: "primordial germ cells",
    28: "differentiated trophoblasts",
    29: "visceral endoderm early",
    30: "presomitic mesoderm",
    31: "neuromesodermal progenitor late",
    32: "angioblasts",
    33: "neural crest",
    34: "pharyngeal arch mesoderm",
    35: "similar to neural crest",
    36: "primitive blood progenitors",
    37: "primitive streak early",
    38: "node",
    39: "future spinal cord",
    40: "visceral endoderm late",
    41: "amnion mesoderm late",
}

kernel_name_map = {k: v.capitalize() for k, v in kernel_name_map.items()}

kernels.columns = [kernel_name_map[x] for x in kernels.columns]
```

--------------------------------

### Import Libraries for Data Analysis

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Imports essential Python libraries for data manipulation, analysis, and visualization, including numpy, pandas, hotspot, matplotlib, seaborn, and scanpy.

```python
import warnings; warnings.simplefilter('ignore')

import numpy as np
import pandas as pd
import hotspot
import matplotlib.pyplot as plt
import matplotlib.colors
import seaborn as sns
import mplscience

from scipy.io import mmread
from scipy.sparse import csr_matrix

import scanpy as sc
```

--------------------------------

### create_modules

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Groups genes into modules based on correlation thresholds and gene counts. The results are stored in `self.modules` and the linkage matrix in `self.linkage`.

```APIDOC
## create_modules(min_gene_threshold=20, core_only=True, fdr_threshold=0.05)

### Description
Groups genes into modules. The results are stored in `self.modules` and the linkage matrix in `self.linkage`.

### Parameters
#### Parameters
- **min_gene_threshold** (*int*) - Optional - Controls how small modules can be. Increase if there are too many modules being formed. Decrease if substructure is not being captured.
- **core_only** (*bool*) - Optional - Whether or not to assign ambiguous genes to a module or leave unassigned. Default is True.
- **fdr_threshold** (*float*) - Optional - Correlation threshold at which to stop assigning genes to modules. Default is 0.05.

### Returns
- **modules** (*pandas.Series*) - Maps gene to module number. Unassigned genes are indicated with -1.
```

--------------------------------

### Compute PCA

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Computes the Principal Component Analysis (PCA) on the prepared data.

```python
sc.tl.pca(adata_cd4)
```

--------------------------------

### Create Gene Modules

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Groups genes into modules using agglomerative clustering based on local correlations. It allows control over merging branches using FDR-adjusted p-values and minimum gene thresholds. Genes not assigned to a module are marked with -1.

```python
modules = hs.create_modules(
    min_gene_threshold=50, core_only=True, fdr_threshold=0.05
)

modules.value_counts()
```

--------------------------------

### Visualize Module Genes Spatially

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Plots the spatial expression of the top genes from a module. Uses a custom colormap and requires the 'adata' object and 'mplscience' for styling.

```python
cmap = matplotlib.colors.LinearSegmentedColormap.from_list(
    'grays', ['#DDDDDD', '#000000'])

module = 2
results = hs.results.join(hs.modules)
results = results.loc[results.Module == module]
genes = results.sort_values('Z', ascending=False).head(6).index

with mplscience.style_context():
    sc.pl.spatial(
        adata, 
        color=genes, 
        cmap=cmap, 
        frameon=False,
        vmin='p0',
        vmax='p95',
        spot_size=30,
    )
```

--------------------------------

### Apply CLR Transformation to Protein Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Applies the Centered Log-Ratio (CLR) transformation to the protein data. This is a common preprocessing step for compositional data.

```python
from muon import prot as pt
pt.pp.clr(mdata['prot'])

prot_data = mdata.mod["prot"]
```

--------------------------------

### Plot Module Scores on UMAP

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Visualizes module scores on the UMAP plot using a specified color scale. Requires `mplscience` for styling.

```python
with mplscience.style_context():
    sc.pl.umap(adata_cd4, color=module_cols, frameon=False, vmin=-1, vmax=1)
```

--------------------------------

### Plot Local Correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Visualizes the computed local correlations. Adjust `vmin` and `vmax` to set the color scale limits.

```python
hs.plot_local_correlations(vmin=-12, vmax=12)
```

--------------------------------

### Plot Gene Expression Heatmaps by Module

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Generates heatmaps of standardized gene expression values for genes within each module. It uses logarithmic scaling for module sizes to adjust subplot heights and displays a color bar only for the last plot. This visualization helps in identifying expression patterns specific to different cell types or states.

```python
# Plot the kernel values for the genes in each module
mods = hs.modules.unique()
mods = [x for x in mods if x != -1 and not np.isnan(x)]
mods = sorted(mods)

sizes = [(hs.modules == x).sum() for x in mods]
sizes = [np.log10(x) for x in sizes]

fig, axs = plt.subplots(
    len(mods), 1, gridspec_kw=dict(height_ratios=sizes, right=.8), 
    figsize=(9, 12.5)
)

cbax = fig.add_axes([.85, .15, .015, .1])

for ax, mod in zip(axs.ravel(), mods):

    plt.sca(ax)
    mod_genes = hs.modules.index[hs.modules == mod]
    mod_genes = pd.Index(mod_genes) & kernels.index
    
    last_plot = mod == max(mods)
    
    if last_plot:
        cbar=True
        cbar_ax=cbax
    else:
        cbar=False
        cbar_ax=None

    sns.heatmap(kernels_z.loc[mod_genes], vmin=-2, vmax=2,
                cmap="RdBu_r", ax=ax, xticklabels=True, rasterized=True,
                cbar=cbar, cbar_ax=cbar_ax)
    plt.ylabel(int(mod))

    plt.yticks([])

    if last_plot:
        ax.tick_params(labelsize=9)
    else:
        plt.xticks([])

plt.tight_layout()
```

--------------------------------

### PCA Variance Ratio Plot

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Generates a plot showing the variance ratio explained by each principal component.

```python
with mplscience.style_context():
    sc.pl.pca_variance_ratio(adata_cd4)
```

--------------------------------

### Calculate Aggregate Module Scores

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Computes aggregate scores for all modules. This function processes all modules and displays progress.

```python
module_scores = hs.calculate_module_scores()

module_scores.head()
```

--------------------------------

### Find Top Genes in a Module

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Retrieves and sorts genes within a specific module by their Z-score to identify the most informative genes.

```python
module = 9

results = hs.results.join(hs.modules)
results = results.loc[results.Module == module]

results.sort_values('Z', ascending=False).head(10)
```

--------------------------------

### Plot Local Correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Generates a heatmap visualization of the local correlation matrix, aiding in the exploration of spatial gene modules.

```python
hs.plot_local_correlations()
```

--------------------------------

### Create AnnData Object and Filter Genes

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Creates a Scanpy AnnData object from the processed count matrix and metadata. It then filters genes that are not detected in at least 10 cells.

```python
adata = sc.AnnData(counts_raw.transpose())
adata.obs_names = barcodes
adata.var_names = genes

adata = adata[valid_barcodes]
sc.pp.filter_genes(adata, min_cells=10)
```

--------------------------------

### Compute PCA with Reduced Components

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Recomputes PCA using a specified number of components (10 in this case).

```python
# rerun with fewer components
sc.tl.pca(adata_cd4, n_comps=10)
```

--------------------------------

### Visualize Module Scores Spatially

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Plots the spatial distribution of the calculated module scores across the dataset. Uses 'adata' and 'mplscience' for styling.

```python
with mplscience.style_context():
    sc.pl.spatial(adata, color=module_cols, frameon=False, vmin="p0", vmax="p99", spot_size=30)
```

--------------------------------

### Show Top Genes in a Module

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Identifies and displays the top 10 genes with the highest autocorrelation within a specified module. This helps in understanding the most informative genes contributing to a module's identity.

```python
# Show the top genes for a module

module = 3

results = hs.results.join(hs.modules)
results = results.loc[results.Module == module]
results.sort_values('Z', ascending=False).head(10)
```

--------------------------------

### Compute Local Correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Computes pair-wise local correlations between genes. This is a prerequisite for module creation.

```python
lcz = hs.compute_local_correlations(hs_genes, jobs=2)
```

--------------------------------

### Read 10x H5 Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Reads data from a 10x Genomics H5 file into a MuData object. Handles potential non-unique variable names.

```python
mdata = mu.read_10x_h5("/content/5k_pbmc_protein_v3_nextgem_filtered_feature_bc_matrix.h5")
```

--------------------------------

### Compute KNN Graph

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/index.md

Computes the K-nearest-neighbors graph for cells. Use `weighted_graph=True` for weighted graphs based on cell-cell distances, and adjust `n_neighbors` to control neighborhood size.

```python
hs.create_knn_graph(weighted_graph=False, n_neighbors=30)
```

--------------------------------

### Create CD4 T Cell Mask and Filter Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Creates a boolean mask to identify CD4 T cells based on protein expression and filters the AnnData object accordingly. It also filters cells with low gene counts, low total counts, and high mitochondrial gene percentage.

```python
# now create a CD4 T cell mask
is_cd4 = np.asarray(
    (prot_data[:, 'CD14_TotalSeqB'].X.toarray() < 2) &
    (prot_data[:, 'CD4_TotalSeqB'].X.toarray() > 1) &
    (prot_data[:, 'CD3_TotalSeqB'].X.toarray() > 1)
).ravel()

adata_cd4 = adata[is_cd4]
sc.pp.filter_cells(adata_cd4, min_genes=1000)
sc.pp.filter_genes(adata_cd4, min_cells=10)
adata_cd4 = adata_cd4[adata_cd4.obs.pct_counts_mt < 16].copy()
```

--------------------------------

### Select Informative Genes

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Selects genes with a False Discovery Rate (FDR) below 0.05 from the autocorrelation results, sorts them by Z-score in descending order, and takes the top 500 genes.

```python
# Select the genes with significant lineage autocorrelation
hs_genes = hs_results.loc[hs_results.FDR < 0.05].sort_values('Z', ascending=False).head(500).index
```

--------------------------------

### Select Genes with Significant Lineage Autocorrelation

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Filters genes based on a False Discovery Rate (FDR) threshold to select those with statistically significant lineage autocorrelation. This is a precursor to grouping genes into modules.

```python
# Select the genes with significant lineage autocorrelation
hs_genes = hs_results.index[hs_results.FDR < 0.05]

```

--------------------------------

### Extract RNA AnnData

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Extracts the RNA modality from the MuData object into a standard AnnData object.

```python
adata = mdata.mod["rna"]
```

--------------------------------

### Plot Module Scores on UMAP

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Plots the computed module scores on top of the UMAP visualization for comparison. Uses `mplscience` for styling and sets `vmin` and `vmax` for consistent color scaling.

```python
module_cols = []
for c in module_scores.columns:
    key = f"Module {c}"
    adata.obs[key] = module_scores[c]
    module_cols.append(key)

with mplscience.style_context():
    sc.pl.umap(adata, color=module_cols, frameon=False, vmin=-1, vmax=1)
```

--------------------------------

### calculate_module_scores

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Calculates scores for each module for each gene. The output is stored in `self.module_scores`.

```APIDOC
## calculate_module_scores()

### Description
Calculate Module Scores. The output is stored in `self.module_scores`.

### Returns
- **module_scores** (*pandas.DataFrame*) - Scores for each module for each gene. Dimensions are genes x modules.
```

--------------------------------

### Calculate Summary Per-Cell Module Scores

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/index.md

Compute summary scores for each module across all cells. This is useful for visualizing the overall expression pattern of genes within a module. The output is a pandas DataFrame mapping cells to module scores.

```default
module_scores = hs.calculate_module_scores()
```

--------------------------------

### Compute Local Gene Correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/index.md

Evaluates pair-wise local correlations between genes, taking into account spatial proximity. This is computationally intensive; it's recommended to run on a subset of informative genes identified by autocorrelation. The `jobs` argument enables parallelization.

```python
hs_genes = hs_results.loc[hs_results.FDR < 0.05].index # Select genes

local_correlations = hs.compute_local_correlations(hs_genes, jobs=4) # jobs for parallelization
```

--------------------------------

### Add Module Scores to AnnData

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Assigns calculated module scores as observation annotations in the AnnData object. Creates new columns for each module score.

```python
module_cols = []
for c in module_scores.columns:
    key = f"Module {c}"
    adata.obs[key] = module_scores[c]
    module_cols.append(key)
```

--------------------------------

### plot_local_correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Plots a clustergrid of the local correlation values, with options for customizing colormaps and axis labels.

```APIDOC
## plot_local_correlations(mod_cmap='tab10', vmin=-8, vmax=8, z_cmap='RdBu_r', yticklabels=False)

### Description
Plots a clustergrid of the local correlation values.

### Parameters
#### Parameters
- **mod_cmap** (*valid matplotlib colormap str or object*) - Optional - Discrete colormap for module assignments on the left side. Default is 'tab10'.
- **vmin** (*float*) - Optional - Minimum value for colorscale for Z-scores. Default is -8.
- **vmax** (*float*) - Optional - Maximum value for colorscale for Z-scores. Default is 8.
- **z_cmap** (*valid matplotlib colormap str or object*) - Optional - Continuous colormap for correlation Z-scores. Default is 'RdBu_r'.
- **yticklabels** (*bool*) - Optional - Whether or not to plot all gene labels. Default is False.
```

--------------------------------

### Compute Autocorrelations for Heritable Variation

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Calculates gene autocorrelations using the lineage metric to identify genes with the most heritable variation. The `jobs` parameter can be used to parallelize the computation.

```python
hs_results = hs.compute_autocorrelations(jobs=1)

hs_results.head(15)
```

--------------------------------

### Count Module Occurrences

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Counts the number of genes assigned to each module, including unassigned genes (-1).

```python
modules.value_counts()
```

--------------------------------

### Filter Cells Based on Lineage Recoverability

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Filters the list of barcodes to keep only those that are present in the set of leaves extracted from the lineage tree. This ensures that only cells with valid lineage information are used for downstream analysis.

```python
# Subset the count matrix to only the cells where the lineage was recoverable
is_valid = [x in leaves for x in barcodes]
is_valid_indices = np.nonzero(is_valid)[0]
valid_barcodes = [barcodes[i] for i in is_valid_indices]
```

--------------------------------

### Compute Gene Autocorrelations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/index.md

Calculates per-gene autocorrelation statistics, including coefficients, Z-scores, p-values, and FDR. The `jobs` argument can be used for parallelization on multi-core machines.

```python
hs_results = hs.compute_autocorrelations()
```

--------------------------------

### compute_local_correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Defines gene-gene relationships with pair-wise local correlations. The results are stored in `self.local_correlation_z`.

```APIDOC
## hotspot.Hotspot.compute_local_correlations(genes, jobs=1)

### Description
Defines gene-gene relationships with pair-wise local correlations. The results are stored in `self.local_correlation_z`.

### Parameters
* **genes** (*iterable* *of* *str*) – Gene identifiers to compute local correlations on. Should be a smaller subset of all genes.
* **jobs** (*int*) – Number of parallel jobs to run. Default is 1.

### Returns
* **local_correlation_z** (*pd.Dataframe*) – Local correlation Z-scores between genes. Shape is genes x genes.
```

--------------------------------

### Compute Local Correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Computes pair-wise local correlations between genes. Specify the number of jobs for parallel processing.

```python
lcz = hs.compute_local_correlations(hs_genes, jobs=4)
```

--------------------------------

### compute_autocorrelations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/hotspot.md

Performs feature selection using local autocorrelation. The results are stored in `self.results`.

```APIDOC
## hotspot.Hotspot.compute_autocorrelations(jobs=1)

### Description
Performs feature selection using local autocorrelation. The output is stored in `self.results`.

### Parameters
* **jobs** (*int*) – Number of parallel jobs to run. Default is 1.

### Returns
* **results** (*pandas.DataFrame*) – A dataframe with four columns: 'C' (Scaled -1:1 autocorrelation coeficients), 'Z' (Z-score for autocorrelation), 'Pval' (P-values computed from Z-scores), and 'FDR' (Q-values using the Benjamini-Hochberg procedure). Gene ids are in the index.
```

--------------------------------

### Standardize Gene Expression Data

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Standardizes the kernel values for each gene by subtracting the mean and dividing by the standard deviation across all cell states. This z-scoring is essential for comparing gene expression levels across different conditions.

```python
# Standardize each gene
kernels_z = kernels \
    .subtract(kernels.mean(axis=1), axis=0) \
    .divide(kernels.std(axis=1), axis=0)
```

--------------------------------

### Add Assignment to AnnData and Plot UMAP

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Lineage_Tutorial.ipynb

Adds the computed cell assignments as a categorical observation to the AnnData object and plots the UMAP colored by these assignments. Ensures 'assignment' is stored as categorical.

```python
adata.obs["assignment"] = assignment
fig, ax = plt.subplots(figsize=(6, 6))
sc.pl.umap(adata, color="assignment", frameon=False, ax=ax)
```

--------------------------------

### Compute Local Correlations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Computes pairwise local correlations between genes that have significant spatial autocorrelation. This is used for grouping genes into spatial modules.

```python
# Select the genes with significant spatial autocorrelation
hs_genes = hs_results.index[hs_results.FDR < 0.05]

# Compute pair-wise local correlations between these genes
lcz = hs.compute_local_correlations(hs_genes, jobs=4)
```

--------------------------------

### Compute Gene Autocorrelations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Computes the autocorrelations for each gene using the specified number of jobs. The results, including Z-scores and p-values, are stored in a pandas DataFrame.

```python
hs_results = hs.compute_autocorrelations(jobs=4)

hs_results.head(15)
```

--------------------------------

### Make Variable Names Unique

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Ensures that all variable names (genes) in the MuData object are unique.

```python
mdata.var_names_make_unique()
```

--------------------------------

### Assign Module Scores to AnnData

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/CD4_Tutorial.ipynb

Assigns calculated module scores as observation annotations in the AnnData object for plotting.

```python
module_cols = []
for c in module_scores.columns:
    key = f"Module {c}"
    adata_cd4.obs[key] = module_scores[c]
    module_cols.append(key)
```

--------------------------------

### Compute Autocorrelations

Source: https://github.com/yoseflab/hotspot/blob/master/docs/source/Spatial_Tutorial.ipynb

Calculates spatial autocorrelations for each gene to identify genes with significant spatial variation. The results are stored in a pandas DataFrame.

```python
hs_results = hs.compute_autocorrelations(jobs=4)

hs_results.head()
```
Tokens:
Raw