Hotspot

https://hotspot.readthedocs.io/en/latest/
Admin
Hotspot is a tool for identifying informative genes and gene modules in single-cell datasets using...
Tokens:21,270
Snippets:192
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Hotspot

Source: https://hotspot.readthedocs.io/en/latest/index.html

Install Hotspot directly from the git repository using pip.

```bash
pip install hotspotsc

```

--------------------------------

### Install and Import Libraries

Source: https://hotspot.readthedocs.io/en/latest/Spatial_Tutorial.html

Installs and imports necessary libraries for spatial analysis, including scanpy and hotspot.

```python
import scanpy as sc
import hotspot

import numpy as np
import mplscience
import matplotlib
```

--------------------------------

### Hotspot Installation Output

Source: https://hotspot.readthedocs.io/en/latest/Spatial_Tutorial.html

This output shows the process of installing the hotspot package from source, including dependency resolution and wheel building. It confirms the successful creation of the hotspot wheel.

```text
     |████████████████████████████████| 3.3 MB 8.3 MB/s 
     |████████████████████████████████| 34.5 MB 1.4 MB/s 
?25hCollecting hotspot
  Cloning https://github.com/yoseflab/hotspot (to revision anndata) to /tmp/pip-install-6tu2qe3n/hotspot_3d0fc0f8c2b94342ad19e622464099fe
  Running command git clone -q https://github.com/yoseflab/hotspot /tmp/pip-install-6tu2qe3n/hotspot_3d0fc0f8c2b94342ad19e622464099fe
  Running command git checkout -b anndata --track origin/anndata
  Switched to a new branch 'anndata'
  Branch 'anndata' set up to track remote branch 'anndata' from 'origin'.
Requirement already satisfied: matplotlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from hotspot) (3.2.2)
Requirement already satisfied: numba>=0.43.1 in /usr/local/lib/python3.7/dist-packages (from hotspot) (0.55.1)
Requirement already satisfied: numpy>=1.16.4 in /usr/local/lib/python3.7/dist-packages (from hotspot) (1.21.5)
Requirement already satisfied: seaborn>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from hotspot) (0.11.2)
Requirement already satisfied: scipy>=1.2.1 in /usr/local/lib/python3.7/dist-packages (from hotspot) (1.4.1)
Requirement already satisfied: pandas>=0.24.0 in /usr/local/lib/python3.7/dist-packages (from hotspot) (1.3.5)
Requirement already satisfied: tqdm>=4.32.2 in /usr/local/lib/python3.7/dist-packages (from hotspot) (4.62.3)
Requirement already satisfied: statsmodels>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from hotspot) (0.10.2)
Requirement already satisfied: scikit-learn>=0.21.2 in /usr/local/lib/python3.7/dist-packages (from hotspot) (1.0.2)
Collecting anndata>=0.7
  Downloading anndata-0.7.8-py3-none-any.whl (91 kB)
     |████████████████████████████████| 91 kB 4.3 MB/s 
?25hRequirement already satisfied: importlib_metadata>=0.7 in /usr/local/lib/python3.7/dist-packages (from anndata>=0.7->hotspot) (4.11.0)
Requirement already satisfied: xlrd<2.0 in /usr/local/lib/python3.7/dist-packages (from anndata>=0.7->hotspot) (1.1.0)
Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from anndata>=0.7->hotspot) (3.1.0)
Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.7/dist-packages (from anndata>=0.7->hotspot) (21.3)
Requirement already satisfied: natsort in /usr/local/lib/python3.7/dist-packages (from anndata>=0.7->hotspot) (5.5.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib_metadata>=0.7->anndata>=0.7->hotspot) (3.7.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib_metadata>=0.7->anndata>=0.7->hotspot) (3.10.0.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->hotspot) (3.0.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->hotspot) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->hotspot) (1.3.2)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->hotspot) (2.8.2)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.43.1->hotspot) (57.4.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in /usr/local/lib/python3.7/dist-packages (from numba>=0.43.1->hotspot) (0.38.0)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.0->hotspot) (2018.9)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib>=3.0.0->hotspot) (1.15.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.2->hotspot) (3.1.0)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.2->hotspot) (1.1.0)
Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from statsmodels>=0.9.0->hotspot) (0.5.2)
Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py->anndata>=0.7->hotspot) (1.5.2)
Building wheels for collected packages: hotspot
  Building wheel for hotspot (setup.py) ... ?25l?25hdone
  Created wheel for hotspot: filename=hotspot-0.9.1-py3-none-any.whl size=25699 sha256=32673675d4f8d6d438cf928d204c37d679c313816ab5132f7cd71ee8958ad92f

```

--------------------------------

### pip Installation Output

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

This output shows the progress and results of the pip installation process, including dependency resolution and wheel building. It also highlights potential dependency conflicts.

```text
?25l
     |████▌                           | 10 kB 15.7 MB/s eta 0:00:01
     |█████████                       | 20 kB 19.3 MB/s eta 0:00:01
     |█████████████▌                  | 30 kB 12.1 MB/s eta 0:00:01
     |██████████████████              | 40 kB 11.0 MB/s eta 0:00:01
     |██████████████████████▌         | 51 kB 7.5 MB/s eta 0:00:01
     |█████████████████████████     | 61 kB 8.7 MB/s eta 0:00:01
     |███████████████████████████████▋| 71 kB 8.3 MB/s eta 0:00:01
     |████████████████████████████████| 72 kB 843 kB/s 
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nbclient 0.5.10 requires jupyter-client>=6.1.5, but you have jupyter-client 5.3.5 which is incompatible.
     |████████████████████████████████| 91 kB 4.3 MB/s 
?25h  Building wheel for hotspot (setup.py) ... ?25l?25hdone
     |████████████████████████████████| 2.0 MB 8.9 MB/s 
     |████████████████████████████████| 86 kB 4.1 MB/s 
     |████████████████████████████████| 1.1 MB 29.4 MB/s 
     |████████████████████████████████| 63 kB 2.0 MB/s 
?25h  Building wheel for umap-learn (setup.py) ... ?25l?25hdone
  Building wheel for pynndescent (setup.py) ... ?25l?25hdone
  Building wheel for sinfo (setup.py) ... ?25l?25hdone
     |████████████████████████████████| 287 kB 7.6 MB/s 
     |████████████████████████████████| 41 kB 115 kB/s 
     |████████████████████████████████| 48 kB 4.3 MB/s 
?25h  Building wheel for loompy (setup.py) ... ?25l?25hdone
  Building wheel for numpy-groupies (setup.py) ... ?25l?25hdone
Collecting mplscience
  Downloading mplscience-0.0.5-py3-none-any.whl (4.8 kB)
Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from mplscience) (0.11.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from mplscience) (3.2.2)
Collecting importlib-metadata<2.0,>=1.0
  Downloading importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata<2.0,>=1.0->mplscience) (3.7.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mplscience) (2.8.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mplscience) (3.0.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mplscience) (1.3.2)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mplscience) (0.11.0)
Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mplscience) (1.21.5)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib->mplscience) (1.15.0)
Requirement already satisfied: pandas>=0.23 in /usr/local/lib/python3.7/dist-packages (from seaborn->mplscience) (1.3.5)
Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from seaborn->mplscience) (1.4.1)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.23->seaborn->mplscience) (2018.9)
Installing collected packages: importlib-metadata, mplscience
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 4.11.0
    Uninstalling importlib-metadata-4.11.0:
      Successfully uninstalled importlib-metadata-4.11.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
markdown 3.3.6 requires importlib-metadata>=4.4; python_version < "3.10", but you have importlib-metadata 1.7.0 which is incompatible.
Successfully installed importlib-metadata-1.7.0 mplscience-0.0.5

```

--------------------------------

### Install Hotspot and Dependencies in Colab

Source: https://hotspot.readthedocs.io/en/latest/Spatial_Tutorial.html

Installs numba, hotspot from a specific branch, scanpy, and mplscience. Use this for Google Colab environments.

```python
import sys

#if branch is stable, will install via pypi, else will install from source
branch = "anndata"
IN_COLAB = "google.colab" in sys.modules

if IN_COLAB:
    !pip install --quiet --upgrade numba
    !pip install git+https://github.com/yoseflab/hotspot@$branch#egg=hotspot
    !pip install --quiet scanpy
    !pip install --quiet mplscience

```

--------------------------------

### Install Dependencies for Hotspot

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Installs necessary libraries including numba, hotspot, scanpy, mplscience, ete3, and xlrd. Use this in Google Colab environments.

```python
import sys

#if branch is stable, will install via pypi, else will install from source
branch = "anndata"
IN_COLAB = "google.colab" in sys.modules

if IN_COLAB:
    !pip install --quiet --upgrade numba
    !pip install --quiet git+https://github.com/yoseflab/hotspot@$branch#egg=hotspot
    !pip install --quiet scanpy
    !pip install --quiet mplscience
    !pip install --quiet ete3
    !pip3 install --upgrade xlrd==1.2.0

```

--------------------------------

### Install Hotspot and Dependencies

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Installs the Hotspot library from GitHub and other necessary packages like jsonschema, scanpy, muon, and mplscience. This is typically run in a Google Colab environment.

```python
import sys

#if branch is stable, will install via pypi, else will install from source
branch = "master"
IN_COLAB = "google.colab" in sys.modules

if IN_COLAB:
    !pip install --quiet --upgrade jsonschema
    !pip install --quiet git+https://github.com/yoseflab/hotspot@$branch#egg=hotspot
    !pip install --quiet scanpy
    !pip install --quiet muon
    !pip install mplscience

```

--------------------------------

### Get Top Genes for a Module

Source: https://hotspot.readthedocs.io/en/latest/Spatial_Tutorial.html

Retrieves and sorts the top 10 genes for a given module based on their 'Z' score. Requires 'hs' (Hotspot results) and 'hs.modules' to be loaded.

```python
module = 1

results = hs.results.join(hs.modules)
results = results.loc[results.Module == module]
results.sort_values('Z', ascending=False).head(10)
```

--------------------------------

### Hotspot.legacy_init()

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Initializes the Hotspot object using legacy settings.

```APIDOC
## Hotspot.legacy_init()

### Description
Initializes the Hotspot object with legacy parameters.

### Method
N/A (This is a constructor or initialization method)

### Parameters
(No parameters explicitly documented in the source)

### Request Example
(No request example provided)

### Response
(No response details provided)
```

--------------------------------

### Hotspot.legacy_init

Source: https://hotspot.readthedocs.io/en/latest/_modules/hotspot/hotspot.html

Initializes a Hotspot object for analysis using a legacy method. Requires either `latent`, `tree`, or `distances` to be provided.

```APIDOC
## Hotspot.legacy_init

### Description
Initializes a Hotspot object for analysis using legacy method. Either `latent` or `tree` or `distances` is required.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **counts** (pandas.DataFrame) - Count matrix (shape is genes x cells)
- **model** (string, optional) - Specifies the null model to use for gene expression. Valid choices are: 'danb', 'bernoulli', 'normal', 'none'.
- **latent** (pandas.DataFrame, optional) - Latent space encoding cell-cell similarities with euclidean distances. Shape is (cells x dims)
- **distances** (pandas.DataFrame, optional) - Distances encoding cell-cell similarities directly. Shape is (cells x cells)
- **tree** (ete3.coretype.tree.TreeNode, optional) - Root tree node. Can be created using ete3.Tree
- **umi_counts** (pandas.Series, optional) - Total umi count per cell. Used as a size factor. If omitted, the sum over genes in the counts matrix is used.

### Request Example
```python
import pandas as pd
import hotspot

# Assuming gene_exp, latent_path are defined
gene_exp = pd.read_csv(path, index_col=0) # genes by cells
latent = pd.read_csv(latent_path, index_col=0) # cells by dims
hs = hotspot.Hotspot.legacy_init(gene_exp, model="normal", latent=latent)
```

### Response
#### Success Response (200)
Returns an initialized Hotspot object.

#### Response Example
None explicitly defined in source.
```

--------------------------------

### Initialize Hotspot with Legacy Method

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Use this class method to initialize a Hotspot object with legacy data formats. Ensure that either latent space, distances, or a tree structure is provided.

```python
>>> gene_exp = pd.read_csv(path, index_col=0) # genes by cells
>>> latent = pd.read_csv(latent_path, index_col=0) # cells by dims
>>> hs = hotspot.Hotspot.legacy_init(gene_exp, model="normal", latent=latent)

```

--------------------------------

### Hotspot Class Legacy Initialization

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Initializes the Hotspot object using a legacy method. Requires either latent space, a tree, or distance matrix.

```APIDOC
## classmethod hotspot.Hotspot.legacy_init(_counts_ , _model ='danb'_, _latent =None_, _distances =None_, _tree =None_, _umi_counts =None_)

### Description
Initialize a Hotspot object for analysis using legacy method. Either latent or tree or distances is required.

### Parameters
* **counts** (_pandas.DataFrame_) – Count matrix (shape is genes by cells)
* **model** (_string_ _,__optional_) – Specifies the null model to use for gene expression. Valid choices are: 'danb', 'bernoulli', 'normal', 'none'.
* **latent** (_pandas.DataFrame_ _,__optional_) – Latent space encoding cell-cell similarities with euclidean distances. Shape is (cells x dims)
* **distances** (_pandas.DataFrame_ _,__optional_) – Distances encoding cell-cell similarities directly Shape is (cells x cells)
* **tree** (_ete3.coretype.tree.TreeNode_) – Root tree node. Can be created using ete3.Tree
* **umi_counts** (_pandas.Series_ _,__optional_) – Total umi count per cell. Used as a size factor. If omitted, the sum over genes in the counts matrix is used

### Request Example
```python
gene_exp = pd.read_csv(path, index_col=0) # genes by cells
latent = pd.read_csv(latent_path, index_col=0) # cells by dims
hs = hotspot.Hotspot.legacy_init(gene_exp, model="normal", latent=latent)
```
```

--------------------------------

### Create Hotspot Object and Neighborhood Graph

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Initializes the Hotspot object with the annotated data, specifying the layer key for counts, the background model, the latent space key for PCA results, and the observation key for UMI counts. It then creates a k-nearest neighbors graph for neighborhood analysis, using binary weights and a specified number of neighbors.

```python
# Create the Hotspot object and the neighborhood graph
# hotspot works a lot faster with a csc matrix!
ada_cd4.layers["counts_csc"] = adata_cd4.layers["counts"].tocsc()
hs = hotspot.Hotspot(
    adata_cd4, 
    layer_key="counts_csc", 
    model='danb', 
    latent_obsm_key="X_pca",
    umi_counts_obs_key="total_counts"
)

hs.create_knn_graph(
    weighted_graph=False, n_neighbors=30,
)

```

--------------------------------

### Import Libraries

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Imports necessary Python libraries for data analysis, including hotspot, scanpy, muon, numpy, and mplscience. Warnings are suppressed for cleaner output.

```python
import warnings; warnings.simplefilter('ignore')

import hotspot
import scanpy as sc
import muon as mu

import numpy as np
import mplscience
```

--------------------------------

### Hotspot Class Initialization

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Initializes the Hotspot object for analysis. Requires either latent space, a tree, or distance matrix.

```APIDOC
## class hotspot.Hotspot(_adata_ , _layer_key =None_, _model ='danb'_, _latent_obsm_key =None_, _distances_obsp_key =None_, _tree =None_, _umi_counts_obs_key =None_, _use_gpu =False_)

### Description
Initialize a Hotspot object for analysis. Either latent or tree or distances is required.

### Parameters
* **adata** (_anndata.AnnData_) – Count matrix (shape is cells by genes)
* **layer_key** (_str_) – Key in adata.layers with count data, uses adata.X if None.
* **model** (_string_ _,__optional_) – Specifies the null model to use for gene expression. Valid choices are: 'danb', 'bernoulli', 'normal', 'none'.
* **latent_obsm_key** (_string_ _,__optional_) – Latent space encoding cell-cell similarities with euclidean distances. Shape is (cells x dims). Input is key in adata.obsm
* **distances_obsp_key** (_pandas.DataFrame_ _,__optional_) – Distances encoding cell-cell similarities directly Shape is (cells x cells). Input is key in adata.obsp
* **tree** (_ete3.coretype.tree.TreeNode_) – Root tree node. Can be created using ete3.Tree
* **umi_counts_obs_key** (_str_) – Total umi count per cell. Used as a size factor. If omitted, the sum over genes in the counts matrix is used
* **use_gpu** (_bool_ _,__optional_) – Whether to use GPU acceleration via CuPy for embarrassingly parallel operations. Requires CuPy and a CUDA-capable GPU. Default is False.
```

--------------------------------

### Download Lineage Tree

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Downloads the reconstructed lineage tree file from a Google Drive URL using the gdown library. This is essential for lineage analysis.

```python
%%capture
import gdown
url = "https://drive.google.com/uc?id=1FlrryhNv9Khwo_BstUTPCBOVaMgZLZ9N"
output = 'tree.txt'
gdown.download(url, output, quiet=False)

```

--------------------------------

### Download Data

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Downloads the specified dataset using wget. This command is intended for use in environments like Google Colab.

```python
if IN_COLAB:
    !wget https://cf.10xgenomics.com/samples/cell-exp/3.1.0/5k_pbmc_protein_v3_nextgem/5k_pbmc_protein_v3_nextgem_filtered_feature_bc_matrix.h5
```

--------------------------------

### Create Hotspot Object and KNN Graph

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Initializes the Hotspot object using log-normalized counts and a 'normal' model. Computes a k-nearest neighbors graph with binary weights and a specified number of neighbors.

```python
# Create the Hotspot object and the neighborhood graph

hs = hotspot.Hotspot(adata, model='normal', tree=tree)

hs.create_knn_graph(
    weighted_graph=False, n_neighbors=30,
)

```

--------------------------------

### Download Data Files

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Downloads necessary data files including count matrices, barcode information, gene annotations, and cell state kernels using wget. This is typically run in a Google Colab environment.

```python
%%capture
if IN_COLAB:
    !wget https://ftp.ncbi.nlm.nih.gov/geo/samples/GSM3302nnn/GSM3302829/suppl/GSM3302829_embryo3_SeuratBarcodes.tsv.gz
    !wget https://ftp.ncbi.nlm.nih.gov/geo/samples/GSM3302nnn/GSM3302829/suppl/GSM3302829_embryo3_SeuratGenes.tsv.gz
    !wget https://ftp.ncbi.nlm.nih.gov/geo/samples/GSM3302nnn/GSM3302829/suppl/GSM3302829_embryo3_SeuratNorm.mtx.gz
    !wget https://ftp.ncbi.nlm.nih.gov/geo/series/GSE122nnn/GSE122187/suppl/GSE122187_CellStateKernels.xls.gz

```

--------------------------------

### Create Hotspot Object

Source: https://hotspot.readthedocs.io/en/latest/index.html

Initialize the Hotspot object with an AnnData object and specify parameters for background model, latent space, and UMI counts.

```python
import hotspot
hs = hotspot.Hotspot(
    adata,
    layer_key="counts",
    model='danb',
    latent_obsm_key="X_pca",
    umi_counts_obs_key="total_counts"
)

```

--------------------------------

### Load and Standardize Kernel Values

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Loads kernel values from a gzipped Excel file, maps numerical kernel IDs to descriptive names, and standardizes the kernel values.

```python
import gzip
with gzip.open("GSE122187_CellStateKernels.xls.gz") as fin:
    kernels = pd.read_excel(fin, skiprows=1).set_index('Gene')
    
kernels = kernels.loc[
        :, kernels.columns.sort_values()
    ]

kernel_name_map = {
    0: "trophoblast stem cells",
    1: "neural ectoderm anterior",
    2: "primitive streak late",
    3: "anterior primitive streak",
    4: "primitive/definitive endoderm",
    5: "allantois",
    6: "secondary heart field/splanchnic lateral plate",
    7: "gut endoderm",
    8: "ectoderm early 1",
    9: "primitive blood early",
    10: "preplacodal ectoderm",
    11: "neural ectoderm posterior",
    12: "posterior lateral plate mesoderm",
    13: "hematopoietic/endothelial progenitors",
    14: "parietal endoderm",
    15: "amnion mesoderm early",
    16: "surface ectoderm",
    17: "epiblast",
    18: "somites",
    19: "ectoderm early 2",
    20: "splanchnic lateral plate/anterior paraxial mesoderm",
    21: "primitive heart tube",
    22: "primitive blood late",
    23: "notochord",
    24: "fore/midbrain",
    25: "distal extraembryonic ectoderm",
    26: "neuromesodermal progenitor early",
    27: "primordial germ cells",
    28: "differentiated trophoblasts",
    29: "visceral endoderm early",
    30: "presomitic mesoderm",
    31: "neuromesodermal progenitor late",
    32: "angioblasts",
    33: "neural crest",
    34: "pharyngeal arch mesoderm",
    35: "similar to neural crest",
    36: "primitive blood progenitors",
    37: "primitive streak early",
    38: "node",
    39: "future spinal cord",
    40: "visceral endoderm late",
    41: "amnion mesoderm late",
}

kernel_name_map = {k: v.capitalize() for k, v in kernel_name_map.items()}

kernels.columns = [kernel_name_map[x] for x in kernels.columns]

# Standardize each gene
kernels_z = kernels \
    .subtract(kernels.mean(axis=1), axis=0) \
    .divide(kernels.std(axis=1), axis=0)
```

--------------------------------

### Hotspot.create_modules()

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Creates modules based on computed correlations.

```APIDOC
## Hotspot.create_modules()

### Description
Generates modules from the calculated correlation data.

### Method
N/A (This is a method call)

### Parameters
(No parameters explicitly documented in the source)

### Request Example
(No request example provided)

### Response
(No response details provided)
```

--------------------------------

### Create Hotspot Object and Neighborhood Graph

Source: https://hotspot.readthedocs.io/en/latest/Spatial_Tutorial.html

Initializes the Hotspot object with the AnnData object, specifying count layer, background model, latent space key, and UMI counts key. Then, it computes the neighborhood graph using specified parameters for neighborhood size and graph weighting.

```python
hs = hotspot.Hotspot(
    adata, 
    layer_key="csc_counts",
    model='bernoulli', 
    latent_obsm_key="spatial", 
    umi_counts_obs_key="total_counts",
)

hs.create_knn_graph(
    weighted_graph=False, n_neighbors=300,
)
```

--------------------------------

### Create KNN Graph

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Constructs the k-nearest neighbors (KNN) graph and associated weights. The graph can be weighted or unweighted, and the number of neighbors can be specified. Use approximate neighbors for efficiency when initialized with latent space.

```python
hs.create_knn_graph(_weighted_graph =False_, _n_neighbors =30_, _neighborhood_factor =3_, _approx_neighbors =True_)
```

--------------------------------

### Initialize Hotspot Object

Source: https://hotspot.readthedocs.io/en/latest/_modules/hotspot/hotspot.html

Initializes the Hotspot class with an AnnData object and specifies parameters for analysis. Requires either latent space, distances, or a tree structure for cell-cell similarity.

```python
import anndata
import numpy as np
import pandas as pd
import warnings

from scipy.sparse import issparse, csr_matrix

from .knn import (
    neighbors_and_weights,
    neighbors_and_weights_from_distances,
    tree_neighbors_and_weights,
    make_weights_non_redundant,
)
from .local_stats import compute_hs
from .local_stats_pairs import compute_hs_pairs, compute_hs_pairs_centered_cond

from . import modules
from .plots import local_correlation_plot
from .gpu import is_gpu_available
from tqdm import tqdm




class Hotspot:
    def __init__(
        self,
        adata,
        layer_key=None,
        model="danb",
        latent_obsm_key=None,
        distances_obsp_key=None,
        tree=None,
        umi_counts_obs_key=None,
        use_gpu=False,
    ):
        """Initialize a Hotspot object for analysis

        Either `latent` or `tree` or `distances` is required.

        Parameters
        ----------
        adata : anndata.AnnData
            Count matrix (shape is cells by genes)
        layer_key: str
            Key in adata.layers with count data, uses adata.X if None.
        model : string, optional
            Specifies the null model to use for gene expression.
            Valid choices are:

                - 'danb': Depth-Adjusted Negative Binomial
                - 'bernoulli': Models probability of detection
                - 'normal': Depth-Adjusted Normal
                - 'none': Assumes data has been pre-standardized

        latent_obsm_key : string, optional
            Latent space encoding cell-cell similarities with euclidean
            distances.  Shape is (cells x dims). Input is key in adata.obsm
        distances_obsp_key : pandas.DataFrame, optional
            Distances encoding cell-cell similarities directly
            Shape is (cells x cells). Input is key in adata.obsp
        tree : ete3.coretype.tree.TreeNode
            Root tree node.  Can be created using ete3.Tree
        umi_counts_obs_key : str
            Total umi count per cell.  Used as a size factor.
            If omitted, the sum over genes in the counts matrix is used
        use_gpu : bool, optional
            Whether to use GPU acceleration via CuPy for embarrassingly
            parallel operations. Requires CuPy and a CUDA-capable GPU.
            Default is False.
        """
        counts = self._counts_from_anndata(adata, layer_key)
        distances = (
            adata.obsp[distances_obsp_key] if distances_obsp_key is not None else None
        )
        latent = adata.obsm[latent_obsm_key] if latent_obsm_key is not None else None
        umi_counts = (
            adata.obs[umi_counts_obs_key] if umi_counts_obs_key is not None else None
        )

        if latent is None and distances is None and tree is None:
            raise ValueError(
                "Neither `latent_obsm_key` or `tree` or `distances_obsp_key` arguments were supplied.  One of these is required"
            )

        if latent is not None and distances is not None:
            raise ValueError(
                "Both `latent_obsm_key` and `distances_obsp_key` provided - only one of these should be provided."
            )

        if latent is not None and tree is not None:
            raise ValueError(
                "Both `latent_obsm_key` and `tree` provided - only one of these should be provided."
            )

        if distances is not None and tree is not None:
            raise ValueError(
                "Both `distances_obsp_key` and `tree` provided - only one of these should be provided."
            )

        if latent is not None:
            latent = pd.DataFrame(latent, index=adata.obs_names)

        # because of transpose we check if its csr
        if issparse(counts) and not isinstance(counts, csr_matrix):
            warnings.warn(
                "Hotspot will work faster when counts are a csr sparse matrix."
            )

        if tree is not None:
            try:
                all_leaves = []
                for x in tree:
                    if x.is_leaf():
                        all_leaves.append(x.name)
            except:
                raise ValueError("Can't parse supplied tree")

            if len(all_leaves) != counts.shape[1] or len(
                set(all_leaves) & set(adata.obs_names)
            ) != len(all_leaves):
                raise ValueError(
                    "Tree leaf labels don't match columns in supplied counts matrix"
                )

        if umi_counts is None:
            umi_counts = counts.sum(axis=0)
            # handles sparse matrix outputs of sum
            umi_counts = np.asarray(umi_counts).ravel()
        else:
            assert umi_counts.size == counts.shape[1]

        if not isinstance(umi_counts, pd.Series):
            umi_counts = pd.Series(umi_counts, index=adata.obs_names)

        valid_models = {"danb", "bernoulli", "normal", "none"}
        if model not in valid_models:
            
```

--------------------------------

### Load and Process Lineage Tree

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Loads the lineage tree using ete3 and enumerates all the leaf nodes, which represent individual cells with recoverable lineage information.

```python
# Load the tree and enumerate the leaves
from ete3 import Tree
tree = Tree("tree.txt", format=1)
leaves = set()
for tn in tree.traverse('postorder'):
    if tn.is_leaf():
        leaves.add(tn.name)
len(leaves)

```

--------------------------------

### create_knn_graph

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Creates the KNN graph and graph weights, storing them in self.neighbors and self.weights.

```APIDOC
## hotspot.Hotspot.create_knn_graph(_weighted_graph =False_, _n_neighbors =30_, _neighborhood_factor =3_, _approx_neighbors =True_)

### Description
Create’s the KNN graph and graph weights. The resulting matrices containing the neighbors and weights are stored in the object at self.neighbors and self.weights.

### Parameters
* **weighted_graph** (_bool_) – Whether or not to create a weighted graph
* **n_neighbors** (_int_) – Neighborhood size
* **neighborhood_factor** (_float_) – Used when creating a weighted graph. Sets how quickly weights decay relative to the distances within the neighborhood. The weight for a cell with a distance d will decay as exp(-d/D) where D is the distance to the n_neighbors/neighborhood_factor-th neighbor.
* **approx_neighbors** (_bool_) – Use approximate nearest neighbors or exact scikit-learn neighbors. Only when hotspot initialized with latent.
```

--------------------------------

### Create Hotspot Object

Source: https://hotspot.readthedocs.io/en/latest/_sources/index.rst

Initialize the Hotspot object with an AnnData object, specifying the layer for counts, the background model, the key for latent space, and the key for UMI counts.

```python
import hotspot
hs = hotspot.Hotspot(
    adata,
    layer_key="counts",
    model='danb',
    latent_obsm_key="X_pca",
    umi_counts_obs_key="total_counts"
)
```

--------------------------------

### Hotspot.create_knn_graph()

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Creates a k-nearest neighbors (KNN) graph.

```APIDOC
## Hotspot.create_knn_graph()

### Description
Constructs a k-nearest neighbors (KNN) graph based on the provided data.

### Method
N/A (This is a method call)

### Parameters
(No parameters explicitly documented in the source)

### Request Example
(No request example provided)

### Response
(No response details provided)
```

--------------------------------

### Compute UMAP for Cell Visualization

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Prepares the AnnData object for UMAP visualization by scaling data, performing PCA, and computing neighbors. Requires scanpy.

```python
adata.raw = adata

sc.pp.scale(adata)
sc.tl.pca(adata)
sc.pp.neighbors(adata)
sc.tl.umap(adata)
```

--------------------------------

### Create Gene Modules with Clustering

Source: https://hotspot.readthedocs.io/en/latest/index.html

Use this method to group genes into modules via agglomerative clustering. It allows control over minimum gene thresholds, core gene inclusion, and FDR thresholds for merging branches. The output is a pandas Series mapping genes to module numbers, with -1 for unassigned genes.

```python
modules = hs.create_modules(
    min_gene_threshold=30, core_only=True, fdr_threshold=0.05
)

```

--------------------------------

### create_knn_graph

Source: https://hotspot.readthedocs.io/en/latest/_modules/hotspot/hotspot.html

Creates the k-Nearest Neighbors (KNN) graph and associated weights. This method can utilize latent space, a tree structure, or pre-computed distances for graph construction.

```APIDOC
## create_knn_graph

### Description
Creates the KNN graph and graph weights. The resulting matrices containing the neighbors and weights are stored in the object at `self.neighbors` and `self.weights`.

### Method
`create_knn_graph(self, weighted_graph=False, n_neighbors=30, neighborhood_factor=3, approx_neighbors=True)`

### Parameters
#### Path Parameters
None

#### Query Parameters
- **weighted_graph** (bool) - Optional - Whether or not to create a weighted graph.
- **n_neighbors** (int) - Optional - Neighborhood size. Defaults to 30.
- **neighborhood_factor** (float) - Optional - Used when creating a weighted graph. Sets how quickly weights decay relative to the distances within the neighborhood. The weight for a cell with a distance d will decay as exp(-d/D) where D is the distance to the `n_neighbors`/`neighborhood_factor`-th neighbor. Defaults to 3.
- **approx_neighbors** (bool) - Optional - Use approximate nearest neighbors or exact scikit-learn neighbors. Only when hotspot initialized with `latent`. Defaults to True.

### Request Example
```python
hotspot_instance.create_knn_graph(weighted_graph=True, n_neighbors=50)
```

### Response
This method modifies the object in-place by setting `self.neighbors` and `self.weights`. It does not return a value.

### Error Handling
- Raises `ValueError` if `weighted_graph=True` is used when the metric space is a `tree`.

```

--------------------------------

### Create Weighted or Unweighted KNN Graph

Source: https://hotspot.readthedocs.io/en/latest/_modules/hotspot/hotspot.html

Generates a k-nearest neighbors graph and associated weights. Supports using latent space, a pre-computed tree, or distance matrices as the metric space.

```python
if self.latent is not None:
    neighbors, weights = neighbors_and_weights(
        self.latent,
        n_neighbors=n_neighbors,
        neighborhood_factor=neighborhood_factor,
        approx_neighbors=approx_neighbors,
    )
elif self.tree is not None:
    if weighted_graph:
        raise ValueError(
            "When using `tree` as the metric space, `weighted_graph=True` is not supported"
        )
    neighbors, weights = tree_neighbors_and_weights(
        self.tree, n_neighbors=n_neighbors, cell_labels=self.adata.obs_names
    )
else:
    neighbors, weights = neighbors_and_weights_from_distances(
        self.distances,
        cell_index=self.adata.obs_names,
        n_neighbors=n_neighbors,
        neighborhood_factor=neighborhood_factor,
    )

neighbors = neighbors.loc[self.adata.obs_names]
weights = weights.loc[self.adata.obs_names]

self.neighbors = neighbors

if not weighted_graph:
    weights = pd.DataFrame(
        np.ones_like(weights.values),
        index=weights.index,
        columns=weights.columns,
    )

weights = make_weights_non_redundant(neighbors.values, weights.values)

weights = pd.DataFrame(
    weights, index=neighbors.index, columns=neighbors.columns
)

self.weights = weights
```

--------------------------------

### Hotspot.calculate_module_scores()

Source: https://hotspot.readthedocs.io/en/latest/hotspot.html

Calculates scores for the created modules.

```APIDOC
## Hotspot.calculate_module_scores()

### Description
Assigns scores to the identified modules.

### Method
N/A (This is a method call)

### Parameters
(No parameters explicitly documented in the source)

### Request Example
(No request example provided)

### Response
(No response details provided)
```

--------------------------------

### Load 10x Genomics Data

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Loads data from a 10x Genomics H5 file into a MuData object. Ensure the file path is correct.

```python
mdata = mu.read_10x_h5("/content/5k_pbmc_protein_v3_nextgem_filtered_feature_bc_matrix.h5")
```

--------------------------------

### Normalize and Log1p Data

Source: https://hotspot.readthedocs.io/en/latest/Spatial_Tutorial.html

Prepares the AnnData object for downstream analysis by normalizing total counts and applying a log1p transformation.

```python
sc.pp.normalize_total(adata)
sc.pp.log1p(adata)
```

--------------------------------

### Display MuData Object Info

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Prints a summary of the MuData object, showing the number of observations and variables across its modalities.

```python
mdata
```

--------------------------------

### create_modules

Source: https://hotspot.readthedocs.io/en/latest/_modules/hotspot/hotspot.html

Groups genes into modules based on correlation thresholds. Results are stored in `self.modules` and `self.linkage`.

```APIDOC
## create_modules

### Description
Groups genes into modules.

### Parameters
* `min_gene_threshold` (int) - Optional - Controls how small modules can be. Defaults to 20.
* `core_only` (bool) - Optional - Whether or not to assign ambiguous genes to a module or leave unassigned. Defaults to True.
* `fdr_threshold` (float) - Optional - Correlation threshold at which to stop assigning genes to modules. Defaults to 0.05.

### Returns
* `modules` (pandas.Series) - Maps gene to module number. Unassigned genes are indicated with -1.
```

--------------------------------

### Import Required Libraries

Source: https://hotspot.readthedocs.io/en/latest/Lineage_Tutorial.html

Imports essential libraries for data manipulation, plotting, and analysis, including numpy, pandas, hotspot, matplotlib, seaborn, scanpy, and scipy.

```python
import warnings; warnings.simplefilter('ignore')

import numpy as np
import pandas as pd
import hotspot
import matplotlib.pyplot as plt
import matplotlib.colors
import seaborn as sns
import mplscience

from scipy.io import mmread
from scipy.sparse import csr_matrix

import scanpy as sc

```

--------------------------------

### Create AnnData from Counts and Distances

Source: https://hotspot.readthedocs.io/en/latest/_modules/hotspot/hotspot.html

Constructs an AnnData object from gene counts, UMI counts, optional distances, and latent space. Useful for preparing data for hotspot analysis.

```python
input_adata = anndata.AnnData(counts)
input_adata = input_adata.transpose()
tc_key = "total_counts"
input_adata.obs[tc_key] = umi_counts.values
dkey = "distances"
if distances is not None:
    input_adata.obsp[dkey] = distances
    dist_input = True
else:
    dist_input = False
lkey = "latent"
if latent is not None:
    input_adata.obsm[lkey] = np.asarray(latent)
    latent_input = True
else:
    latent_input = False

return cls(
    input_adata,
    model=model,
    latent_obsm_key=lkey if latent_input else None,
    distances_obsp_key=dkey if dist_input else None,
    umi_counts_obs_key=tc_key,
    tree=tree,
)
```

--------------------------------

### Create Gene Modules

Source: https://hotspot.readthedocs.io/en/latest/_modules/hotspot/hotspot.html

Groups genes into modules based on local correlation. Stores results in `self.modules` and `self.linkage`.

```python
self.create_modules()
```

--------------------------------

### Plot PCA Variance Ratio

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Visualizes the proportion of variance explained by each principal component. This helps in determining the number of components to retain for downstream analysis.

```python
with mplscience.style_context():
    sc.pl.pca_variance_ratio(adata_cd4)
```

--------------------------------

### Plot Module Scores on UMAP

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Visualizes the module scores on the UMAP projection using a specified style context.

```python
with mplscience.style_context():
    sc.pl.umap(adata_cd4, color=module_cols, frameon=False, vmin=-1, vmax=1)
```

--------------------------------

### Plot Violin Plots for QC Metrics

Source: https://hotspot.readthedocs.io/en/latest/CD4_Tutorial.html

Generates violin plots to visualize key quality control metrics such as the number of genes, total counts, and percentage of mitochondrial counts. Uses mplscience for enhanced plotting style.

```python
with mplscience.style_context():
    sc.pl.violin(adata, ['n_genes_by_counts', 'total_counts', 'pct_counts_mt'],
                jitter=0.4, multi_panel=True, log=True)
```
Tokens:
Raw