### Download and Execute Setup Script

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_colab.ipynb

Downloads and runs the setup script to configure the environment, including setting up GPU support and installing necessary packages for cell2location. This is essential for accelerating computations on Colab.

```python
! wget https://raw.githubusercontent.com/BayraktarLab/cell2location/master/docs/notebooks/colab/setup_collab.sh
! bash setup_collab.sh 
```

--------------------------------

### Install Miniconda

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/installing_pymc3.md

Download and install Miniconda. Use the specified prefix for the installation directory.

```bash
cd /path/to/software
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# use prefix /path/to/software/miniconda3
```

--------------------------------

### Install Cell2location with Tutorials

Source: https://github.com/bayraktarlab/cell2location/blob/master/README.md

After activating the conda environment, install the cell2location package with tutorial dependencies using pip.

```bash
pip install cell2location[tutorials]
```

--------------------------------

### Install Miniconda

Source: https://github.com/bayraktarlab/cell2location/blob/master/README.md

Instructions for downloading and installing Miniconda on Linux. Ensure you are in the desired directory before running these commands.

```bash
cd /path/to/software
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
```

--------------------------------

### Download NanostringWTA Example Data

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA.ipynb

Downloads the smallest example dataset for NanostringWTA analysis and a CSV file containing mean expression profiles. Ensure sufficient disk space and internet connectivity.

```python
# Download data:
os.mkdir('./data')
os.system('cd ./data && wget https://cell2location.cog.sanger.ac.uk/nanostringWTA/nanostringWTA_fetailBrain_AnnData_smallestExample.p')
os.system('cd ./data && wget https://cell2location.cog.sanger.ac.uk/nanostringWTA/polioudakis2019_meanExpressionProfiles.csv')
```

--------------------------------

### Compile Documentation

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/README.md

Navigate to the docs directory, install Python dependencies, and build the HTML documentation using Sphinx. Do not commit compiled documentation files.

```shell
cd docs
pip install -r requirements.txt
make html
```

--------------------------------

### Install cell2location Package

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/installing_pymc3.md

Activate the 'cellpymc' environment and install the cell2location package directly from its GitHub repository using pip.

```bash
conda activate cellpymc
pip install git+https://github.com/BayraktarLab/cell2location.git
```

--------------------------------

### Setup Conda Environment for A100 GPUs

Source: https://github.com/bayraktarlab/cell2location/blob/master/README.md

Creates and activates a conda environment with specific Python and PyTorch versions for A100 GPUs. Installs scvi-tools and cell2location with tutorial and development dependencies. Registers the environment for Jupyter notebooks.

```bash
export PYTHONNOUSERSITE="True"
conda create -y -n cell2location_cuda118_torch22 python=3.10
conda activate cell2location_cuda118_torch22

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

pip3 install scvi-tools==1.1.2

pip install git+https://github.com/BayraktarLab/cell2location.git#egg=cell2location[tutorials,dev]
python -m ipykernel install --user --name=cell2location_cuda118_torch22 --display-name='Environment (cell2location_cuda118_torch22)'
```

--------------------------------

### Get help for RegressionGeneBackgroundCoverageTorch

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_estimating_signatures.ipynb

Use this to view the documentation for the RegressionGeneBackgroundCoverageTorch model.

```python
help(cell2location.models.RegressionGeneBackgroundCoverageTorch)
```

--------------------------------

### Load packages and initialize GPU

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_colab.ipynb

Imports necessary libraries for spatial transcriptomics analysis and configures Theano to use a specific GPU. This setup is crucial for performance.

```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import os

data_type = 'float32'

# this line forces theano to use the GPU and should go before importing cell2location
os.environ["THEANO_FLAGS"] = 'device=cuda0,floatX=' + data_type + ',force_device=True'
# if using the CPU uncomment this:
#os.environ["THEANO_FLAGS"] = 'device=cpu,floatX=float32,openmp=True,force_device=True'

import cell2location

import matplotlib as mpl
from matplotlib import rcParams
import matplotlib.pyplot as plt
import seaborn as sns

# silence scanpy that prints a lot of warnings
import warnings
warnings.filterwarnings('ignore')

```

--------------------------------

### Install cell2location and dependencies

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Installs the cell2location package and its dependencies, including tutorials, using pip. This is necessary for running the analysis in Google Colab.

```python
import sys
IN_COLAB = "google.colab" in sys.modules
if IN_COLAB:
    !pip install --quiet scvi-colab
    from scvi_colab import install
    install()
    !pip install --quiet git+https://github.com/BayraktarLab/cell2location#egg=cell2location[tutorials]
```

--------------------------------

### Load Modules and Configure Theano

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA.ipynb

Imports necessary libraries and configures Theano for GPU acceleration with specified data types. Ensure CUDA is available and Theano is installed.

```python
import sys,os
import pickle
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scanpy as sc
from IPython.display import Image
data_type = 'float32'
os.environ["THEANO_FLAGS"] = 'device=cuda,floatX=' + data_type + ',force_device=True' + ',dnn.enabled=False'
import cell2location
```

--------------------------------

### Setup Anndata for Cell2location Model

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA_Pyro.ipynb

Prepares the anndata object for the cell2location model by specifying keys for batch, nuclei, and negative probes.

```python
cell2location.models.Cell2location_WTA.setup_anndata(adata=adata_wta, batch_key = 'slide', nuclei_key = 'nuclei',
                                                neg_probes_key = 'negative_probes')
```

--------------------------------

### Run Docker Container with GPU

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/dockersingularity.md

Launches the cell2location Docker container with GPU support enabled. Ensure Nvidia Container Toolkit is installed.

```none
docker run -i --rm -p 8848:8888 --gpus all quay.io/vitkl/cell2location:latest
```

--------------------------------

### Create and Inspect Regression Model

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Initializes a RegressionModel with the prepared AnnData object and displays a summary of the setup. This step is crucial for verifying that the AnnData object has been correctly configured for model training.

```python
# create the regression model
from cell2location.models import RegressionModel
mod = RegressionModel(adata_ref) 

# view anndata_setup as a sanity check
mod.view_anndata_setup()
```

--------------------------------

### Load NanostringWTA AnnData Object

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA.ipynb

Loads the downloaded NanostringWTA example data into an AnnData object using pickle. Ensure the data file is present in the 'data' directory.

```python
adata_wta = pickle.load(open("data/nanostringWTA_fetailBrain_AnnData_smallestExample.p", "rb" ))
```

--------------------------------

### Install PyMC3 and Dependencies with Pip

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/installing_pymc3.md

Activate the 'cellpymc' conda environment and install PyMC3 (versions 3.8 to <3.10), ArviZ, torch, and pyro-ppl using pip. This step is crucial for ensuring GPU compatibility.

```bash
conda activate cellpymc
pip install plotnine "arviz==0.10.0" "pymc3>=3.8,<3.10" torch pyro-ppl
```

--------------------------------

### Compute Posterior Distribution Quantiles

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Calculate arbitrary quantiles of the posterior distribution. This example also visualizes the relationship between the median and mean of the posterior samples for 'w_sf'.

```python
medians = mod.posterior_quantile(q=0.5, batch_size=mod.adata.n_obs, accelerator='gpu')

with mpl.rc_context({'axes.facecolor':  'white',
                     'figure.figsize': [5, 5]}):
    plt.scatter(medians['w_sf'].flatten(), mod.samples['post_sample_means']['w_sf'].flatten());
    plt.xlabel('median');
    plt.ylabel('mean');
```

--------------------------------

### Run Docker Container with GPU and Volume Mount

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/dockersingularity.md

Starts the cell2location Docker container with GPU support and mounts a local directory into the container for data access.

```none
docker run -v /host/directory:/container/directory -i --rm -p 8848:8888 --gpus all quay.io/vitkl/cell2location:latest
```

--------------------------------

### Setup AnnData for Cell2location

Source: https://github.com/bayraktarlab/cell2location/blob/master/README.md

Configures an AnnData object for use with cell2location models by specifying the batch key. This function is part of the cell2location library.

```python
cell2location.models.Cell2location.setup_anndata(
    adata=adata_vis,
    batch_key="batch")
```

--------------------------------

### Load Packages and Setup GPU for cell2location

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo.ipynb

Imports necessary Python packages for data analysis and machine learning. It also configures Theano to use the GPU for accelerated computations, which is crucial for large datasets. Ensure 'device=cuda0' matches your available GPU.

```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import os
import gc

# this line forces theano to use the GPU and should go before importing cell2location
os.environ["THEANO_FLAGS"] = 'device=cuda0,floatX=float32,force_device=True'
# if using the CPU uncomment this:
#os.environ["THEANO_FLAGS"] = 'device=cpu,floatX=float32,openmp=True,force_device=True'

import cell2location

import matplotlib as mpl
from matplotlib import rcParams
import matplotlib.pyplot as plt
import seaborn as sns

# silence scanpy that prints a lot of warnings
import warnings
warnings.filterwarnings('ignore')
```

--------------------------------

### Create Conda Environment Manually

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/installing_pymc3.md

Manually create a conda environment named 'cellpymc' with specified Python packages. Note that PyMC3 and Theano should be installed via pip.

```bash
conda create -n cellpymc python=3.7 numpy pandas jupyter leidenalg python-igraph scanpy \
louvain hyperopt loompy cmake nose tornado dill ipython bbknn seaborn matplotlib request \
mkl-service pygpu --channel bioconda --channel conda-forge
```

--------------------------------

### Download and Unzip Cell2location Results

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_downstream.ipynb

Checks if the results directory exists and downloads/unzips the pre-computed cell2location results from a specified URL if it does not. This is a convenient way to obtain example data.

```python
if os.path.exists(f'{results_folder}{r["run_name"]}') is not True:
    os.mkdir('./results')
    os.mkdir(f'{results_folder}')
    os.system(f'cd {results_folder} && wget https://cell2location.cog.sanger.ac.uk/tutorial/mouse_brain_visium_results/{["run_name"]}.zip')
    os.system(f'cd {results_folder} && unzip {r["run_name"]}.zip')
```

--------------------------------

### Batch Correction and UMAP Calculation

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_estimating_signatures.ipynb

Applies BBKNN for batch correction across samples and then computes the UMAP embedding for visualization. Ensure BBKNN is installed.

```python
# Here BBKNN (https://github.com/Teichlab/bbknn) is used to align batches (10X experiments) 
import bbknn
bbknn.bbknn(adata_snrna_raw, neighbors_within_batch = 3, batch_key = 'sample', n_pcs = 79)
sc.tl.umap(adata_snrna_raw, min_dist = 0.8, spread = 1.5)

#########################

adata_snrna_raw = adata_snrna_raw[adata_snrna_raw.obs['annotation_1'].argsort(),:]
```

--------------------------------

### Configure CUDA Environment Variables

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/commonerrors.md

Manually set system CUDA environment variables in your .bashrc file to help Theano locate and utilize your CUDA installation, including cuDNN. This is an alternative to using pre-configured Docker or Singularity images.

```bash
# cuda v
cuda_v=-10.2
export CUDA_HOME=/usr/local/cuda$cuda_v
export CUDA_PATH=$CUDA_HOME
export LD_LIBRARY_PATH=/usr/local/cuda$cuda_v/lib64:$LD_LIBRARY_PATH
export PATH=/usr/local/cuda$cuda_v/bin:$PATH
```

--------------------------------

### Get Posterior Distribution Samples

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Fetch samples from the posterior distribution for specific variables. This can be memory intensive and is not suitable for Google Colab. Specify variables to limit memory usage.

```python
samples_w_sf = mod.sample_posterior(num_samples=1000, accelerator='gpu', return_samples=True,
                                    batch_size=2020,
                                    return_sites=['w_sf', 'm_g', 'u_sf_mRNA_factors'])
samples_w_sf['posterior_samples']['w_sf'].shape
```

--------------------------------

### Initialize and Train CoLocatedGroupsSklearnNMF Model

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_downstream.ipynb

Initializes and trains the CoLocatedGroupsSklearnNMF model multiple times to assess the consistency of identified compartments. Ensure cell2location is installed and necessary data is loaded.

```python
# number of cell type combinations - educated guess assuming that most cell types don't co-locate
n_fact = int(30)

# extract cell abundance from cell2location
X_data = adata_vis.uns['mod']['post_sample_q05']['spot_factors']

import cell2location.models as c2l
# create model class
mod_sk = c2l.CoLocatedGroupsSklearnNMF(n_fact, X_data, 
        n_iter = 10000,
        verbose = True,
        var_names=adata_vis.uns['mod']['fact_names'],
        obs_names=adata_vis.obs_names,
        fact_names=['fact_' + str(i) for i in range(n_fact)],
        sample_id=adata_vis.obs['sample'],
        init='random', random_state=0,
        nmf_kwd_args={'tol':0.0001})

# train 5 times to evaluate stability
mod_sk.fit(n=5, n_type='restart') 

```

--------------------------------

### Get cell2location Version

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Check the installed version of the cell2location package.

```python
cell2location.__version__
```

--------------------------------

### Example NaN Error Message

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/commonerrors.md

A typical error message indicating numerical instability during model optimization. This often arises from poor data quality, incorrect model setup, or outdated CUDA versions.

```text
FloatingPointError: NaN occurred in optimization. 
The current approximation of RV `gene_level_beta_hyp_log__`.ravel()[0] is NaN.
...
```

--------------------------------

### Prevent Python User Site Installation

Source: https://github.com/bayraktarlab/cell2location/blob/master/README.md

Ensures Python does not use the user site for package installations, which helps in creating fully isolated conda environments. This should be run before creating and activating conda environments.

```bash
export PYTHONNOUSERSITE="True"
```

--------------------------------

### Set Data and Results Folders

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_estimating_signatures.ipynb

Define local paths for storing downloaded data and analysis results. Ensure these directories exist before proceeding.

```python
sc_data_folder = './data/'
results_folder = './results/mouse_brain_snrna/'
```

--------------------------------

### Create Environment from File (Preferred)

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/installing_pymc3.md

Clone the cell2location repository and create the conda environment using the provided environment.yml file. This method is preferred for resolving potential environment creation issues.

```bash
git clone https://github.com/BayraktarLab/cell2location.git
cd cell2location
conda env create -f environment.yml
```

--------------------------------

### Run Code Formatting and Linting

Source: https://github.com/bayraktarlab/cell2location/blob/master/CLAUDE.md

Execute these commands to ensure code adheres to project standards for formatting and linting.

```bash
black --line-length 120 .
isort .
flake8
```

--------------------------------

### Add Jupyter Kernel for Environment

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/installing_pymc3.md

Activate the 'cellpymc' conda environment and install an IPython kernel for it, which allows Jupyter notebooks to use this specific environment.

```bash
conda activate cellpymc
python -m ipykernel install --user --name=cellpymc --display-name='Environment (cellpymc)'
```

--------------------------------

### Theano GPU Initialization Confirmation

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo.ipynb

This output confirms that Theano has successfully initialized and is using the specified GPU device. Pay attention to any warnings regarding cuDNN versions, as they might indicate performance issues.

```text
/nfs/team283/vk7/software/miniconda3farm5/envs/cellpymc/lib/python3.7/site-packages/theano/gpuarray/dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
  warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 8201 on context None
Mapped name None to device cuda0: Tesla V100-SXM2-32GB (0000:62:00.0)

```

--------------------------------

### Run Colocation Pipeline with Varying Factors

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_downstream.ipynb

Automates NMF model training for a range of factors to identify meaningful cell type groups. Specify the range of factors, iterations, sample identification column, and model parameters.

```python
from cell2location import run_colocation
res_dict, adata_vis = run_colocation(
                   adata_vis, model_name='CoLocatedGroupsSklearnNMF',
                   verbose=False, return_all=True,
    
                   train_args={
                    'n_fact': np.arange(10, 40), # IMPORTANT: range of number of factors (10-40 here)
                    'n_iter': 20000, # maximum number of training iterations
                       
                    'sample_name_col': 'sample', # columns in adata_vis.obs that identifies sample
                       
                    'mode': 'normal',
                    'n_type': 'restart', 'n_restarts': 5 # number of training restarts
                   },
    
                   model_kwargs={'init': 'random', 'random_state': 0, 'nmf_kwd_args': {'tol': 0.00001}},
    
                   posterior_args={},
                   export_args={'path': results_folder + 'std_model/'+r['run_name']+'/CoLocatedComb/',
                                'run_name_suffix': ''})

```

--------------------------------

### Alternative Theano GPU Initialization Confirmation

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo.ipynb

This is an alternative confirmation message indicating successful Theano GPU initialization. It may vary slightly based on your environment and Theano/cuDNN versions.

```text
/lib/python3.7/site-packages/theano/gpuarray/dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
  warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 7605 on context None
Mapped name None to device cuda0: Tesla V100-SXM2-32GB (0000:89:00.0)

```

--------------------------------

### Import Required Modules and Configure Theano

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA_Pyro.ipynb

Imports necessary libraries for data analysis and cell2location, and configures Theano settings for plotting.

```python
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl
import os
import pickle
from IPython.display import Image

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text for PDFs
```

--------------------------------

### Define results folder and run names

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Sets up the directory structure for saving analysis results. Defines paths for reference regression and cell2location model outputs.

```python
results_folder = './results/lymph_nodes_analysis/'

# create paths and names to results folders for reference regression and cell2location models
ref_run_name = f'{results_folder}/reference_signatures'
run_name = f'{results_folder}/cell2location_map'
```

--------------------------------

### Download Reference Data

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_estimating_signatures.ipynb

Check if data directories exist and create them if necessary. Download the single-cell RNA-seq data and annotation files using wget. This step requires an internet connection.

```python
if os.path.exists(sc_data_folder) is not True:
    os.mkdir(sc_data_folder)
os.system(f'cd {sc_data_folder} && wget https://cell2location.cog.sanger.ac.uk/tutorial/mouse_brain_snrna/all_cells_20200625.h5ad')
os.system(f'cd {sc_data_folder} && wget https://cell2location.cog.sanger.ac.uk/tutorial/mouse_brain_snrna/snRNA_annotation_astro_subtypes_refined59_20200823.csv')
```

--------------------------------

### Prevent User Site Packages in Conda

Source: https://github.com/bayraktarlab/cell2location/blob/master/README.md

Set the PYTHONNOUSERSITE environment variable before creating or activating a conda environment to ensure a fully isolated installation. This prevents Python from using user-specific site packages.

```bash
export PYTHONNOUSERSITE="literallyanyletters"
```

--------------------------------

### Initialize and Train Cell2location Model

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Initialize the Cell2location model with the annotated data and reference cell state signatures. Key hyperparameters like `N_cells_per_location` and `detection_alpha` should be carefully chosen based on tissue characteristics and RNA detection sensitivity.

```python
# create and train the model
mod = cell2location.models.Cell2location(
    adata_vis, cell_state_df=inf_aver, 
    # the expected average cell abundance: tissue-dependent 
    # hyper-prior which can be estimated from paired histology:
    N_cells_per_location=30,
    # hyperparameter controlling normalisation of
    # within-experiment variation in RNA detection:
    detection_alpha=20
) 
mod.view_anndata_setup()
```

--------------------------------

### Download Singularity Image

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/dockersingularity.md

Fetches the cell2location Singularity image file from the specified URL.

```default
wget https://cell2location.cog.sanger.ac.uk/singularity/cell2location-v0.06-alpha.sif
```

--------------------------------

### Create Results Directory

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_estimating_signatures.ipynb

Ensure the main results directory and a specific subdirectory for mouse brain analysis are created. This is essential for saving intermediate and final outputs.

```python
if os.path.exists(results_folder) is not True:
    os.mkdir('./results')
    os.mkdir(results_folder)
```

--------------------------------

### Set Data and Results Paths

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_colab.ipynb

Define the directory paths for input data and output results. This is essential for organizing files and ensuring the script can locate them.

```python
sp_data_folder = './data/mouse_brain_visium_wo_cloupe_data/'
results_folder = './results/mouse_brain_snrna/'

regression_model_output = 'RegressionGeneBackgroundCoverageTorch_65covariates_40532cells_12819genes'
reg_path = f'{results_folder}regression_model/{regression_model_output}/'

```

--------------------------------

### Load Necessary Python Packages

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_downstream.ipynb

Imports essential libraries for data manipulation, scientific computing, and visualization, including cell2location specific modules. It also configures warning filters.

```python
import sys
import pickle
import scanpy as sc
import anndata
import numpy as np
import os

import cell2location

import matplotlib as mpl
from matplotlib import pyplot as plt

# scanpy prints a lot of warnings
import warnings
warnings.filterwarnings('ignore')
```

--------------------------------

### Plot QC Metrics per Sample

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_colab.ipynb

Generates distribution plots for total counts and number of genes per location for each Visium sample. This visualization aids in identifying potential outliers or batch effects and informs filtering strategies.

```python
# PLOT QC FOR EACH SAMPLE 
fig, axs = plt.subplots(len(slides), 4, figsize=(15, 4*len(slides)-4))
for i, s in enumerate(adata.obs['sample'].unique()):
    #fig.suptitle('Covariates for filtering')
    
    slide = select_slide(adata, s)
    sns.distplot(slide.obs['total_counts'], 
                 kde=False, ax = axs[i, 0])
    axs[i, 0].set_xlim(0, adata.obs['total_counts'].max())
    axs[i, 0].set_xlabel(f'total_counts | {s}')
    
    sns.distplot(slide.obs['total_counts']\
                 [slide.obs['total_counts']<20000],
                 kde=False, bins=40, ax = axs[i, 1])
    axs[i, 1].set_xlim(0, 20000)
    axs[i, 1].set_xlabel(f'total_counts | {s}')
    
    sns.distplot(slide.obs['n_genes_by_counts'],
                 kde=False, bins=60, ax = axs[i, 2])
    axs[i, 2].set_xlim(0, adata.obs['n_genes_by_counts'].max())
    axs[i, 2].set_xlabel(f'n_genes_by_counts | {s}')
    
    sns.distplot(slide.obs['n_genes_by_counts']\
                 [slide.obs['n_genes_by_counts']<6000],
                 kde=False, bins=60, ax = axs[i, 3])
    axs[i, 3].set_xlim(0, 6000)
    axs[i, 3].set_xlabel(f'n_genes_by_counts | {s}')
    
plt.tight_layout()
```

--------------------------------

### Prepare AnnData for Regression Model

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Prepares an AnnData object for the regression model by specifying keys for batch, cell type, and covariates. Ensure 'Sample' is used for batch, 'Subset' for cell type, and 'Method' for categorical covariates.

```python
# prepare anndata for the regression model
cell2location.models.RegressionModel.setup_anndata(adata=adata_ref, 
                        # 10X reaction / sample / batch
                        batch_key='Sample', 
                        # cell type, covariate used for constructing signatures
                        labels_key='Subset', 
                        # multiplicative technical effects (platform, 3' vs 5', donor effect)
                        categorical_covariate_keys=['Method']
                       )
```

--------------------------------

### Download and Unzip Spatial and Reference Data

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo.ipynb

This snippet sets up directory paths and downloads Visium spatial data and reference snRNA-seq data. It ensures necessary directories exist before downloading and unzipping the data archives.

```python
# Set paths to data and results used through the document:
sp_data_folder = './data/mouse_brain_visium_wo_cloupe_data/'
results_folder = './results/mouse_brain_snrna/'

regression_model_output = 'RegressionGeneBackgroundCoverageTorch_65covariates_40532cells_12819genes'
reg_path = f'{results_folder}regression_model/{regression_model_output}/'

# Download and unzip spatial data
if os.path.exists('./data') is not True:
    os.mkdir('./data')
    os.system('cd ./data && wget https://cell2location.cog.sanger.ac.uk/tutorial/mouse_brain_visium_wo_cloupe_data.zip')
    os.system('cd ./data && unzip mouse_brain_visium_wo_cloupe_data.zip')

# Download and unzip snRNA-seq data with signatures of reference cell types 
# (if the output folder was not created by tutorial 1/3)
if os.path.exists(reg_path) is not True:
    os.mkdir('./results')
    os.mkdir(f'{results_folder}')
    os.mkdir(f'{results_folder}regression_model')
    os.mkdir(f'{reg_path}')
    os.system(f'cd {reg_path} && wget https://cell2location.cog.sanger.ac.uk/tutorial/mouse_brain_snrna/regression_model/RegressionGeneBackgroundCoverageTorch_65covariates_40532cells_12819genes/sc.h5ad')
```

--------------------------------

### Run Project Tests

Source: https://github.com/bayraktarlab/cell2location/blob/master/CLAUDE.md

Execute the pytest command to run all project tests, including model training smoke tests.

```bash
pytest
```

--------------------------------

### Prepare AnnData and Signatures

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Find shared genes between the annotated data and reference signatures, then subset both to include only these shared genes. This step is crucial for aligning the datasets before model training.

```python
# find shared genes and subset both anndata and reference signatures
intersect = np.intersect1d(adata_vis.var_names, inf_aver.index)
ata_vis = adata_vis[:, intersect].copy()
inf_aver = inf_aver.loc[intersect, :].copy()

# prepare anndata for cell2location model
cell2location.models.Cell2location.setup_anndata(adata=adata_vis, batch_key="sample")
```

--------------------------------

### Create and Train Cell2location Model

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA_Pyro.ipynb

Initializes and sets up the cell2location model with the prepared anndata object and reference cell type expression profiles. The 'detection_alpha' hyperparameter controls normalization of within-experiment variation.

```python
# create model
mod = cell2location.models.Cell2location_WTA(
    adata_wta, cell_state_df=inf_aver, 
    # hyperparameter controlling normalisation of
    # within-experiment variation in RNA detection:
    detection_alpha=20
) 
mod.view_anndata_setup()
```

--------------------------------

### Print Module Versions

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_downstream.ipynb

Iterates through loaded modules and prints their versions. This is useful for reproducibility and debugging.

```python
import sys
for module in sys.modules:
    try:
        print(module,sys.modules[module].__version__)
    except:
        try:
            if  type(sys.modules[module].version) is str:
                print(module,sys.modules[module].version)
            else:
                print(module,sys.modules[module].version())
        except:
            try:
                print(module,sys.modules[module].VERSION)
            except:
                pass

```

--------------------------------

### Configure Theano to Use GPU

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/commonerrors.md

Ensure Theano utilizes the GPU for faster training by setting the THEANO_FLAGS environment variable before importing cell2location. This is crucial for performance, especially on large datasets.

```python
# this line should go before importing cell2location
os.environ["THEANO_FLAGS"] = 'device=cuda,floatX=float32,force_device=True'
import cell2location
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/bayraktarlab/cell2location/blob/master/README.md

Use these commands to create a new conda environment named 'cell2loc_env' with Python 3.10 and activate it.

```bash
conda create -y -n cell2loc_env python=3.10
```

```bash
conda activate cell2loc_env
```

--------------------------------

### Load Visium Data

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Loads Visium spatial data using scanpy's datasets module. Assigns a sample ID to the observation metadata.

```python
adata_vis = sc.datasets.visium_sge(sample_id="V1_Human_Lymph_Node")
ada_vis.obs['sample'] = list(adata_vis.uns['spatial'].keys())[0]
```

--------------------------------

### Load Nanostring WTA Anndata Object

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA_Pyro.ipynb

Loads the Nanostring WTA anndata object from a specified H5AD file for further analysis.

```python
adata_wta = sc.read_h5ad('/nfs/team283/aa16/data/c2l_wta_tutorial_braindata.h5ad')
```

--------------------------------

### Run cell2location Training Pipeline

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo.ipynb

This snippet shows the complete pipeline for running cell2location, including setting up plotting parameters, defining input data, and configuring training and export arguments. Use this to train the model on your single-cell and spatial transcriptomics data.

```python
sc.settings.set_figure_params(dpi = 100, color_map = 'viridis', dpi_save = 100,
                              vector_friendly = True, format = 'pdf',
                              facecolor='white')

r = cell2location.run_cell2location(
    
      # Single cell reference signatures as pd.DataFrame 
      # (could also be data as anndata object for estimating signatures 
      #  as cluster average expression - `sc_data=adata_snrna_raw`)
      sc_data=inf_aver, 
      # Spatial data as anndata object
      sp_data=adata_vis,

      # the column in sc_data.obs that gives cluster idenitity of each cell
      summ_sc_data_args={'cluster_col': "annotation_1",
                        },

      train_args={'use_raw': True, # By default uses raw slots in both of the input datasets.
                  'n_iter': 40000, # Increase the number of iterations if needed (see QC below)
                  
                  # Whe analysing the data that contains multiple experiments, 
                  # cell2location automatically enters the mode which pools information across experiments
                  'sample_name_col': 'sample'}, # Column in sp_data.obs with experiment ID (see above)

    
      export_args={'path': results_folder, # path where to save results
                   'run_name_suffix': '' # optinal suffix to modify the name the run
                  },

      model_kwargs={ # Prior on the number of cells, cell types and co-located groups
          
                    'cell_number_prior': {
                        # - N - the expected number of cells per location:
                        'cells_per_spot': 8, # < - change this
                        # - A - the expected number of cell types per location (use default):
                        'factors_per_spot': 7, 
                        # - Y - the expected number of co-located cell type groups per location (use default):
                        'combs_per_spot': 7
                    },
          
                     # Prior beliefs on the sensitivity of spatial technology:
                    'gene_level_prior':{
                        # Prior on the mean
                        'mean': 1/2, 
                        # Prior on standard deviation,
                        # a good choice of this value should be at least 2 times lower that the mean
                        'sd': 1/4
                    }
      }
)

```

--------------------------------

### Load cell2location and supporting packages

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_estimating_signatures.ipynb

Imports necessary libraries for cell2location, scanpy, data manipulation, and plotting. It also sets the data type for computations and silences scanpy warnings.

```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import os

data_type = 'float32'

import cell2location

import matplotlib as mpl
from matplotlib import rcParams
import matplotlib.pyplot as plt
import seaborn as sns

# silence scanpy that prints a lot of warnings
import warnings
warnings.filterwarnings('ignore')
```

--------------------------------

### List Imported Modules and Versions

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb

Utility function to list all imported modules and their versions, useful for debugging and reporting issues.

```python
cell2location.utils.list_imported_modules()
```

--------------------------------

### Pull Docker Image

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/dockersingularity.md

Use this command to download the latest cell2location Docker image.

```none
docker pull quay.io/vitkl/cell2location
```

--------------------------------

### Submit Singularity Job with GPU

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/dockersingularity.md

Submits an LSF cluster job to run a Jupyter notebook within the Singularity container, requesting GPU resources. Adjust paths and queue names as needed.

```bash
bsub -q gpu_queue_name -M60000 \
  -R"select[mem>60000] rusage[mem=60000, ngpus_physical=1.00] span[hosts=1]"  \
  -gpu "mode=shared:j_exclusive=yes" -Is \
  /bin/singularity exec \
  --no-home  \
  --nv \
  -B /nfs/working_directory:/working_directory \
  path/to/cell2location-v0.06-alpha.sif \
  /bin/bash -c "cd /working_directory && HOME=$(mktemp -d) jupyter notebook --notebook-dir=/working_directory --NotebookApp.token='cell2loc' --ip=0.0.0.0 --port=1237 --no-browser --allow-root"
```

--------------------------------

### Scanpy Preprocessing Pipeline for Spatial Data

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_colab.ipynb

Applies standard scanpy preprocessing steps including log-transformation, finding highly variable genes per sample, scaling, PCA, KNN, and UMAP. Use this for initial data exploration and batch effect assessment.

```python
adata_vis_plt = adata.copy()

# Log-transform (log(data + 1))
sc.pp.log1p(adata_vis_plt)

# Find highly variable genes within each sample
adata_vis_plt.var['highly_variable'] = False
for s in adata_vis_plt.obs['sample'].unique():
    
    adata_vis_plt_1 = adata_vis_plt[adata_vis_plt.obs['sample'].isin([s]), :]
    sc.pp.highly_variable_genes(adata_vis_plt_1, min_mean=0.0125, max_mean=5, min_disp=0.5, n_top_genes=1000)
    
    hvg_list = list(adata_vis_plt_1.var_names[adata_vis_plt_1.var['highly_variable']])
    adata_vis_plt.var.loc[hvg_list, 'highly_variable'] = True
    
# Scale the data ( (data - mean) / sd )
sc.pp.scale(adata_vis_plt, max_value=10)
# PCA, KNN construction, UMAP
sc.tl.pca(adata_vis_plt, svd_solver='arpack', n_comps=40, use_highly_variable=True)
sc.pp.neighbors(adata_vis_plt, n_neighbors = 20, n_pcs = 40, metric='cosine')
sc.tl.umap(adata_vis_plt, min_dist = 0.3, spread = 1)

with mpl.rc_context({'figure.figsize': [8, 8],
                     'axes.facecolor': 'white'}):
    sc.pl.umap(adata_vis_plt, color=['sample'], size=30,
               color_map = 'RdPu', ncols = 1, #legend_loc='on data',
               legend_fontsize=10)
```

--------------------------------

### Download Cortical Layer Annotations

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_short_demo_colab.ipynb

Downloads CSV files containing manual annotations for cortical layers from a GitHub repository. These files are necessary for subsetting the spatial data to specific cortical regions.

```bash
#!/bin/bash
%%capture
! mkdir cortical_layers
! cd cortical_layers && wget https://raw.githubusercontent.com/vitkl/cell2location_paper/master/notebooks/selected_results/mouse_visium_snrna/manual_SSp_layers/SSp_ManLayerAnn_ST8059048.csv
! cd cortical_layers && wget https://raw.githubusercontent.com/vitkl/cell2location_paper/master/notebooks/selected_results/mouse_visium_snrna/manual_SSp_layers/SSp_ManLayerAnn_ST8059049.csv
! cd cortical_layers && wget https://raw.githubusercontent.com/vitkl/cell2location_paper/master/notebooks/selected_results/mouse_visium_snrna/manual_SSp_layers/SSp_ManLayerAnn_ST8059050.csv
! cd cortical_layers && wget https://raw.githubusercontent.com/vitkl/cell2location_paper/master/notebooks/selected_results/mouse_visium_snrna/manual_SSp_layers/SSp_ManLayerAnn_ST8059051.csv
! cd cortical_layers && wget https://raw.githubusercontent.com/vitkl/cell2location_paper/master/notebooks/selected_results/mouse_visium_snrna/manual_SSp_layers/SSp_ManLayerAnn_ST8059052.csv
```

--------------------------------

### Run Docker Container without GPU

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/dockersingularity.md

Executes the cell2location Docker container without utilizing GPU resources.

```none
docker run -i --rm -p 8848:8888 quay.io/vitkl/cell2location:latest
```

--------------------------------

### Plot training history

Source: https://github.com/bayraktarlab/cell2location/blob/master/docs/notebooks/cell2location_for_NanostringWTA_Pyro.ipynb

Visualizes the ELBO loss history during model training. The first 1000 epochs are removed from the plot to focus on the later stages of training.

```python
mod.plot_history(1000)
plt.legend(labels=['full data training']);
```