scArches

https://docs.scarches.org
Admin
scArches is a PyTorch-based tool for single-cell architecture surgery that enables integration of...
Tokens:101,256
Snippets:798
Trust Score:5.7
Update:10 months ago
Show doc for...
Tokens:
Raw
### Setup MuData for TOTALVI Model

Source: https://docs.scarches.org/en/latest/api/models

The `setup_mudata` classmethod prepares a MuData object for use with the TOTALVI model. It maps data fields to their locations within the MuData object and adds necessary fields for scVI processing. It does not modify existing data but adds setup information to `.uns` and encoded data to `.obs`.

```python
import muon
import scvi

mdata = muon.read_10x_h5("pbmc_10k_protein_v3_filtered_feature_bc_matrix.h5")
scvi.model.TOTALVI.setup_mudata(
    mdata, modalities={"rna_layer": "rna", "protein_layer": "prot"}
)
vae = scvi.model.TOTALVI(mdata)
```

--------------------------------

### Download Kang Dataset using gdown

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Downloads the Kang dataset from a Google Drive URL using the 'gdown' library. The downloaded file is saved locally as 'kang_tutorial.h5ad'. Ensure 'gdown' is installed (`pip install gdown`).

```python
import gdown

url = 'https://drive.google.com/uc?id=1t3oMuUfueUz_caLm5jmaEYjBxVNSsfxG'
output = 'kang_tutorial.h5ad'
gdown.download(url, output, quiet=False)

```

--------------------------------

### Install scArches from GitHub

Source: https://docs.scarches.org/en/latest/SageNet_mouse_embryo

Installs the scArches package directly from its GitHub repository. This is recommended for accessing the latest developments. It involves cloning the repository, changing the directory, and then installing the package using pip.

```bash
!git clone https://github.com/theislab/scarches
%cd scarches
!pip install .

```

--------------------------------

### Environment Setup and Import Libraries

Source: https://docs.scarches.org/en/latest/totalvi_surgery_pipeline

Sets up the working directory and imports necessary libraries for data manipulation, analysis, and visualization. It also suppresses specific warnings to keep the output clean.

```python
import os
os.chdir('../')
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)
```

--------------------------------

### Install mvTCR Package

Source: https://docs.scarches.org/en/latest/mvTCR_borcherding

Installs the mvTCR package using pip. This is a prerequisite for using the mvTCR functionalities described in the tutorial. It ensures all necessary dependencies are downloaded and installed in the current environment.

```python
%%capture
!pip install mvtcr

```

--------------------------------

### Initialize nbproject Header

Source: https://docs.scarches.org/en/latest/scanvi_surgery_pipeline

This snippet attempts to import and run the 'header' function from the 'nbproject' library. If 'nbproject' is not installed, it prints an informative message suggesting installation.

```python
try:
    from nbproject import header
    header()
except ModuleNotFoundError:
    print("If you want to see the header with dependencies, please install nbproject - pip install nbproject")

```

--------------------------------

### Install PyTorch Geometric and Dependencies

Source: https://docs.scarches.org/en/latest/SageNet_mouse_embryo

Installs PyTorch Geometric (PyG), which is used by SageNet for implementing graph neural networks. The installation commands are specific to the PyTorch and CUDA versions (e.g., 1.12.1+cu113), ensuring compatibility. It also installs related libraries like torch-scatter and torch-sparse.

```bash
!pip install -q torch-scatter -f https://data.pyg.org/whl/torch-1.12.1+cu113.html
!pip install -q torch-sparse -f https://data.pyg.org/whl/torch-1.12.1+cu113.html
!pip install -q git+https://github.com/pyg-team/pytorch_geometric.git

```

--------------------------------

### Setup AnnData

Source: https://docs.scarches.org/en/latest/api/models

Sets up the AnnData object for the model. This method creates a mapping between data fields used by the model and their locations in the AnnData object without modifying the original data.

```APIDOC
## POST /websites/scarches/setup_anndata

### Description
Sets up the AnnData object for this model. A mapping will be created between data fields used by this model to their respective locations in adata. None of the data in adata are modified. Only adds fields to adata.

### Method
POST

### Endpoint
/websites/scarches/setup_anndata

### Parameters
#### Request Body
- **adata** (AnnData) - Required - AnnData object. Rows represent cells, columns represent features.
- **layer** (str | None) - Optional - If not None, uses this as the key in adata.layers for raw count data.
- **batch_key** (str | None) - Optional - Key in adata.obs for batch information. Categories will automatically be converted into integer categories and saved to adata.obs['_scvi_batch']. If None, assigns the same batch to all the data.
- **labels_key** (str | None) - Optional - Key in adata.obs for label information. Categories will automatically be converted into integer categories and saved to adata.obs['_scvi_labels']. If None, assigns the same label to all the data.
- **size_factor_key** (str | None) - Optional - Key in adata.obs for size factor information. Instead of using library size as a size factor, the provided size factor column will be used as offset in the mean of the likelihood. Assumed to be on linear scale.
- **categorical_covariate_keys** (list[str] | None) - Optional - Keys in adata.obs that correspond to categorical data. These covariates can be added in addition to the batch covariate and are also treated as nuisance factors.
- **continuous_covariate_keys** (list[str] | None) - Optional - Keys in adata.obs that correspond to continuous data. These covariates can be added in addition to the batch covariate and are also treated as nuisance factors.

### Request Example
```json
{
  "adata": { ... AnnData object ... },
  "layer": "counts",
  "batch_key": "batch",
  "labels_key": "cell_type",
  "categorical_covariate_keys": ["treatment"],
  "continuous_covariate_keys": ["cell_cycle_phase"]
}
```

### Response
#### Success Response (200)
- **message** (str) - Indicates successful setup of AnnData.

#### Response Example
```json
{
  "message": "AnnData setup complete."
}
```
```

--------------------------------

### Import Libraries and Setup Environment

Source: https://docs.scarches.org/en/latest/scvi_surgery_pipeline

Imports necessary libraries like scanpy, torch, and scarches. It also includes warnings suppression and sets up the environment for plotting and tensor operations. This is a foundational step for running the rest of the pipeline.

```python
import os
os.chdir('../')
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)
```

```python
import scanpy as sc
import torch
import scarches as sca
from scarches.dataset.trvae.data_handling import remove_sparsity
import matplotlib.pyplot as plt
import numpy as np
import gdown
```

```python
sc.settings.set_figure_params(dpi=200, frameon=False)
sc.set_figure_params(dpi=200)
sc.set_figure_params(figsize=(4, 4))
torch.set_printoptions(precision=3, sci_mode=False, edgeitems=7)
```

--------------------------------

### Install Auxiliary Packages

Source: https://docs.scarches.org/en/latest/SageNet_mouse_embryo

Installs additional Python packages required for scArches functionality. This includes 'squidpy' for spatial data preprocessing, 'captum' for model interpretability (specifically for GNNs), and 'patchworklib' for combining matplotlib plots.

```bash
!pip install squidpy
!pip install captum
!pip install patchworklib

```

--------------------------------

### Setup AnnData for Model

Source: https://docs.scarches.org/en/latest/api/models

Prepares an AnnData object for use with the model by creating mappings to data fields. It does not modify the original AnnData object but adds necessary fields. Requires an AnnData object and keys for protein expression, batch, and optional covariates.

```python
_classmethod _setup_anndata(_adata : AnnData_, _protein_expression_obsm_key : str_, _protein_names_uns_key : str | None = None_, _batch_key : str | None = None_, _layer : str | None = None_, _size_factor_key : str | None = None_, _categorical_covariate_keys : list[str] | None = None_, _continuous_covariate_keys : list[str] | None = None_, _** kwargs_)
```

--------------------------------

### Setup and Train SCVI Model

Source: https://docs.scarches.org/en/latest/scvi_surgery_pipeline

Configures the SCVI model for the source dataset, specifying batch keys and other training parameters. It then trains the model using the prepared data. This step generates a latent representation of the reference dataset.

```python
sca.models.SCVI.setup_anndata(source_adata, batch_key=condition_key)
```

```python
vae = sca.models.SCVI(
    source_adata,
    n_layers=2,
    encode_covariates=True,
    deeply_inject_covariates=False,
    use_layer_norm="both",
    use_batch_norm="none",
)
```

```python
vae.train()
```

--------------------------------

### Load Reference CITE-seq PBMC Dataset

Source: https://docs.scarches.org/en/latest/totalvi_surgery_pipeline

Loads a reference CITE-seq PBMC dataset using scvi-tools. The `run_setup_anndata=False` argument indicates that initial setup steps are skipped, assuming the data is already in a suitable format or will be processed subsequently. This dataset serves as the baseline for integration.

```python
adata_ref = scv.data.pbmcs_10x_cite_seq(run_setup_anndata=False)
```

--------------------------------

### Setup Python Environment for scArches

Source: https://docs.scarches.org/en/latest/scgen_map_query

Imports necessary libraries and configures warning filters for the scArches project. It inserts the project root into the system path for module imports.

```python
import os
import sys
sys.path.insert(0, "../")

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)

```

--------------------------------

### MuData Setup API

Source: https://docs.scarches.org/en/latest/api/models

Sets up the MuData object for a scvi-tools model. This method maps data fields used by the model to their locations within the MuData object. It adds fields to the MuData object without modifying existing data.

```APIDOC
## CLASS METHOD _setup_mudata MuData

### Description
Sets up the `MuData` object for this model. A mapping will be created between data fields used by this model to their respective locations in adata. None of the data in adata are modified. Only adds fields to adata.

### Method
`classmethod`

### Endpoint
`/websites/scarches` (Conceptual endpoint for documentation structure)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **mdata** (MuData) - MuData object. Rows represent cells, columns represent features.
* **rna_layer** (str | None) - RNA layer key. If None, will use .X of specified modality key.
* **protein_layer** (str | None) - Protein layer key. If None, will use .X of specified modality key.
* **batch_key** (str | None) - key in adata.obs for batch information. Categories will automatically be converted into integer categories and saved to adata.obs[‘_scvi_batch’]. If None, assigns the same batch to all the data.
* **size_factor_key** (str | None) - key in adata.obs for size factor information. Instead of using library size as a size factor, the provided size factor column will be used as offset in the mean of the likelihood. Assumed to be on linear scale.
* **categorical_covariate_keys** (list[str] | None) - keys in adata.obs that correspond to categorical data. These covariates can be added in addition to the batch covariate and are also treated as nuisance factors (i.e., the model tries to minimize their effects on the latent space). Thus, these should not be used for biologically-relevant factors that you do _not_ want to correct for.
* **continuous_covariate_keys** (list[str] | None) - keys in adata.obs that correspond to continuous data. These covariates can be added in addition to the batch covariate and are also treated as nuisance factors (i.e., the model tries to minimize their effects on the latent space). Thus, these should not be used for biologically-relevant factors that you do _not_ want to correct for.
* **modalities** (dict[str, str] | None) - Dictionary mapping parameters to modalities.

### Request Example
```python
>>> mdata = muon.read_10x_h5("pbmc_10k_protein_v3_filtered_feature_bc_matrix.h5")
>>> scvi.model.TOTALVI.setup_mudata(
        mdata, modalities={"rna_layer": "rna", "protein_layer": "prot"}
    )
>>> vae = scvi.model.TOTALVI(mdata)
```

### Response
#### Success Response (200)
* **None** - Adds the following fields:
  * _.uns[‘_scvi’]_ – scvi setup dictionary
  * _.obs[‘_scvi_labels’]_ – labels encoded as integers
  * _.obs[‘_scvi_batch’]_ – batch encoded as integers

#### Response Example
None (modifies input MuData object in-place)
```

--------------------------------

### Setup AnnData for MultiVAE Model

Source: https://docs.scarches.org/en/latest/multigrate

Configures an AnnData object for use with the MultiVAE model. This involves specifying categorical covariates for batch correction and integration, and optionally setting RNA indices for size factor calculation if using raw counts. Inputs are the AnnData object and a list of covariate keys. Outputs are the setup AnnData object.

```python
import scarches as sca

# Assuming 'adata' is the organized AnnData object from the previous step
# and 'Modality' and 'Samplename' are columns in adata.obs
sca.models.MultiVAE.setup_anndata(
    adata,
    categorical_covariate_keys=['Modality', 'Samplename'],
    rna_indices_end=4000,
)
```

--------------------------------

### Setup AnnData for scArches

Source: https://docs.scarches.org/en/latest/api/models

Configures an AnnData object for use with a scArches model. This involves specifying the AnnData object and the key for protein expression in the `obsm` attribute. Essential for preparing input data.

```python
model.setup_anndata(adata=my_adata, protein_expression_obsm_key='protein_expression')

```

--------------------------------

### Setup AnnData for Model Training

Source: https://docs.scarches.org/en/latest/api/models

Configures an AnnData object for use with the model by creating mappings between data fields and their locations within the AnnData object. This function does not modify the original AnnData but adds necessary fields. It supports specifying layers for raw count data, batch keys, label keys, size factor keys, and covariate keys (both categorical and continuous).

```python
model._setup_anndata(
    _adata=adata,
    _layer='counts',
    _batch_key='batch',
    _labels_key='cell_type',
    _size_factor_key='size_factors',
    _categorical_covariate_keys=['patient'],
    _continuous_covariate_keys=['gene_expression']
)
```

--------------------------------

### Initialize MultiVAE Model

Source: https://docs.scarches.org/en/latest/multigrate

Initializes the MultiVAE model for multi-modal data integration. It takes the prepared AnnData object and specifies the loss functions for each modality, loss coefficients, and the covariate key for integration. Dependencies include the setup AnnData object and Scarches library. Outputs the initialized model.

```python
import scarches as sca

# Assuming 'adata' is the setup AnnData object
model = sca.models.MultiVAE(
    adata,
    losses=['nb', 'mse', 'mse'],
    loss_coefs={'kl': 1e-1,
               'integ': 3000,
               },
    integrate_on='Modality',
    mmd='marginal',
)
```

--------------------------------

### SCVI.setup_anndata()

Source: https://docs.scarches.org/en/latest/api/models

Sets up an AnnData object for scVI, performing necessary transformations and checks.

```APIDOC
## POST /websites/scarches/scvi/setup_anndata

### Description
Prepares an AnnData object for scVI model training by setting up necessary fields and performing basic checks.

### Method
POST

### Endpoint
/websites/scarches/scvi/setup_anndata

### Parameters
#### Query Parameters
- **adata** (AnnData) - Required - The AnnData object to setup.
- **batch_key** (str) - Optional - Key for batch information. If None, no batch correction is performed.
- **labels_key** (str) - Optional - Key for cell type or label information.
- **protein_key** (str) - Optional - Key for protein data.
- **layer** (str) - Optional - Layer in adata to use for counts. Defaults to 'counts'.
- ****kwargs** (dict) - Optional - Additional keyword arguments for setup.

### Response
#### Success Response (200)
- **message** (str) - Confirmation message of successful setup.

#### Response Example
```json
{
  "message": "AnnData object setup complete."
}
```
```

--------------------------------

### Download Reference Data (Python)

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Downloads a reference dataset ('pbmc_tutorial.h5ad') from a Google Drive URL using the gdown library. This is a prerequisite for subsequent analysis steps.

```python
url = 'https://drive.google.com/uc?id=1Rnm-XKEqPLdOq3lpa3ka2aV4bOXVCLP0'
output = 'pbmc_tutorial.h5ad'
gdown.download(url, output, quiet=False)
```

--------------------------------

### Initialize and Train expiMap Model in Python

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Initializes an expiMap model with specified architecture and training parameters, then trains it on the provided AnnData object. It includes settings for reconstruction loss, hidden layer sizes, and early stopping criteria. The training process monitors various loss metrics and saves the best performing state.

```python
intr_cvae = sca.models.EXPIMAP(
    adata=adata,
    condition_key='study',
    hidden_layer_sizes=[256, 256, 256],
    recon_loss='nb'
)

ALPHA = 0.7

early_stopping_kwargs = {
    "early_stopping_metric": "val_unweighted_loss", # val_unweighted_loss
    "threshold": 0,
    "patience": 50,
    "reduce_lr": True,
    "lr_patience": 13,
    "lr_factor": 0.1,
}
intr_cvae.train(
    n_epochs=400,
    alpha_epoch_anneal=100,
    alpha=ALPHA,
    alpha_kl=0.5,
    weight_decay=0.,
    early_stopping_kwargs=early_stopping_kwargs,
    use_early_stopping=True,
    monitor_only_val=False,
    seed=2020,
)
```

--------------------------------

### Initialize and Train scANVI Model in Python

Source: https://docs.scarches.org/en/latest/api/models

Demonstrates how to set up an AnnData object for scANVI, initialize the model, train it, and extract results. This process involves reading data, registering keys for batch and labels, creating the scANVI instance, training the model, and then obtaining the latent representation and predictions.

```python
>>> import anndata
>>> import scvi
>>> adata = anndata.read_h5ad(path_to_anndata)
>>> scvi.model.SCANVI.setup_anndata(adata, batch_key="batch", labels_key="labels")
>>> vae = scvi.model.SCANVI(adata, "Unknown")
>>> vae.train()
>>> adata.obsm["X_scVI"] = vae.get_latent_representation()
>>> adata.obs["pred_label"] = vae.predict()

```

--------------------------------

### Initialize and Load Query Data

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Loads query data for the EXPIMAP model. This involves specifying the dataset (kang) and the model configuration (intr_cvae). The output is a model object ready for training.

```python
q_intr_cvae = sca.models.EXPIMAP.load_query_data(kang, intr_cvae)
```

--------------------------------

### Initialize Environment and Import Libraries (Python)

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Initializes the Python environment by ignoring warnings and importing necessary libraries for data analysis and manipulation, including scanpy, torch, scarches, numpy, and gdown.

```python
import warnings
warnings.simplefilter(action='ignore')

import scanpy as sc
import torch
import scarches as sca
import numpy as np
import gdown
```

--------------------------------

### Load and Prepare Kang Dataset for Analysis

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Loads the downloaded 'kang_tutorial.h5ad' file into an AnnData object named 'kang'. It then selects variables matching 'adata' and copies the data. Subsequently, it assigns 'Kang' as the study and merges terms from 'adata.uns['terms']' into the 'kang' object for further analysis.

```python
import scanpy as sc

kang = sc.read('kang_tutorial.h5ad')[:, adata.var_names].copy()
kang.obs['study'] = 'Kang'
kang.uns['terms'] = adata.uns['terms']

```

--------------------------------

### Get Prototypes Info API

Source: https://docs.scarches.org/en/latest/api/models

Generate an AnnData file containing prototype features and their annotations. Allows specifying the prototype set to retrieve.

```APIDOC
## GET /websites/scarches/get_prototypes_info

### Description
Generates anndata file with prototype features and annotations.

### Method
GET

### Endpoint
`/websites/scarches/get_prototypes_info`

### Parameters
#### Query Parameters
- **prototype_set** (string) - Optional - Specifies the set of prototypes to retrieve. Allowed values: 'labeled', 'unlabeled'. Defaults to 'labeled'.

### Response
#### Success Response (200)
- **prototypes_info** (AnnData) - AnnData object containing prototype features and annotations.

#### Response Example
```json
{
  "prototypes_info": "<AnnData object representation>"
}
```
```

--------------------------------

### get_latent_library_size

Source: https://docs.scarches.org/en/latest/api/models

Retrieves the latent library size for each cell, denoted as ℓn in the totalVI paper. Users can choose to get the mean or a sample from the posterior distribution.

```APIDOC
## GET /websites/scarches/get_latent_library_size

### Description
Returns the latent library size for each cell.
This is denoted as ℓn in the totalVI paper.

### Method
GET

### Endpoint
/websites/scarches/get_latent_library_size

### Parameters
#### Query Parameters
- **adata** (AnnData | None) - AnnData object with equivalent structure to initial AnnData. If None, defaults to the AnnData object used to initialize the model.
- **indices** (Sequence[int] | None) - Indices of cells in adata to use. If None, all cells are used.
- **give_mean** (bool) - Return the mean or a sample from the posterior distribution.
- **batch_size** (int | None) - Minibatch size for data loading into model. Defaults to scvi.settings.batch_size.

### Response
#### Success Response (200)
- **np.ndarray** - Latent library sizes for each cell.

#### Response Example
```json
{
  "latent_library_size": "[NumPy array of latent library sizes]"
}
```
```

--------------------------------

### Train EXPIMAP Model

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Trains the initialized EXPIMAP model (q_intr_cvae) for query training. Key parameters include the number of epochs, learning rate annealing, weight decay, KL divergence weight, and early stopping criteria.

```python
q_intr_cvae.train(n_epochs=400, alpha_epoch_anneal=100, weight_decay=0., alpha_kl=0.1, seed=2020, use_early_stopping=True)
```

--------------------------------

### Get Conditional Embeddings API

Source: https://docs.scarches.org/en/latest/api/models

Retrieve the conditional embeddings from the model. These embeddings represent the model's understanding of the data conditioned on specific factors.

```APIDOC
## GET /websites/scarches/get_conditional_embeddings

### Description
Returns anndata object of the conditional embeddings.

### Method
GET

### Endpoint
`/websites/scarches/get_conditional_embeddings`

### Response
#### Success Response (200)
- **embeddings** (AnnData) - AnnData object containing the conditional embeddings.

#### Response Example
```json
{
  "embeddings": "<AnnData object representation>"
}
```
```

--------------------------------

### Get Protein Background Mean

Source: https://docs.scarches.org/en/latest/api/models

Retrieves the mean of the background component for protein expression. It takes an AnnData object, cell indices, and batch size as input.

```python
get_protein_background_mean(_adata_ , _indices_ , _batch_size_)
```

--------------------------------

### Initialize and Train SCVI Model (Python)

Source: https://docs.scarches.org/en/latest/api/models

This snippet demonstrates how to initialize and train the SCVI model using an AnnData object. It involves registering the AnnData, creating an SCVI instance, training the model, and retrieving the latent representation and normalized expression. Dependencies include the 'anndata' and 'scvi' libraries.

```python
>>> adata = anndata.read_h5ad(path_to_anndata)
>>> scvi.model.SCVI.setup_anndata(adata, batch_key="batch")
>>> vae = scvi.model.SCVI(adata)
>>> vae.train()
>>> adata.obsm["X_scVI"] = vae.get_latent_representation()
>>> adata.obsm["X_normalized_scVI"] = vae.get_normalized_expression()

```

--------------------------------

### Get Latent Space API

Source: https://docs.scarches.org/en/latest/api/models

Map input data to the latent space using the model's encoder network. This function can return either the mean or a random sample from the latent distribution.

```APIDOC
## POST /websites/scarches/get_latent

### Description
Map x in to the latent space using the encoder network.

### Method
POST

### Endpoint
`/websites/scarches/get_latent`

### Parameters
#### Request Body
- **x** (np.ndarray) - Required - Input data matrix.
- **c** (np.ndarray) - Required - Condition onehot matrix.
- **mean** (boolean) - Optional - Return mean instead of random sample from the latent space. Defaults to False.

### Request Example
```json
{
  "x": "<numpy array representation>",
  "c": "<onehot matrix representation>",
  "mean": true
}
```

### Response
#### Success Response (200)
- **latent_space** (np.ndarray) - Array containing latent space encoding of ‘x’.

#### Response Example
```json
{
  "latent_space": "<numpy array representation>"
}
```
```

--------------------------------

### Download and Prepare Query Dataset in Python

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Downloads a specified H5AD file from a Google Drive URL using the 'gdown' library and reads it into an AnnData object. It then subsets the data to match the reference dataset's variable names and adds/modifies observational annotations, including the 'study' key and 'terms' from the reference.

```python
import gdown
import scanpy as sc

url = 'https://drive.google.com/uc?id=1t3oMuUfueUz_caLm5jmaEYjBxVNSsfxG'
output = 'kang_tutorial.h5ad'
gdown.download(url, output, quiet=False)

kang = sc.read('kang_tutorial.h5ad')[:, adata.var_names].copy()
kang.obs['study'] = 'Kang'
kang.uns['terms'] = adata.uns['terms']
```

--------------------------------

### Get All Zenodo Deposition IDs

Source: https://docs.scarches.org/en/latest/api/zenodo

Retrieves a list of all deposition IDs associated with a Zenodo account. Requires a Zenodo access token. Returns a list of strings, where each string is a deposition ID.

```python
import scarches.zenodo.deposition

_access_token = "YOUR_ZENODO_ACCESS_TOKEN"

deposition_ids = scarches.zenodo.deposition.get_all_deposition_ids(_access_token)
print(f"Found deposition IDs: {deposition_ids}")
```

--------------------------------

### Configure and Train EXPIMAP Model with Early Stopping

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Configures and trains the initialized EXPIMAP model. The training utilizes an alpha annealing schedule and specifies early stopping parameters to prevent overfitting, monitoring validation unweighted loss with a patience of 50 epochs. It also includes parameters for learning rate adjustment and random seed.

```python
ALPHA = 0.7

early_stopping_kwargs = {
    "early_stopping_metric": "val_unweighted_loss",
    "threshold": 0,
    "patience": 50,
    "reduce_lr": True,
    "lr_patience": 13,
    "lr_factor": 0.1,
}
intr_cvae.train(
    n_epochs=400,
    alpha_epoch_anneal=100,
    alpha=ALPHA,
    alpha_kl=0.5,
    weight_decay=0.,
    early_stopping_kwargs=early_stopping_kwargs,
    use_early_stopping=True,
    seed=2020
)

```

--------------------------------

### Get Protein Foreground Probability

Source: https://docs.scarches.org/en/latest/api/models

Calculates the foreground probability for proteins, denoted as (1−πnt) in the totalVI paper. It supports conditioning on batches, subsetting genes, and controlling sample output. Dependencies include AnnData and sequence types.

```python
get_protein_foreground_probability(_adata : AnnData | None = None_, _indices : Sequence[int] | None = None_, _transform_batch : Sequence[Number | str] | None = None_, _protein_list : Sequence[str] | None = None_, _n_samples : int = 1_, _batch_size : int | None = None_, _return_mean : bool = True_, _return_numpy : bool | None = None_)
```

--------------------------------

### expiMap Model Initialization

Source: https://docs.scarches.org/en/latest/api/models

Documentation for initializing the expiMap model, a Conditional Variational Auto-encoder, within the scArches library. This includes a detailed list of parameters for configuring the model.

```APIDOC
## expiMap Model Class

### Description
Model for scArches class. This class contains the implementation of Conditional Variational Auto-encoder.

### Parameters

#### Required Parameters
- **adata** (AnnData) - Annotated data matrix. Must be count data for 'nb' and 'zinb' loss, and normalized log-transformed data for 'mse' loss.

#### Optional Parameters
- **condition_key** (String) - Column name of conditions in adata.obs dataframe.
- **conditions** (List) - List of condition names that the data will contain for correct encoding upon reloading.
- **hidden_layer_sizes** (List) - A list of hidden layer sizes for the encoder network. The decoder network will have the reversed order.
- **latent_dim** (Integer) - Bottleneck layer (z) size.
- **dr_rate** (Float) - Dropout rate applied to all layers. If `dr_rate` is 0, no dropout is applied.
- **recon_loss** (String) - Reconstruction loss method. Options: 'mse' or 'nb'.
- **use_l_encoder** (Boolean) - If True and `decoder_last_layer` is 'softmax', the library size encoder is used.
- **use_bn** (Boolean) - If True, batch normalization is applied to layers.
- **use_ln** (Boolean) - If True, layer normalization is applied to layers.
- **mask** (Array or List) - If not None, an array of 0s and 1s from `utils.add_annotations` to create a VAE with a masked linear decoder.
- **mask_key** (String) - A key in `adata.varm` for the mask if the mask is not provided.
- **decoder_last_layer** (String or None) - The last layer of the decoder. Must be 'softmax' (default for 'nb' loss), 'identity' (default for 'mse' loss), 'softplus', 'exp', or 'relu'.
- **soft_mask** (Boolean) - Use soft mask option. If True, the model enforces the mask with L1 regularization instead of multiplying the linear decoder's weight by the binary mask.
- **n_ext** (Integer) - Number of unconstrained extension terms. Used for query mapping.
- **n_ext_m** (Integer) - Number of constrained extension terms. Used for query mapping.
- **use_hsic** (Boolean) - If True, adds HSIC regularization for unconstrained extension terms. Used for query mapping.
- **hsic_one_vs_all** (Boolean) - If True, calculates the sum of HSIC losses for each unconstrained term versus the other terms. If False, calculates HSIC for all unconstrained terms versus the other terms. Used for query mapping.
- **ext_mask** (Array or List) - Mask (similar to the `mask` argument) for unconstrained extension terms. Used for query mapping.
- **soft_ext_mask** (Boolean) - Use the soft mask mode for training with the constrained extension terms. Used for query mapping.
```

--------------------------------

### Get Current Working Directory using os.getcwd

Source: https://docs.scarches.org/en/latest/scgen_map_query

This function returns the current working directory. It is a standard Python function and does not have external dependencies beyond the `os` module.

```python
import os
os.getcwd()
```

--------------------------------

### Configure Plotting and Torch Options (Python)

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Sets up plotting parameters for scanpy, such as disabling frames, setting DPI, and defining figure size. Also configures PyTorch tensor display options for clarity.

```python
sc.set_figure_params(frameon=False)
sc.set_figure_params(dpi=200)
sc.set_figure_params(figsize=(4, 4))
torch.set_printoptions(precision=3, sci_mode=False, edgeitems=7)
```

--------------------------------

### Define Terms for Removal

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Defines lists of Reactome pathway terms related to interferon signaling and B cell functions that will be removed from the reference dataset. This is a preparatory step for model extension node setup.

```python
rm_terms = ['INTERFERON_SIGNALING', 'INTERFERON_ALPHA_BETA_SIGNALIN',
            'CYTOKINE_SIGNALING_IN_IMMUNE_S', 'ANTIVIRAL_MECHANISM_BY_IFN_STI']
rm_terms += ['SIGNALING_BY_THE_B_CELL_RECEPT', 'MHC_CLASS_II_ANTIGEN_PRESENTAT']
```

--------------------------------

### Initialize Query Data Model with Extensions

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Initializes a query data model (EXPIMAP) for analyzing the 'kang' dataset. This function adds new unconstrained and constrained extension nodes to the model. It utilizes HSIC regularization for the unconstrained nodes to promote independence and allows for a soft mask on the constrained node. Dependencies include 'sca' and 'intr_cvae'.

```python
import sca

q_intr_cvae = sca.models.EXPIMAP.load_query_data(kang,
                                                 intr_cvae,
                                                 unfreeze_ext=True,
                                                 new_n_ext=3,
                                                 new_n_ext_m=1,
                                                 new_ext_mask=query_mask.T,
                                                 new_soft_ext_mask=True,
                                                 use_hsic=True,
                                                 hsic_one_vs_all=True
                                                )

```

--------------------------------

### Initialize and Train TOTALVI Model in Python

Source: https://docs.scarches.org/en/latest/api/models

This snippet demonstrates how to initialize and train the TOTALVI model using an AnnData object. It involves setting up the AnnData object with necessary keys for batch and protein expression, instantiating the TOTALVI model, training the model, and then retrieving the latent representation.

```python
import anndata
import scvi

# Load the AnnData object
adata = anndata.read_h5ad(path_to_anndata)

# Register AnnData with necessary keys for TOTALVI
scvi.model.TOTALVI.setup_anndata(
    adata, batch_key="batch", protein_expression_obsm_key="protein_expression"
)

# Instantiate the TOTALVI model
vae = scvi.model.TOTALVI(adata)

# Train the model
vae.train()

# Get the latent representation
adata.obsm["X_totalVI"] = vae.get_latent_representation()
```

--------------------------------

### Get Non-zero Terms (Python)

Source: https://docs.scarches.org/en/latest/api/models

The nonzero_terms function is a simple utility that likely returns a list or indicator of terms that have non-zero values or significance within the context of the Scarches analysis. Its specific output would depend on the internal data structures it operates on.

```python
nonzero_terms()[source]
    

```

--------------------------------

### Setup AnnData for MultiVAE model

Source: https://docs.scarches.org/en/latest/multigrate

Sets up an AnnData object for use with the MultiVAE model. It specifies categorical covariates and the end index for RNA data, preparing the data for downstream analysis and model training.

```python
sca.models.MultiVAE.setup_anndata(
    query,
    categorical_covariate_keys=['Modality', 'Samplename'],
    rna_indices_end=4000,
)

```

--------------------------------

### Load Pretrained Model and Prepare for Finetuning (Python)

Source: https://docs.scarches.org/en/latest/mvTCR_borcherding

Loads a pretrained model, adds new embedding vectors for query datasets, and freezes all weights except the conditional embedding layer. It then switches the model's adata to the hold-out dataset to prepare for finetuning on this new data.

```Python
# Load pretrained model
model = sca.models.mvTCR.utils_training.load_model(adata_train, f'saved_models/best_model_by_metric.pt', base_path='.')
model.add_new_embeddings(len(holdout_cohorts))  # add new cond embeddings
model.freeze_all_weights_except_cond_embeddings()
model.change_adata(adata_hold_out)  # change the adata to finetune on the holdout data
```

--------------------------------

### Set Up Environment and Warnings

Source: https://docs.scarches.org/en/latest/scanvi_surgery_pipeline

This code changes the current working directory to the parent directory and suppresses future and user warnings. This is often done to ensure consistent execution and a cleaner output.

```python
import os
os.chdir('../')
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)

```

--------------------------------

### Get latent representations for query and reference

Source: https://docs.scarches.org/en/latest/multigrate

Obtains the latent space representation for both the query and reference AnnData objects using the trained model. This is crucial for subsequent visualization and analysis steps like UMAP.

```python
q_model.get_latent_representation(adata=query)
q_model.get_latent_representation(adata=adata)

```

--------------------------------

### Import Core Libraries

Source: https://docs.scarches.org/en/latest/mvTCR_borcherding

Imports essential Python libraries for data analysis and machine learning, including PyTorch for deep learning, Scanpy for single-cell data analysis, and scikit-learn for machine learning models and metrics.

```python
import torch
import scanpy as sc
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score

```

--------------------------------

### Get and Save Latent Representation (Python)

Source: https://docs.scarches.org/en/latest/multigrate

This snippet retrieves the latent representation for all cells using `model.get_latent_representation()` and saves it to `adata.obsm['latent_ref']`. It ensures that the original latent representation is preserved before potential overwrites during fine-tuning.

```python
model.get_latent_representation()
adata.obsm['latent_ref'] = adata.obsm['latent'].copy()
adata
```

--------------------------------

### Get Latent Space Directions (Python)

Source: https://docs.scarches.org/en/latest/api/models

The latent_directions function calculates the directions of upregulation for each latent dimension. This is useful for ensuring that positive latent scores correspond to upregulation. It supports 'sum' or 'counts' methods for calculation and can optionally compute confidence intervals for the 'counts' method.

```python
latent_directions(_method ='sum'_, _get_confidence =False_, _adata =None_, _key_added ='directions'_)[source]
    
Get directions of upregulation for each latent dimension. Multipling this by raw latent scores ensures positive latent scores correspond to upregulation. 

Parameters:
    
  * **method** (_String_) – Method of calculation, it should be ‘sum’ or ‘counts’.
  * **get_confidence** (_Boolean_) – Only for method=’counts’. If ‘True’, also calculate confidence of the directions.
  * **adata** (_AnnData_) – An AnnData object to store dimensions. If ‘None’, self.adata is used.
  * **key_added** (_String_) – key of adata.uns where to put the dimensions.
```

--------------------------------

### Load AnnData Query Data

Source: https://docs.scarches.org/en/latest/hlca_map_classify

Loads an AnnData object from a specified .h5ad file. This is the initial step to get the query dataset into the analysis environment. It assumes the file path is correctly defined.

```python
adata_query_unprep = sc.read_h5ad(path_query_data)
```

--------------------------------

### Get Latent Representation (Python)

Source: https://docs.scarches.org/en/latest/mvTCR_borcherding

Generates the latent representation of the data using the loaded (finetuned) model. The function can return the mean of the latent representation and takes optional metadata for context. This representation is crucial for downstream visualization and analysis.

```Python
latent_adata = model.get_latent(adata, metadata=metadata, return_mean=True)
```

--------------------------------

### SCANVI.from_scvi_model()

Source: https://docs.scarches.org/en/latest/api/models

Initializes a scANVI model from a pre-trained scVI model.

```APIDOC
## POST /websites/scarches/scanvi/from_scvi_model

### Description
Creates a new scANVI model instance, inheriting the architecture and weights from an existing scVI model.

### Method
POST

### Endpoint
/websites/scarches/scanvi/from_scvi_model

### Parameters
#### Query Parameters
- **scvi_model** (SCVI) - Required - The pre-trained scVI model instance.
- **adata** (AnnData) - Required - The AnnData
```

--------------------------------

### Initialize EXPIMAP Reference Network

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Initializes an EXPIMAP model for reference data. This model uses a specified condition key and reconstruction loss, with configurable hidden layer sizes. It's designed for gene expression data, indicated by the 'nb' (Negative Binomial) reconstruction loss.

```python
intr_cvae = sca.models.EXPIMAP(
    adata=adata,
    condition_key='study',
    hidden_layer_sizes=[300, 300, 300],
    recon_loss='nb'
)

```

--------------------------------

### Save Trained Model

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Saves the trained EXPIMAP model to a specified file. This allows for later loading and use of the model without retraining.

```python
q_intr_cvae.save('query_kang_tutorial')
```

--------------------------------

### Get Genes from Extension Node (Unconstrained 1)

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Retrieves genes associated with the 'unconstrained_1' extension node, sorted by their weights. This function helps identify genes most affected by this specific latent dimension and provides their corresponding weights and mask information.

```python
q_intr_cvae.term_genes('unconstrained_1', terms=kang_pbmc.uns['terms'])
```

--------------------------------

### Initialize and Train SageNet Model

Source: https://docs.scarches.org/en/latest/SageNet_mouse_embryo

Initializes the SageNet model and then trains it using the reference AnnData object. The training process uses specified observation columns for partitioning and computes feature importances. The trained model is associated with a tag 'seqFISH_ref1'.

```python
sg_obj = sca.models.sagenet(device=device)
import torch_geometric.data as geo_dt
sg_obj.train(adata_seqFISH1_1, comm_columns=['leiden_0.05', 'leiden_0.1', 'leiden_0.5'], tag='seqFISH_ref1', epochs=15, verbose = False, importance=True)
```

--------------------------------

### Get Genes from Extension Node (Unconstrained 2)

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_advanced

Retrieves genes associated with the 'unconstrained_2' extension node, sorted by their weights. This function is used to identify genes most influenced by this latent variable, outputting gene names, weights, and mask inclusion status.

```python
q_intr_cvae.term_genes('unconstrained_2', terms=kang_pbmc.uns['terms'])
```

--------------------------------

### Prepare Query AnnData for Integration

Source: https://docs.scarches.org/en/latest/api/models

Prepares an AnnData object for integration with a reference scArches model. This step is crucial for query datasets that will be mapped onto a pre-existing reference.

```python
prepared_adata = prepare_query_anndata(adata=query_adata, reference_model=reference_model)

```

--------------------------------

### Setup AnnData for SCVI Model

Source: https://docs.scarches.org/en/latest/reference_building_from_scratch

Prepares an AnnData object for SCVI or SCANVI models by registering count data, batch keys, and other covariates. Ensures data is correctly formatted and registered before model training. Assumes data is in adata.X and specifies batch_key.

```python
sca.models.SCVI.setup_anndata(source_adata, batch_key="batch")
```

--------------------------------

### Setup AnnData for SCVI Model

Source: https://docs.scarches.org/en/latest/scanvi_surgery_pipeline

Registers the 'source_adata' anndata object for use with SCVI models. It specifies the batch and label keys, ensuring that the model can correctly interpret the data structure. Crucially, it indicates that the anndata object should not be modified further until the model is trained.

```python
sca.models.SCVI.setup_anndata(source_adata, batch_key=condition_key, labels_key=cell_type_key)

```

--------------------------------

### Extract and Prepare Conditional Embeddings with scPoli

Source: https://docs.scarches.org/en/latest/_sources/scpoli_ATAC.ipynb

Retrieves conditional embeddings from a scPoli model and prepares the observation metadata. This involves getting the embeddings and then aligning the observation data with the sample names from the original AnnData object. Dependencies include scPoli and Scanpy libraries.

```python
import scanpy as sc
# Assuming 'scpoli_model' is your trained scPoli model and 'adata' is your original AnnData object
adata_emb = scpoli_model.get_conditional_embeddings()
ata_emb.obs = adata.obs.groupby('Samplename').first().reindex(adata_emb.obs.index)
```

--------------------------------

### Configure scPoli Model Training Parameters

Source: https://docs.scarches.org/en/latest/scpoli_surgery_pipeline

This code block defines the configuration parameters for training an scPoli model. It includes settings for early stopping, the key for conditions ('study'), the key for cell types ('cell_type'), and lists of reference and query datasets for integration and transfer learning.

```python
early_stopping_kwargs = {
    "early_stopping_metric": "val_prototype_loss",
    "mode": "min",
    "threshold": 0,
    "patience": 20,
    "reduce_lr": True,
    "lr_patience": 13,
    "lr_factor": 0.1,
}

condition_key = 'study'
cell_type_key = 'cell_type'
reference = [
    'inDrop1',
    'inDrop2',
    'inDrop3',
    'inDrop4',
    'fluidigmc1',
    'smartseq2',
    'smarter'
]
query = ['celseq', 'celseq2']
```

--------------------------------

### Generate and Visualize Latent Representation (UMAP)

Source: https://docs.scarches.org/en/latest/reference_building_from_scratch

Extracts the latent representation from the trained SCVI model and prepares it for visualization. Uses UMAP for dimensionality reduction and Leiden clustering for cell communities. Visualizes the UMAP plot colored by batch and cell type. Requires scvi-tools and scanpy to be installed.

```python
reference_latent = sc.AnnData(vae.get_latent_representation())
reference_latent.obs["cell_type"] = source_adata.obs["final_annotation"].tolist()
reference_latent.obs["batch"] = source_adata.obs["batch"].tolist()

sc.pp.neighbors(reference_latent, n_neighbors=8)
sc.tl.leiden(reference_latent)
sc.tl.umap(reference_latent)
sc.pl.umap(reference_latent,
           color=['batch', 'cell_type'],
           frameon=False,
           wspace=0.6,
           )
```

--------------------------------

### Load and Prepare AnnData Object (Python)

Source: https://docs.scarches.org/en/latest/expimap_surgery_pipeline_basic

Loads the downloaded reference dataset into an AnnData object using scanpy. It then replaces the default data matrix in `adata.X` with the raw counts from `adata.layers['counts']`.

```python
adata = sc.read('pbmc_tutorial.h5ad')
adata.X = adata.layers["counts"].copy()
```
Tokens:
Raw