### Initial Setup

Source: https://github.com/biomap-research/scfoundation/blob/main/model/check_consistency.ipynb

Imports necessary libraries and sets up the plotting environment. Note: %pylab is deprecated.

```python
%pylab inline
import numpy as np
```

--------------------------------

### Load Example Embedding

Source: https://github.com/biomap-research/scfoundation/blob/main/model/check_consistency.ipynb

Loads an example embedding for comparison.

```python
demoa5 = np.load('./examples/enhancement/Baron_01B-resolution_singlecell_cell_embedding_a5_resolution.npy')
```

--------------------------------

### Initial Setup and Imports

Source: https://github.com/biomap-research/scfoundation/blob/main/GEARS/Plot.ipynb

Imports necessary libraries and sets up the plotting environment.

```python
%pylab inline
import pickle
```

```python
import os
os.getpid()
```

```python
from gears import PertData, GEARS
```

```python
from os.path import join as pjoin
```

```python
from gears.inference import evaluate, compute_metrics, deeper_analysis, non_dropout_analysis, compute_synergy_loss,GI_subgroup
```

```python
import pickle
with open('Norman_results/baseline_testpred.pkl','wb') as f:
    pickle.dump([test_res,test_metrics,test_pert_res],f)
with open('Norman_results/scFoundation_testpred.pkl','wb') as f:
    pickle.dump([test_res_pt,test_metrics_pt,test_pert_res_pt],f)
```

```python
import pickle
with open('Norman_results/baseline_testpred.pkl','rb') as f:
    [test_res,test_metrics,test_pert_res] = pickle.load(f)
with open('Norman_results/scFoundation_testpred.pkl','rb') as f:
    [test_res_pt,test_metrics_pt,test_pert_res_pt] = pickle.load(f)
```

--------------------------------

### Initial Imports and Setup

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/plot-publish.ipynb

Imports necessary libraries for plotting, data manipulation, and machine learning, and sets up the plotting environment.

```python
%pylab inline
from sklearn.metrics import *
import pandas as pd
import sklearn.preprocessing as sk
import os
import sys
```

--------------------------------

### Initial Imports and Setup

Source: https://github.com/biomap-research/scfoundation/blob/main/ablation/ablation-00.ipynb

Imports necessary libraries and sets up the plotting environment.

```python
%pylab inline
import pandas as pd
import scanpy as sc
```

--------------------------------

### Initial Setup and Imports

Source: https://github.com/biomap-research/scfoundation/blob/main/ablation/ablation-02.ipynb

Initializes the environment and imports necessary libraries like scanpy and pandas. It also notes the deprecation of %pylab.

```python
%pylab inline
import scanpy as sc
import pandas as pd
```

--------------------------------

### Cell Embedding Inference Example

Source: https://github.com/biomap-research/scfoundation/blob/main/apiexample/README.md

Bash script example for inferring cell embeddings using the scFoundation/xTrimoGene API.

```bash
### Cell embedding
taskname=Baron
tgthighres=a5
mkdir -p ./demo/${taskname}/${tgthighres}
python ./client.py --input_type singlecell --output_type cell --pool_type all --pre_normalized F --version 0.2 --tgthighres $tgthighres --data_path ./data/baron_human_samp_19264_fromsaver_demo.csv --save_path ./demo/${taskname}/${tgthighres}/
```

--------------------------------

### Inference Example (Bash)

Source: https://github.com/biomap-research/scfoundation/blob/main/model/README.md

Example bash command for inferring cell embeddings using the scFoundation model.

```bash
### Cell embedding
python get_embedding.py --task_name Baron --input_type singlecell --output_type cell --pool_type all --tgthighres a5 --data_path ./examples/enhancement/Baron_enhancement.csv --save_path ./examples/enhancement/ --pre_normalized F --version rde
```

--------------------------------

### Example Usage

Source: https://github.com/biomap-research/scfoundation/blob/main/genemodule/plot_geneemb.ipynb

Example of how to load an AnnData object, preprocess it by normalizing total counts and log1p transformation, and then use the score_metagenes function.

```python
adata = sc.read_h5ad('./data/zheng_downsampled_cd8t_b_mono.h5ad')

sc.pp.normalize_total(adata)
sc.pp.log1p(adata)

score_metagenes(adata,mgs)
```

--------------------------------

### Zheng68K Dataset Embeddings

Source: https://github.com/biomap-research/scfoundation/blob/main/model/check_consistency.ipynb

Example of embeddings generated from the Zheng68K dataset.

```python
array([[-1.6754574 , -0.9752933 ],
        ...,
        [-1.0960114 ,  0.5838227 , -1.199613  , ..., -0.0532345 ,
         -1.2262417 , -1.4325197 ],
        [-1.1670277 , -0.30191293, -1.3319362 , ..., -0.00449257,
         -1.5309587 , -1.1473409 ],
        [-1.3597002 ,  0.04165971, -0.7901947 , ..., -0.09515837,
         -1.3980008 , -1.5304178 ]], dtype=float32),
array([[-0.9632143 ,  0.19995748, -1.0301558 , ...,  0.1359183 ,
         -1.3282764 , -1.3091614 ],
        [-0.716611  ,  0.26273462, -0.9527265 , ..., -0.07826823,
         -1.254329  , -1.1973946 ],
        [-0.75764936,  0.01225338, -1.3379053 , ...,  0.16142447,
         -1.6754574 , -0.9752933 ],
        ...,
        [-1.0960114 ,  0.5838227 , -1.199613  , ..., -0.0532345 ,
         -1.2262417 , -1.4325197 ],
        [-1.1670277 , -0.30191293, -1.3319362 , ..., -0.00449257,
         -1.5309587 , -1.1473409 ],
        [-1.3597002 ,  0.04165971, -0.7901947 , ..., -0.09515837,
         -1.3980008 , -1.5304178 ]], dtype=float32))
```

--------------------------------

### Demo Usage Commands

Source: https://github.com/biomap-research/scfoundation/blob/main/GEARS/README.md

Commands to run different demo scenarios for GEARS.

```bash
# cd to the GEARS folder
## no embedding
bash run_sh/run_singlecell_maeautobin-demo-baseline.sh

## using embedding from API call 
bash run_sh/run_singlecell_maeautobin-demo-emb.sh

## using embedding from local model
bash run_sh/run_singlecell_maeautobin-demo-train.sh
```

--------------------------------

### Get Number of Clusters

Source: https://github.com/biomap-research/scfoundation/blob/main/ablation/ablation-00.ipynb

Retrieves the number of unique clusters found.

```python
numcls = refAdata.obs['leiden'].unique().shape[0]
numcls
```

--------------------------------

### Get shapes of embeddings

Source: https://github.com/biomap-research/scfoundation/blob/main/model/check_consistency.ipynb

Prints the shapes of the loaded training and inference embeddings.

```python
trainemb.shape,inferemb.shape
```

--------------------------------

### Import scib

Source: https://github.com/biomap-research/scfoundation/blob/main/mapping/mapping-publish.ipynb

Imports the scib library for calculating integration metrics.

```python
import scib
```

--------------------------------

### Apply Basic Filtering and QC Metrics

Source: https://github.com/biomap-research/scfoundation/blob/main/preprocessing/demo.ipynb

Applies basic filtering to the AnnData object and calculates QC metrics.

```python
adata_uni = BasicFilter(adata_uni,qc_min_genes=200,qc_min_cells=0) # filter cell and gene by lower limit
ada_uni = QC_Metrics_info(adata_uni)
```

--------------------------------

### Get Metadata Shapes

Source: https://github.com/biomap-research/scfoundation/blob/main/DeepCDR/plot.ipynb

Prints the shapes of the created metadata DataFrames to understand their dimensions.

```python
metadata_test.shape
```

```python
metadata_train.shape
```

```python
metadata_all.Cancer_type.value_counts().shape
```

```python
metadata_all.CellLine.value_counts().shape
```

```python
metadata_all.pubchem_id.value_counts().shape
```

```python
metadata_all.shape
```

--------------------------------

### Execute Shell Script

Source: https://github.com/biomap-research/scfoundation/blob/main/preprocessing/demo.ipynb

Executes a shell script named 'demo.sh'.

```python
! bash demo.sh
```

--------------------------------

### Load 10x Data and Set Figure Directory

Source: https://github.com/biomap-research/scfoundation/blob/main/preprocessing/demo.ipynb

Loads data from a 10x Genomics directory and sets the figure directory for plots.

```python
from scRNA_workflow import *
sc.settings.figdir='./figures_new/' # set figure folder

path = './GSM4653863_HC1/'
ada = sc.read_10x_mtx(path) # read from 10x file
```

--------------------------------

### Model Initialization and Loading

Source: https://github.com/biomap-research/scfoundation/blob/main/GEARS/Plot.ipynb

Initializes GEARS models and loads pre-trained weights.

```python
gears_model = GEARS(pert_data, device = 'cuda:1')
modelPath = '/nfs_beijing/minsheng/scbig/bioinfoDownStream/gears_singlecell/results/gse133344_k562gi_oe_pert227_84986_19264_withtotalcount/0.75/split_simulation_seed_1_hidden_128_epochs_15_batch_30/2023-03-21_20-58-58'
gears_model.load_pretrained(modelPath)
```

```python
pretrain_model = GEARS(pert_data, device = 'cuda:1')
modelPath = '/nfs_beijing/minsheng/scbig/bioinfoDownStream/gears_singlecell/results/gse133344_k562gi_oe_pert227_84986_19264_withtotalcount/0.75/50m-0.1B_split_simulation_seed_1_hidden_512_bin_autobin_resolution_append_singlecell_maeautobin_finetune_frozen_epochs_15_batch_6_accmu_5_mode_v1_highres_0_lr_0.01/2023-03-27_15-46-01/'
pretrain_model.load_pretrained(modelPath)
```

--------------------------------

### Get Unique Identifiers

Source: https://github.com/biomap-research/scfoundation/blob/main/DeepCDR/plot.ipynb

Extracts lists of unique cancer types, drug identifiers, and cell lines from the metadata.

```python
allcancer = metadata_test.Cancer_type.unique().tolist()
alldrug = metadata_test.pubchem_id.unique().tolist()
allcellline = metadata_test.CellLine.unique().tolist()
```

--------------------------------

### Required Libraries

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/README.md

List of Python libraries required for the project.

```text
scanpy
pandas
pytorch
scikit-learn
```

--------------------------------

### Gene Symbol Conversion Function

Source: https://github.com/biomap-research/scfoundation/blob/main/apiexample/README.md

Python function to convert gene symbols in a DataFrame to match a specified list.

```python
X_df represents your single cell data with cells in rows and genes in columns
gene_list_df = pd.read_csv('../OS_scRNA_gene_index.19264.tsv', header=0, delimiter='\t')
gene_list = list(gene_list_df['gene_name'])
X_df, to_fill_columns, var = main_gene_selection(X_df, gene_list)
```

--------------------------------

### SCAD Training Demo

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/README.md

Commands to run the SCAD training script for Sorafenib, both without and with embeddings.

```bash
# cd to the SCAD folder
# Sorafenib
## without embedding
CUDA_VISIBLE_DEVICES=1 python model/SCAD_train_binarized_5folds-pub.py -e FX -d Sorafenib -g _norm -s 42 -h_dim 512 -z_dim 128 -ep 20 -la1 5 -mbS 8 -mbT 8 -emb 0

## with embedding
CUDA_VISIBLE_DEVICES=1 python model/SCAD_train_binarized_5folds-pub.py -e FX -d Sorafenib -g _norm -s 42 -h_dim 1024 -z_dim 256 -ep 80 -la1 0.2 -mbS 32 -mbT 32 -emb 1
```

--------------------------------

### Demo Usage Commands

Source: https://github.com/biomap-research/scfoundation/blob/main/DeepCDR/README.md

Commands to train the baseline and embedding-based DeepCDR models.

```bash
# cd to the DeepCDR folder
mkdir log
mkdir checkpoint
cd ./prog/
## baseline model
CUDA_VISIBLE_DEVICES=0 python run_DeepCDR.py -use_gexp > ../log/Base_rep1.log 2>&1
## embedding based model
CUDA_VISIBLE_DEVICES=0 python run_DeepCDR.py --ckpt_name 50M-0.1B-res -use_gexp > ../log/50M-0.1B-res_rep1.log 2>&1

```

--------------------------------

### Define Function to Get Metagenes

Source: https://github.com/biomap-research/scfoundation/blob/main/genemodule/plot_geneemb.ipynb

Defines a function to extract metagenes (gene programs) from Leiden clusters. Modified from scGPT GitHub repository.

```python
# The function is modified from https://github.com/bowang-lab/scGPT
import collections
def get_metagenes(gdata):
    metagenes = collections.defaultdict(list)
    for x, y in zip(gdata.obs["leiden"], gdata.obs.index):
        metagenes[x].append(y)
    return metagenes

metagenes = get_metagenes(gene_adata)

# Obtain the set of gene programs from clusters with #genes >= 5
mgs = dict()
for mg, genes in metagenes.items():
    if len(genes) > 4:
        mgs[mg] = genes
```

--------------------------------

### Expected Output Files

Source: https://github.com/biomap-research/scfoundation/blob/main/GEARS/README.md

List of files expected in the results folder after running experiments.

```text
config.pkl
model.pt
params.csv
train.log
```

--------------------------------

### Compare Embeddings

Source: https://github.com/biomap-research/scfoundation/blob/main/model/check_consistency.ipynb

Compares the loaded high-resolution target embedding and the example embedding. The results show thousandth-place accuracy, indicating consistency across different computational clusters.

```python
tgtres5,demoa5 # We used different computational clusters to test. As you can see, the results guaranteed thousandth-place accuracy
```

--------------------------------

### Load Data from CSV (Gzipped)

Source: https://github.com/biomap-research/scfoundation/blob/main/preprocessing/demo.ipynb

Loads data from a gzipped CSV file into an AnnData object.

```python
path = './GSM4914711/data.csv.gz'
ada = pd.read_csv(path,index_col=0) # read from csv file
```

--------------------------------

### Initializing dictionaries for silhouette scores

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/plot-publish.ipynb

Initializes empty dictionaries to store silhouette scores for single-cell and bulk data.

```python
sil_sc={}
sil_bulk={}
```

--------------------------------

### Loading and comparing Zheng68K embeddings

Source: https://github.com/biomap-research/scfoundation/blob/main/model/check_consistency.ipynb

Loads original and demo embeddings for the Zheng68K dataset and displays the first 10 elements and the entire demo embedding.

```python
orizhengemb = np.load('../enhancement/pbmc68ksorted_count_50M-0.1B-res_embedding.npy')
demozheng = np.load('./examples/enhancement/Zheng68K_01B-resolution_singlecell_cell_embedding_f1_resolution.npy')
```

```python
orizhengemb[:10],demozheng
```

--------------------------------

### Download Dataset

Source: https://github.com/biomap-research/scfoundation/blob/main/preprocessing/README.md

Script to download raw data files. The demo usage is provided in demo.sh.

```bash
bash demo.sh
```

--------------------------------

### Preparing Target Metadata

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/plot-publish.ipynb

Copies target metadata, adds a 'response' column from target expression data, and converts 'EpiSen_score' to float.

```python
tgtmeta = cellmeta.loc[targetexp.index,:].copy()
tgtmeta['response'] = pd.DataFrame(targetexp.iloc[:,0])
tgtmeta['EpiSen_score'] = tgtmeta['EpiSen_score'].astype(float)
```

--------------------------------

### Save AnnData Object (Second Dataset)

Source: https://github.com/biomap-research/scfoundation/blob/main/preprocessing/demo.ipynb

Saves the processed AnnData object for the second dataset to a .h5ad file.

```python
save_path = './GSM4914711/demo.h5ad'
save_adata_h5ad(adata_uni,save_path)
```

--------------------------------

### Additional Imports

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/plot-publish.ipynb

Imports additional scientific computing and data visualization libraries.

```python
import scipy.stats as ss
import scanpy as sc
import seaborn as sns
import colorbm as cbm
import scipy.stats
import sklearn
sns.set_palette(sns.color_palette(cbm.pal('npg').as_hex))
```

--------------------------------

### Loading Raw Prediction Results

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/plot-publish.ipynb

Reads raw prediction results from a tab-separated file.

```python
rawpredresults = pd.read_csv(Raw_SCresult[1],sep='\t',header=None,index_col=1)
```

--------------------------------

### Save AnnData Object

Source: https://github.com/biomap-research/scfoundation/blob/main/preprocessing/demo.ipynb

Saves the processed AnnData object to a .h5ad file.

```python
save_path = './GSM4653863_HC1/demo.h5ad'
save_adata_h5ad(adata_uni,save_path)
```

--------------------------------

### Running enhancement script

Source: https://github.com/biomap-research/scfoundation/blob/main/README.md

This command is used to obtain the results of scFoundation for the read depth enhancement task.

```bash
bash enhancement/run.sh
```

--------------------------------

### Batch correction and UMAP

Source: https://github.com/biomap-research/scfoundation/blob/main/mapping/mapping-publish.ipynb

Applies BBKNN for batch correction and then performs Uniform Manifold Approximation and Projection (UMAP) for visualization.

```python
sc.external.pp.bbknn(merged, batch_key='batch_id')  # running bbknn 1.3.6
sc.tl.umap(merged)
```

--------------------------------

### Data Loading and Preparation

Source: https://github.com/biomap-research/scfoundation/blob/main/GEARS/Plot.ipynb

Loads perturbation data and prepares it for model training and evaluation.

```python
pert_data = PertData('/nfs_beijing/minsheng/scbig/bioinfoDownStream/gears_singlecell/data/')
pert_data.load(data_path = pjoin('/nfs_beijing/minsheng/scbig/bioinfoDownStream/gears_singlecell/data/', 'gse133344_k562gi_oe_pert227_84986_19264_withtotalcount'))
pert_data.prepare_split(split = 'simulation', seed = 1, train_gene_set_size=0.75)
pert_data.get_dataloader(batch_size = 6, test_batch_size = 6)
```

--------------------------------

### Loading Target and Source Embeddings

Source: https://github.com/biomap-research/scfoundation/blob/main/SCAD/plot-publish.ipynb

Loads pre-computed embeddings for target and source data using NumPy.

```python
dname = 'NVP-TAE684'
tgtemb = np.load(f'data/split_norm/Target_expr_resp_19264.{dname}_50M-0.1B-res_tgthighres4_embedding.npy')
srcemb = np.load(f'data/split_norm/Source_exprs_resp_19264.{dname}_50M-0.1B-res_embedding.npy')
```

--------------------------------

### Load and integrate scBERT embedding

Source: https://github.com/biomap-research/scfoundation/blob/main/mapping/mapping-publish.ipynb

Loads the rawmerged AnnData object, adds the scBERT embedding to the .obsm attribute, and runs BBKNN integration.

```python
merged = sc.read_h5ad('rawmerged.h5ad')
merged.obsm['scb']=scbemb
scbadata = merged.copy()
sc.external.pp.bbknn(scbadata, batch_key='batch_id',use_rep='scb',n_pcs=scbemb.shape[1])  # running bbknn 1.3.6
sc.tl.umap(scbadata)
```

--------------------------------

### Generate Venn Diagram for SYNERGY (Alternative)

Source: https://github.com/biomap-research/scfoundation/blob/main/GEARS/Plot.ipynb

Creates a 3-set Venn diagram comparing scEPT, Baseline, and Ground Truth for the SYNERGY ptype, with different color settings.

```python
gtset = set(aucdf[aucdf.ptype=='SYNERGY'].variable.tolist())
fisize(7,7)

venn3(subsets=[set(t1.variable.tolist()[:N]),set(t2.variable.tolist()[:N]),gtset],
     set_colors=('#C44E52','#55A868','#4C72B0'),
     # set_colors=cbm.pal('npg').as_hex,
     alpha=0.5)

c=venn3_circles(subsets =[set(t1.variable.tolist()[:N]),set(t2.variable.tolist()[:N]),gtset], linestyle='--', linewidth=1, color="black")
c[2].set_ls('-')
# plt.savefig(f'Venn_SYNERGY.pdf',bbox_inches='tight')
```

--------------------------------

### Import Venn Diagram Library

Source: https://github.com/biomap-research/scfoundation/blob/main/GEARS/Plot.ipynb

Imports the necessary functions for creating Venn diagrams.

```python
from matplotlib_venn import venn3,venn3_circles
```