### Set Up Environment with Conda

Source: https://github.com/patchouli-m/sequencingcancerfinder/blob/master/README.md

Instructions for creating and activating a Conda environment for Cancer-Finder, specifying Python version and installing dependencies via pip.

```bash
conda create -n scf python==3.9.16
conda activate scf
pip install -r requirements.txt
```

--------------------------------

### Environment Setup using Conda and Pip (Bash)

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Provides commands to set up the necessary Python environment for the Cancer Finder project using Conda and Pip. It includes creating a new Conda environment, activating it, and installing dependencies from a requirements file.

```bash
# Create conda environment
conda create -n scf python==3.9.16
conda activate scf

# Install dependencies
pip install -r requirements.txt

# Requirements:
# anndata==0.9.1
# numpy==1.23.0
# pandas==1.5.2
# scanpy==1.9.3
# torch==1.13.1
# torchvision==0.14.1
# zarr
```

--------------------------------

### Advanced Training Options

Source: https://github.com/patchouli-m/sequencingcancerfinder/blob/master/README.md

Command for training Cancer-Finder with customizable parameters including training/validation directories, batch size, learning rate, maximum epochs, output directory, and GPU ID.

```bash
python -u train.py \
    --train_dir=<train_dir> \
    --val_dir=<val_dir> \
    --batch_size=<batch_size> \
    --lr=<learning_rate> \
    --max_epoch=<max_epoch> \
    --output=<output_dir> \
    --gpu_id=<id>
```

--------------------------------

### Data Loaders for Multi-Domain Training (Python)

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Sets up data loaders for training the VREx model across multiple domains. It utilizes domain loader utilities to create separate DataLoaders for each training and validation domain, enabling multi-domain training.

```python
from data_loaders.domian_loaders import train_domian_loaders_l, val_domian_loaders_l
from utils import args_utils

# Get training arguments
args = args_utils.get_args()

# Create training data loaders (one per domain)
train_loaders = train_domian_loaders_l(args, shuffle_state=True)
# Returns: list of DataLoaders, one per training domain file

# Create validation data loaders
val_loaders = val_domian_loaders_l(args, shuffle_state=True)
# Returns: dict mapping domain names to DataLoaders

# Training loop with multiple domains
for epoch in range(args.max_epoch):
    # Zip loaders to get minibatches from all domains
    train_minibatches_iterator = zip(*train_loaders)

    for single_train_minibatches in train_minibatches_iterator:
        # Each iteration gets one batch from each domain
        step_vals = algorithm.update(single_train_minibatches, opt, sch)
```

--------------------------------

### Training a Pre-trained Model with Cancer-Finder

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Train a new model for inference using your own multi-domain training data. The training process uses the VREx algorithm to learn domain-invariant representations across multiple source domains. Custom parameters for directories, batch size, learning rate, epochs, output, and GPU ID can be specified.

```bash
# Basic training with default parameters
python -u train.py

# Training with custom parameters
python -u train.py \
    --train_dir=data/train \
    --val_dir=data/val \
    --batch_size=10 \
    --lr=1e-3 \
    --max_epoch=5 \
    --output=my_training_output \
    --gpu_id=0

# Training data format requirements:
# - Place multiple domain files in train_dir (tsv, csv, or h5ad)
# - Each file represents one training domain
# - Matrix format: genes as rows, cells as columns
# - Include a 'label' row with binary labels (0=normal, 1=malignant)

# Example training data structure:
# data/train/domain_1.tsv
# data/train/domain_2.h5ad
# data/val/domain_val.csv
```

--------------------------------

### Command Line Inference with Cancer-Finder

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Run inference on gene expression matrices using pre-trained models from the command line. Supports various input formats and allows for custom thresholds. Outputs binary predictions for cells or spots.

```bash
# Basic inference with default threshold (0.5)
python -u infer.py \
    --ckp=checkpoints/sc_pretrain_article.pkl \
    --matrix=data_matrix.tsv \
    --out=out.csv

# Inference with custom threshold for MERFISH data
python -u infer.py \
    --ckp=checkpoints/sc_pretrain_article.pkl \
    --matrix=spatial_data.h5ad \
    --out=merfish_results.csv \
    --threshold=0.9766

# Supported input formats: tsv, csv, h5ad, h5, zarr
# Recommended thresholds:
# - scRNA-seq, 10x Visium, legacy ST, slide-seq: 0.5
# - MERFISH: 0.9766
```

--------------------------------

### Train Cancer-Finder with Gene Set

Source: https://github.com/patchouli-m/sequencingcancerfinder/blob/master/README.md

Command to initiate the training process for Cancer-Finder when a specific gene set is to be utilized.

```bash
python -u train_gene_set.py
```

--------------------------------

### Utility Functions for Model Training and Data Processing (Python)

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Provides essential utility functions for the Cancer Finder project, including setting random seeds for reproducibility, generating a gene list from training data, saving model checkpoints, calculating accuracy, and normalizing count matrices.

```python
from utils import opt_utils, args_utils

# Set random seed for reproducibility
args_utils.set_random_seed(seed=42)

# Generate gene list from training data
# Selects top variable genes across all domains
args.HVG_list = opt_utils.generate_genelist(args)
# Output: list of gene_num most variable genes

# Save model checkpoint with gene list
opt_utils.save_checkpoint(
    filename='model_epoch5.pkl',
    alg=algorithm,
    args=args
)
# Saved dict contains: {'model_dict': state_dict, 'HVG_list': gene_list}

# Calculate model accuracy on validation data
accuracy = opt_utils.accuracy(network=algorithm, loader=val_loader)
print(f"Validation accuracy: {accuracy:.4f}")

# Normalize count matrix and select features
normalized_df = opt_utils.normalize_matrix_counts(
    raw_df=expression_matrix,
    HVG_list=args.HVG_list,
    target_sum=10000
)
# Applies: normalize_total -> log1p -> gene selection
```

--------------------------------

### Training a Pre-trained Model

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Train a new model for inference using your own multi-domain training data. The training process utilizes the VREx algorithm to learn domain-invariant representations across multiple source domains.

```APIDOC
## Training a Pre-trained Model

### Description
Train a new model for inference using your own multi-domain training data. The training process uses the VREx algorithm to learn domain-invariant representations across multiple source domains.

### Method
`python train.py`

### Endpoint
N/A (Command Line Tool)

### Parameters
#### Command Line Arguments
- **`--train_dir`** (string) - Optional - Directory containing training data files (TSV, CSV, or H5AD). Each file represents a training domain.
- **`--val_dir`** (string) - Optional - Directory containing validation data files.
- **`--batch_size`** (int) - Optional - Batch size for training (default: determined by system).
- **`--lr`** (float) - Optional - Learning rate (default: 1e-4).
- **`--max_epoch`** (int) - Optional - Maximum number of training epochs (default: 100).
- **`--output`** (string) - Optional - Base name for output files and model checkpoints (default: 'training_output').
- **`--gpu_id`** (int) - Optional - GPU ID to use for training (default: -1 for CPU).

### Training Data Format Requirements
- Place multiple domain files (tsv, csv, or h5ad) in the specified `train_dir`.
- Each file represents one training domain.
- The matrix format should have genes as rows and cells as columns.
- A 'label' row must be included with binary labels (0=normal, 1=malignant).

### Request Example
```bash
# Basic training with default parameters
python -u train.py

# Training with custom parameters
python -u train.py \
    --train_dir=data/train \
    --val_dir=data/val \
    --batch_size=10 \
    --lr=1e-3 \
    --max_epoch=5 \
    --output=my_training_output \
    --gpu_id=0
```

### Response
#### Output Files
- Model checkpoints will be saved.
- Training logs will be generated.
```

--------------------------------

### Command Line Inference

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Perform inference on gene expression matrices using pre-trained models from the command line. The tool outputs binary predictions for each cell or spot.

```APIDOC
## Command Line Inference

### Description
Run inference on gene expression matrices using pre-trained models from the command line. The model outputs binary predictions (0=normal, 1=malignant) for each cell or spot based on a configurable threshold.

### Method
`python infer.py`

### Endpoint
N/A (Command Line Tool)

### Parameters
#### Command Line Arguments
- **`--ckp`** (string) - Required - Path to the pre-trained checkpoint file.
- **`--matrix`** (string) - Required - Path to the input gene expression matrix (supported formats: tsv, csv, h5ad, h5, zarr).
- **`--out`** (string) - Required - Path to save the output CSV file.
- **`--threshold`** (float) - Optional - Custom threshold for classification (default: 0.5). Recommended thresholds vary by data type.

### Request Example
```bash
# Basic inference with default threshold (0.5)
python -u infer.py \
    --ckp=checkpoints/sc_pretrain_article.pkl \
    --matrix=data_matrix.tsv \
    --out=out.csv

# Inference with custom threshold for MERFISH data
python -u infer.py \
    --ckp=checkpoints/sc_pretrain_article.pkl \
    --matrix=spatial_data.h5ad \
    --out=merfish_results.csv \
    --threshold=0.9766
```

### Response
#### Output File (`out.csv`)
- **`sample`** (string) - Cell/spot barcode.
- **`predict`** (float) - Binary prediction (0.0 for normal, 1.0 for malignant).

#### Response Example
```csv
sample,predict
AAACCTGAG...,1.0
AAACCTGCA...,0.0
AAACCTGTC...,1.0
```
```

--------------------------------

### Advanced Inference Options

Source: https://github.com/patchouli-m/sequencingcancerfinder/blob/master/README.md

Detailed command for running Cancer-Finder inference with configurable parameters for checkpoint, data file, output file, and inference threshold, accommodating various data types.

```bash
python -u infer.py \
    --ckp=<ckp_file> \
    --matrix=<data_file> \
    --out=<output_file> \
    --threshold=<threshold>
```

--------------------------------

### Train Cancer-Finder for Inference

Source: https://github.com/patchouli-m/sequencingcancerfinder/blob/master/README.md

Command to train a pre-trained model for inference purposes, without specifying a gene set. Uses default data directories for training and validation.

```bash
python -u train.py
```

--------------------------------

### Input Matrix Format for Cancer-Finder

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Prepare gene expression count matrices in the required format for Cancer-Finder. The input should have genes as rows and cells/spots as columns, with gene symbols as row indices. For training data, a 'label' row with binary labels is required.

```text
# Example TSV/CSV format:
# SYMBOL    Cell_1    Cell_2    Cell_3    ...    Cell_n
# TP53      0         2         0         ...    1
# BRCA1     1         0         3         ...    0
# MYC       0         1         0         ...    2
# ...       ...       ...       ...       ...    ...

# For training data, include a 'label' row:
# SYMBOL    Cell_1    Cell_2    Cell_3    ...    Cell_n
# TP53      0         2         0         ...    1
```

--------------------------------

### Run Cancer-Finder Inference

Source: https://github.com/patchouli-m/sequencingcancerfinder/blob/master/README.md

Command to perform inference using a pre-trained checkpoint on a given data matrix. Supports multiple input formats like tsv, csv, h5ad, zarr, and h5.

```bash
python -u infer.py --ckp=checkpoints/sc_pretrain_article.pkl --matrix=data_matrix.tsv --out=out.csv
```

--------------------------------

### Training with Saliency Map Generation in Cancer-Finder

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Train a model while simultaneously generating saliency maps to identify important genes for classification. This interpretability module tracks gradient information during training to produce gene importance rankings and visualizations.

```bash
# Train with saliency map generation
python -u train_gene_set.py

# Output files generated in <output_dir>_saliency/:
# - grad.csv: gradient values for all genes across epochs
# - top_20_gene_list_<epoch>.txt: top 20 important genes per epoch
# - saliency_map.png: heatmap visualization of gene importance
# - train_log_loss.txt: training loss log
# - train_log_val.txt: validation accuracy log
```

--------------------------------

### Training with Saliency Map Generation

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Train a model while simultaneously generating saliency maps to identify important genes for classification. This interpretability module tracks gradient information during training to produce gene importance rankings.

```APIDOC
## Training with Saliency Map Generation

### Description
Train a model while generating saliency maps to identify important genes for classification. This interpretability module tracks gradient information during training to produce gene importance rankings.

### Method
`python train_gene_set.py`

### Endpoint
N/A (Command Line Tool)

### Parameters
#### Command Line Arguments
- This script typically uses the same arguments as `train.py` for data directories, batch size, learning rate, epochs, etc. Specific arguments for saliency map generation might be implicitly handled or require additional flags not detailed here.

### Request Example
```bash
# Train with saliency map generation
python -u train_gene_set.py
```

### Response
#### Output Files (generated in `<output_dir>_saliency/`)
- **`grad.csv`**: Gradient values for all genes across epochs.
- **`top_20_gene_list_<epoch>.txt`**: Top 20 important genes per epoch.
- **`saliency_map.png`**: Heatmap visualization of gene importance.
- **`train_log_loss.txt`**: Training loss log.
- **`train_log_val.txt`**: Validation accuracy log.
```

--------------------------------

### Inference Data Iterator with Scanpy Support (Python)

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Implements the InferLoaders class for efficient batch processing of large expression matrices during inference. It supports loading data from files (e.g., .h5ad) and directly from AnnData objects, integrating with Scanpy.

```python
from utils.opt_utils import InferLoaders
import scanpy as sc

# Create inference iterator from file
args.matrix = 'large_dataset.h5ad'
infer_loaders = InferLoaders(args, step_num=10000)

# Iterate through data in batches
for input_data, input_loader in infer_loaders:
    # input_data: DataFrame with cell barcodes as index
    # input_loader: DataLoader with normalized tensor data

    for data in input_loader:
        predictions = torch.softmax(algorithm.predict(data[0]), axis=1)[:, 1]

    # Map predictions back to cell barcodes
    for idx, pred in enumerate(predictions):
        cell_barcode = input_data.index[idx]
        malignant_prob = pred.item()

# Direct AnnData object support
adata = sc.read_h5ad('my_data.h5ad')
args.matrix = adata  # Pass AnnData directly
infer_loaders = InferLoaders(args)
```

--------------------------------

### Input Matrix Format

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Details on the required format for gene expression count matrices used as input for Cancer-Finder.

```APIDOC
## Input Matrix Format

### Description
Prepare gene expression count matrices in the required format for Cancer-Finder. The input should have genes as rows and cells/spots as columns, with gene symbols as row indices.

### Format Details
- **Genes**: Rows
- **Cells/Spots**: Columns
- **Row Index**: Gene symbols (e.g., TP53, BRCA1)
- **Values**: Gene expression counts or normalized values.

### Example TSV/CSV Format
```
SYMBOL	Cell_1	Cell_2	Cell_3	...	Cell_n
TP53	0	2	0	...	1
BRCA1	1	0	3	...	0
MYC	0	1	0	...	2
...	...	...	...	...	...
```

### Training Data Specifics
For training data, an additional 'label' row must be included at the beginning of the file:

```
SYMBOL	Cell_1	Cell_2	Cell_3	...	Cell_n
label	0	1	0	...	1
TP53	0	2	0	...	1
BRCA1	1	0	3	...	0
MYC	0	1	0	...	2
...	...	...	...	...	...
```

### Supported File Formats
- TSV
- CSV
- H5AD
- H5
- ZARR
```

--------------------------------

### VREx Model Architecture Definition (Python)

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Defines the VREx neural network architecture, comprising a fully connected feature extractor (FcNet) and a classification head, implementing the VREx domain generalization algorithm. It takes gene expression data as input and outputs class logits.

```python
from models.model import VREx, FcNet, feat_classifier

# Model architecture overview:
# FcNet: Fully connected feature extractor
#   - Input: gene_num features (default 4572)
#   - Hidden: Linear -> Dropout -> Linear
#   - Output: 512-dimensional bottleneck features

# feat_classifier: Classification head
#   - Input: 512 bottleneck features
#   - Output: num_classes logits (default 2)

# VREx: Combined network with domain generalization
class VREx(nn.Module):
    def __init__(self, args):
        # Featurizer extracts domain-invariant features
        self.featurizer = FcNet(in_size=len(args.HVG_list))
        # Classifier maps features to class predictions
        self.classifier = feat_classifier(args.num_classes, 512)
        self.network = nn.Sequential(self.featurizer, self.classifier)

    def predict(self, x):
        # Returns raw logits
        return self.network(x)

    def update(self, minibatches, opt, sch):
        # VREx loss: mean loss + penalty_weight * variance(losses)
        # Encourages consistent performance across domains
        return {'loss': loss, 'mean': mean, 'penalty': penalty}
```

--------------------------------

### Python API Inference with Cancer-Finder

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Perform inference directly from Python code using the `infering` function. This function accepts AnnData objects or file paths, enabling seamless integration into existing single-cell analysis workflows. It can save output to a CSV file or return a DataFrame.

```python
from infer import infering
import scanpy as sc

# Load your single-cell data
adata = sc.read_h5ad('my_scRNA_data.h5ad')

# Run inference with default parameters
result_df = infering(
    matrix=adata,
    ckp='checkpoints/sc_pretrain_article.pkl',
    threshold=0.5,
    out='predictions.csv',
    save_output=True
)

# Result DataFrame contains:
# - 'sample': cell/spot barcodes
# - 'predict': binary prediction (0=normal, 1=malignant)
print(result_df.head())
#        sample  predict
# 0  AAACCTGAG...      1.0
# 1  AAACCTGCA...      0.0
# 2  AAACCTGTC...      1.0

# Run inference without saving output file
result_df = infering(
    matrix='expression_matrix.csv',
    ckp='checkpoints/sc_pretrain_article.pkl',
    threshold=0.5,
    save_output=False
)
```

--------------------------------

### Python API - infering Function

Source: https://context7.com/patchouli-m/sequencingcancerfinder/llms.txt

Perform inference directly from Python code using the `infering` function, which accepts AnnData objects or file paths. This function integrates seamlessly into existing single-cell analysis workflows.

```APIDOC
## Python API - infering Function

### Description
Perform inference directly from Python code using the `infering` function. This function accepts AnnData objects or file paths and enables seamless integration into existing single-cell analysis workflows.

### Method
`infering(matrix, ckp, threshold=0.5, out=None, save_output=True)`

### Endpoint
N/A (Python Function)

### Parameters
#### Function Arguments
- **`matrix`** (AnnData or string) - Required - Either an AnnData object or a file path to the input gene expression matrix (supported formats: tsv, csv, h5ad, h5, zarr).
- **`ckp`** (string) - Required - Path to the pre-trained checkpoint file.
- **`threshold`** (float) - Optional - Custom threshold for classification (default: 0.5).
- **`out`** (string) - Optional - Path to save the output predictions. If `None` and `save_output` is `True`, a default name will be used.
- **`save_output`** (boolean) - Optional - Whether to save the predictions to a file (default: `True`).

### Request Example
```python
from infer import infering
import scanpy as sc

# Load your single-cell data
adata = sc.read_h5ad('my_scRNA_data.h5ad')

# Run inference with default parameters and save output
result_df = infering(
    matrix=adata,
    ckp='checkpoints/sc_pretrain_article.pkl',
    threshold=0.5,
    out='predictions.csv',
    save_output=True
)

# Print the head of the result DataFrame
print(result_df.head())

# Run inference without saving output file
result_df_no_save = infering(
    matrix='expression_matrix.csv',
    ckp='checkpoints/sc_pretrain_article.pkl',
    threshold=0.5,
    save_output=False
)
```

### Response
#### Return Value
- **`result_df`** (pandas.DataFrame) - A DataFrame containing the inference results.
  - **`sample`** (string) - Cell/spot barcode.
  - **`predict`** (float) - Binary prediction (0.0 for normal, 1.0 for malignant).

#### Response Example
```python
# Example output DataFrame structure:
#        sample  predict
# 0  AAACCTGAG...      1.0
# 1  AAACCTGCA...      0.0
# 2  AAACCTGTC...      1.0
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.