Mepylome

https://mepylome.readthedocs.io/en/latest/
Admin
Mepylome is a high-performance Python toolkit tailored for parsing, processing, and analyzing...
Tokens:81,311
Snippets:469
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install mepylome from source

Source: https://mepylome.readthedocs.io/en/latest/_sources/installation.rst.txt

Clone the repository and install the package locally to obtain the latest version.

```sh
git clone https://github.com/brj0/mepylome.git && cd mepylome && pip install .
```

--------------------------------

### Launch Mepylome GUI Tutorial

Source: https://mepylome.readthedocs.io/en/latest/_sources/gui_cli.rst.txt

Executes the Mepylome GUI with example data for a quick demonstration.

```sh
mepylome --tutorial
```

--------------------------------

### Running the Dash Application

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Starts the Dash server, optionally opening a browser tab automatically.

```python
def run_app(self, *, open_tab: bool = False) -> None:
        """Runs the mepylome Dash application.

        Args:
            open_tab: Whether to automatically open a new browser tab with the
                application URL.
        """
        self.app = self.get_app()
        free_port = get_free_port(self.port)
        if open_tab:

            def open_browser_tab() -> None:
                webbrowser.open_new_tab(f"http://{self.host}:{free_port}")

            threading.Timer(1, open_browser_tab).start()

        self.app.run(
            debug=self.debug,
            host=self.host,
            use_reloader=False,
            port=free_port,
        )
```

--------------------------------

### Install mepylome from PyPI

Source: https://mepylome.readthedocs.io/en/latest/_sources/installation.rst.txt

Use pip to install the package directly from the Python Package Index.

```sh
pip install mepylome
```

--------------------------------

### Launch Interactive GUI

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Starts the web-based GUI for interactive methylation analysis.

```python
>>> methyl_analysis.run_app(open_tab=True)
```

--------------------------------

### Example Project Directory Structure

Source: https://mepylome.readthedocs.io/en/latest/_sources/gui_cli.rst.txt

Illustrates the recommended organization for analysis and reference directories containing IDAT files and annotation data.

```text
projects/
    ├── analysis_dir/
    │   ├── annotation.csv    # Annotation file with sample information
    │   └── idat_files/       # Subdirectory containing IDAT files
    │       ├── 200925700133_R02C01_Grn.idat
    │       ├── 200925700133_R02C01_Red.idat
    │       ├── 200925700133_R03C01_Grn.idat
    │       ├── 200925700133_R03C01_Red.idat
    │       ├── 200925700133_R04C01_Grn.idat
    │       ├── 200925700133_R04C01_Red.idat
    │       ├── 200925700133_R05C01_Grn.idat
    │       ├── 200925700133_R05C01_Red.idat
    └── reference_dir/
        ├── 201904410008_R02C01_Grn.idat
        ├── 201904410008_R02C01_Red.idat
        ├── 201904410008_R03C01_Grn.idat
        ├── 201904410008_R03C01_Red.idat
        ├── 201904410008_R04C01_Grn.idat
        ├── 201904410008_R04C01_Red.idat
        ├── 201904410008_R05C01_Grn.idat
        ├── 201904410008_R05C01_Red.idat
        └── 201904410008_R06C01_Grn.idat
```

--------------------------------

### Initialize Manifest Objects

Source: https://mepylome.readthedocs.io/en/latest/api.html

Examples of initializing a Manifest object using array type strings or ArrayType enums.

```python
>>> # To initialize a manifest object for Illumina 450k array:
>>> manifest = Manifest("450k")
>>> manifest
```

```python
>>> # To initialize a manifest object for Illumina EPIC array
>>> manifest = Manifest(ArrayType.ILLUMINA_EPIC)
>>> type_1 = manifest.probe_info(ProbeType.ONE)
```

--------------------------------

### Setup Supervised Classification

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Imports necessary scikit-learn components and initializes the MethylAnalysis object for classification tasks.

```python
>>> import numpy as np
>>> import pandas as pd
>>> from sklearn.ensemble import (
>>>     ExtraTreesClassifier,
>>>     RandomForestClassifier,
>>> )
>>> from sklearn.feature_selection import (
>>>     SelectKBest,
>>> )
>>> from sklearn.pipeline import Pipeline

>>> from mepylome.analysis import MethylAnalysis

>>> methyl_analysis = MethylAnalysis(
>>>     analysis_dir=ANALYSIS_DIR,
>>>     reference_dir=REFERENCE_DIR,
>>> )
```

--------------------------------

### MethylAnalysis.run_app

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Starts the Dash application for interactive analysis and visualization.

```APIDOC
## MethylAnalysis.run_app

### Description
Launches the interactive Dash application associated with the current MethylAnalysis instance. This allows users to visualize CNV data and UMAP plots.
```

--------------------------------

### Seek to manifest data start

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/manifests.html

Moves the file pointer to the beginning of the data section by locating the 'IlmnID' header.

```python
    @staticmethod
    def _seek_to_start(manifest_file: IO[bytes]) -> None:
        """Move the manifest file pointer to the start of the data section.

        Details:
            The function searches for the first occurrence of the left-most
            column named "IlmnID".
        """
        reset_file(manifest_file)

        current_pos = manifest_file.tell()
        header_line = manifest_file.readline()

        while not header_line.startswith(b"IlmnID"):
            current_pos = manifest_file.tell()
            if not header_line:
                msg = (
                    "The first (left-most) column in your manifest must "
                    "contain 'IlmnID'. This defines the header row."
                )
                raise EOFError(msg)
            header_line = manifest_file.readline()

        if current_pos == 0:
            manifest_file.seek(current_pos)
        else:
            manifest_file.seek(current_pos - 1)
```

--------------------------------

### Annotation CSV Format

Source: https://mepylome.readthedocs.io/en/latest/gui_cli.html

Example content for the annotation.csv file required for sample metadata.

```csv
IDAT,Diagnosis,Site
200925700133_R02C01,Osteosarcoma,Femur
200925700133_R03C01,Osteosarcoma,Humerus
200925700133_R04C01,Osteosarcoma,Femur
200925700133_R05C01,Osteosarcoma,Femur
```

--------------------------------

### ReferenceMethylData usage example

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/beads.html

Demonstrates initializing a reference set and using it to calculate CNV for different array types.

```python
>>> # 'directory' contains 450k, EPIC and EPICv2 idat files
>>> reference = ReferenceMethylData(file=directory, prep="illumina")
>>> sample_450k = MethylData(file=idat_file_450k)
>>> sample_epic = MethylData(file=idat_file_epic)
>>> sample_epicv2 = MethylData(file=idat_file_epicv2)
>>> # reference can be used for all types
>>> cnv_450k = CNV(sample_450k, reference)
>>> cnv_epic = CNV(sample_epic, reference)
>>> cnv_epicv2 = CNV(sample_epicv2, reference)
```

--------------------------------

### Retrieve unique IDAT basepaths

Source: https://mepylome.readthedocs.io/en/latest/api.html

Examples of extracting unique basepaths from directories, lists of files, or compressed IDAT files.

```python
>>> idat_basepaths("/path/to/dir")
[PosixPath('/path/to/dir/file1'), PosixPath('/path/to/dir/file2')]
```

```python
>>> idat_basepaths(["/path1/file1_Grn.idat", "/path2/file2_Red.idat"])
[PosixPath('/path1/file1'), PosixPath('/path2/file2')]
```

```python
>>> idat_basepaths("/path/to/idat/file_Grn.idat.gz")
[PosixPath('/path/to/idat/file')]
```

--------------------------------

### Setting up tutorial directories

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Define the base paths for analysis and reference data.

```python
>>> DIR = Path.home() / "mepylome" / "tutorial"
>>> ANALYSIS_DIR = DIR / "tutorial_analysis"
>>> REFERENCE_DIR = DIR / "tutorial_reference"
```

--------------------------------

### Downloading tutorial data

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Download and prepare the required data files for the tutorial.

```python
>>> from mepylome.utils import setup_tutorial_files

>>> setup_tutorial_files(ANALYSIS_DIR, REFERENCE_DIR)
```

--------------------------------

### Initialize and use Manifest objects

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/manifests.html

Demonstrates how to instantiate a manifest for a specific array type and access probe information, as well as how to trigger a global load of all manifests.

```python
>>> # To initialize a manifest object for Illumina EPIC array
>>> manifest = Manifest(ArrayType.ILLUMINA_EPIC)
>>> type_1 = manifest.probe_info(ProbeType.ONE)

>>> # To load all manifests when first used:
>>> Manifest.load()
```

--------------------------------

### Get CNV Segments

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/cnv.html

Performs circular binary segmentation to identify CNV segments.

```python
@staticmethod
def _get_segments(df: pd.DataFrame) -> pd.DataFrame:
    """Performs circular binary segmentation to identify CNV segments.

    This method applies the circular binary segmentation (CBS) algorithm
    to identify copy number variation (CNV) segments based on the median
    values of genomic bins. The CBS algorithm is a time-intensive
    operation that segments genomic regions based on change-points in
    intensity values.

    Args:
        df: DataFrame containing the median values of genomic bins.
```

--------------------------------

### Access Mepylome Help

Source: https://mepylome.readthedocs.io/en/latest/_sources/gui_cli.rst.txt

Display all available command-line parameters and options.

```sh
mepylome --help
```

--------------------------------

### Initialize Sample Data

Source: https://mepylome.readthedocs.io/en/latest/tutorial.html

Set up the sample methylation data for analysis.

```python
>>> sample = methyl_data
```

--------------------------------

### Get Sex Chromosome Indices

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/beads.html

Maps X and Y chromosome probes to their positions in a sample array.

```python
@memoize
def _get_sex_indices(
    sample_probes: np.ndarray, array_type: ArrayType
) -> tuple[np.ndarray, np.ndarray]:
    """Return probe indices for X and Y chromosomes in a sample array.

    Maps manifest-defined X and Y chromosome probes to their positions
    within a given sample probe array.

    Returns:
        Tuple containing:
        - indices of X-chromosome probes in sample_probes
        - indices of Y-chromosome probes in sample_probes
    """
    manifest_df = Manifest(array_type).data_frame

    x_probes = manifest_df[manifest_df.Chromosome == Chromosome.CHRX][
        "IlmnID"
    ].values
    y_probes = manifest_df[manifest_df.Chromosome == Chromosome.CHRY][
        "IlmnID"
    ].values

    return (
        _overlap_indices(sample_probes, x_probes)[0],
        _overlap_indices(sample_probes, y_probes)[0],
    )
```

--------------------------------

### Setting Up CNV Analysis

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Initialize sample data and reference datasets for CNV analysis.

```python
>>> sample = methyl_data
```

```python
>>> reference = MethylData(file=REFERENCE_DIR)
```

```python
>>> reference_all = ReferenceMethylData(REFERENCE_DIR)
```

--------------------------------

### visualize_region

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Visualizes methylation across an arbitrary genomic region defined by chromosome, start, and end coordinates.

```APIDOC
## visualize_region(chromosome, start, end, array_type=None, ids=None, show=True)

### Description
Visualizes methylation for a custom genomic region. This is useful for loci without gene annotations or for specific regions of interest.

### Parameters
- **chromosome** (str | Chromosome) - Required - Chromosome identifier (e.g., "chr3" or "3").
- **start** (int) - Required - Region start position (genomic coordinate).
- **end** (int) - Required - Region end position (genomic coordinate).
- **array_type** (str | ArrayType) - Optional - Array type used for lookup.
- **ids** (Sequence[str]) - Optional - Sample IDs to include.
- **show** (bool) - Optional - If True, displays the figure immediately and returns None.
```

--------------------------------

### Compute pOOBAH Detection p-values

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/beads.html

Example usage and implementation of the poobah method for calculating detection p-values.

```python
            >>> methyl = MethylData(file=idat_basepath)
            >>> pvals = methyl.poobah()
            >>> mask = pvals >= 0.05
```

```python
        ci = MethylData._cached_indices(
            self.manifest, self.bead_addresses, "noob"
        )

        # NOTE: sesame additionaly filteres probes with backgroundMask
        grn_oob = np.concatenate(
            [
                self.green[:, ci.bead_t1r_a],
                self.green[:, ci.bead_t1r_b],
            ],
            axis=1,
        )
        red_oob = np.concatenate(
            [
                self.red[:, ci.bead_t1g_a],
                self.red[:, ci.bead_t1g_b],
            ],
            axis=1,
        )

        n_probes = len(ci.probe_idx)
        n_samples = len(self.sample_ids)
        pvals = np.full((n_samples, n_probes), np.nan)

        def ecdf(x: np.ndarray, ref: np.ndarray) -> np.ndarray:
            """Evaluate ECDF of OOB green/red at values x."""
            return np.searchsorted(ref, x, side="right") / len(ref)

        for s in range(n_samples):
            oob_g = np.sort(grn_oob[s])
            oob_r = np.sort(red_oob[s])

            # Type I Red probes
            m_1red = self.red[s, ci.bead_t1r_b]
            u_1red = self.red[s, ci.bead_t1r_a]
            pvals[s, ci.meth_t1r] = 1.0 - np.maximum(
                ecdf(m_1red, oob_r), ecdf(u_1red, oob_r)
            )

            # Type I Green probes
            m_1grn = self.green[s, ci.bead_t1g_b]
            u_1grn = self.green[s, ci.bead_t1g_a]
            pvals[s, ci.meth_t1g] = 1.0 - np.maximum(
                ecdf(m_1grn, oob_g), ecdf(u_1grn, oob_g)
            )

            # Type II probes
            m_2 = self.green[s, ci.bead_t2_a]
            u_2 = self.red[s, ci.bead_t2_a]
            pvals[s, ci.meth_t2] = 1.0 - np.maximum(
                ecdf(m_2, oob_g), ecdf(u_2, oob_r)
            )

        return pd.DataFrame(
            pvals.T,
            columns=self.sample_ids,
            index=ci.probe_ids,
        )
```

--------------------------------

### Initialize RawData from IDAT files

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/beads.html

Demonstrates how to instantiate the RawData class using single or multiple IDAT basepaths.

```python
>>> idat_basepath0 = directory_path / "200925700125_R07C01"
>>> idat_basepath1 = directory_path / "200925700133_R02C01_Grn.idat"
>>> raw_data = RawData(idat_basepath0)
>>> raw_data = RawData([idat_basepath0, idat_basepath1])
```

--------------------------------

### Initialize and Run MethylAnalysis

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Demonstrates basic instantiation and execution of the analysis application, as well as configuration with specific directory paths.

```python
>>> # Basic usage
>>> from mepylome import MethylAnalysis
>>> analysis0 = MethylAnalysis()
>>> analysis0.run_app()
>>> # Usage if directories are known in advance
>>> analysis1 = MethylAnalysis(
>>>     analysis_dir='/path/to/idat/dir',
>>>     reference_dir='/path/to/reference/idat/dir',
>>>     annotation='/path/to/annotation/spread/sheat/with/2/cols',
>>>     output_dir='/path/to/mepylome/output',
>>> )
>>> analysis1.run_app()
```

--------------------------------

### MethylAnalysis Class

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Main class for configuring and running methylation analysis, including data setup and visualization.

```APIDOC
## MethylAnalysis

### Description
Main class for methylation analysis. Provides methods for setting up analysis parameters, reading data, and running a Dash-based web application for data visualization.

### Constructor Arguments
- **analysis_dir** (str) - Directory containing IDAT files for analysis.
- **annotation** (str) - Path to an annotation spreadsheet.
- **reference_dir** (str) - Directory containing CNV neutral reference IDAT files.
- **output_dir** (str) - Directory where output files will be saved (default: "/tmp/mepylome/analysis").
- **test_dir** (str) - Directory for test files.
- **prep** (str) - Prepreparation method: 'illumina', 'swan', or 'noob'.
- **cpgs** (list|set|np.ndarray|str) - Specifies CpG sites to analyze (e.g., '450k', 'epic', 'auto').
- **cpg_blacklist** (list|set) - CpG sites to exclude.
- **n_cpgs** (int) - Number of CpG sites to select for UMAP.
- **classifiers** (dict) - Classifier model configurations.
```

--------------------------------

### Dash Application Initialization

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Configures and returns a Dash application object with predefined assets and layout.

```python
def get_app(self) -> Dash:
        """Returns a Dash application object for methylation analysis."""
        current_dir = Path(__file__).resolve().parent
        assets_folder = current_dir.parent / "data" / "assets"
        app = Dash(
            __name__,
            update_title="",
            assets_folder=str(assets_folder),
            external_stylesheets=[dbc.themes.BOOTSTRAP],
        )
        app._favicon = "favicon.svg"  # type: ignore[assignment]
        app.title = "mepylome"
        build_layout(app, self)
        register_callbacks(app, self)

        return app
```

--------------------------------

### Extract SNP probes from manifest

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/manifests.html

Filters the internal data frame to return only rows where the IlmnID starts with 'rs'.

```python
def _read_snp_probes(self) -> pd.DataFrame:
    """Extracts SNP probes from the manifest data frame."""
    snp_df = self.data_frame.copy()
    return snp_df[snp_df.IlmnID.str.match("rs", na=False)]
```

--------------------------------

### Set Up Reference Data

Source: https://mepylome.readthedocs.io/en/latest/tutorial.html

Load reference data using either specific IDAT pairs or a directory containing multiple array types.

```python
>>> reference = MethylData(file=REFERENCE_DIR)
```

```python
>>> reference_all = ReferenceMethylData(REFERENCE_DIR)
```

--------------------------------

### Loading manifest files

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Initialize manifest objects for different array types. Note that the first initialization triggers a download to the local cache.

```python
>>> manifest_450k = Manifest("450k")
>>> manifest_epic = Manifest("epic")
>>> manifest_epic_v2 = Manifest("epicv2")
>>> manifest_msa48 = Manifest("msa48")
```

--------------------------------

### Configure CNV Bins and Detail

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Initializes binning and detail settings for the CNV analysis object.

```python
>>> cnv.set_bins()
>>> cnv.set_detail()
```

--------------------------------

### Visualize Methylation by Genomic Region

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Visualizes methylation for an arbitrary genomic region defined by chromosome, start, and end coordinates.

```python
    def visualize_region(
        self,
        chromosome: str | Chromosome,
        start: int,
        end: int,
        array_type: str | ArrayType | None = None,
        ids: Sequence[str] | None = None,
        show: bool = True,
    ) -> go.Figure | None:
        """Visualizes methylation across an arbitrary genomic region.

        Like `visualize_gene`, but for any region given as
        chromosome/start/end instead of a gene symbol -- useful for loci
        without a gene annotation, or custom regions of interest.

        Args:
            chromosome: Chromosome (e.g. "chr3" or "3").

            start: Region start position (genomic coordinate).

            end: Region end position (genomic coordinate).
```

--------------------------------

### Get Balanced Indices

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Returns indices for a balanced selection of feature labels, optionally using a random seed for reproducibility.

```python
def get_balanced_indices(
    feature_labels: Sequence[str],
    seed: int | None = None,
) -> np.ndarray:
    """Returns indices of a balanced selection of feature labels.

    Args:
        feature_labels: Labels of features to balance.

        seed: Random seed for reproducibility.

```

--------------------------------

### Configure Classifiers

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Demonstrates setting up classifier configurations using a list of models or dictionaries containing model, name, and cross-validation parameters.

```python
>>> clf = {
>>>     "model": RandomForestClassifier(),
>>>     "name": "Custom RF",
>>>     "cv": 10,
>>> }
>>> analysis.classifiers = ["kbest-rf", clf]
>>> analysis.classifiers
[{'model': 'kbest-rf', 'name': 'Custom_Classifier_0', 'cv':
StratifiedKFold(n_splits=5, random_state=None, shuffle=True)},
{'model': RandomForestClassifier(), 'name': 'Custom RF', 'cv':
StratifiedKFold(n_splits=10, random_state=None, shuffle=True)}]
```

--------------------------------

### MethylAnalysis Class Definition

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Main class for managing methylation analysis, including data setup, file handling, and visualization.

```python
[docs]
class MethylAnalysis:
    """Main class for methylation analysis including a GUI application.

    Main class for methylation analysis, providing methods for
    setting up analysis parameters, reading data, and running a Dash-based
    web application for data visualization.

    Args:
        analysis_dir: Directory containing IDAT files for analysis.

        annotation: Path to an annotation spreadsheet used to map sample files
            located in both `analysis_dir` and `test_dir`. One of the columns
            must contain the ID corresponding to the IDAT files (such as
            SentrixID or ID from files downloaded from GEO). If not provided,
            the system will attempt to identify the correct column
            automatically. If the annotation file is missing, it will search
            for a spreadsheet within the `analysis_dir` if available.

        reference_dir: Directory containing CNV neutral reference IDAT files.
            Must be provided if you wanna generate CNV plots.

        output_dir: Directory where output files will be saved.
            (default: "/tmp/mepylome/analysis").

        test_dir: Directory for test files, including new cases for analysis or
            validation. Files uploaded via the GUI will be placed here. If set
            to `None`, the application will automatically use a temporary
            directory.

        prep: Prepreparation method used for methylation microarrays:
            'illumina', 'swan', or 'noob.

        cpgs: Specifies the CpG sites to analyze. Possible values:

            1. A list, set, or NumPy array of official Illumina CpG site names.
            2. A path to a CSV file containing the CpG sites.
            3. A string specifying a predefined array type:

                - '450k'   : The CpG sites from the Illumina 450k array.
                - 'epic'   : The CpG sites from the Illumina EPIC array.
                - 'epicv2' : The CpG sites from the Illumina EPIC v2 array.
                - 'msa48'  : The CpG sites from the Illumina MSA array.

            4. A '+' -joined string of the options above combining multiple
            array types, returning the intersection of their CpG sites. For
            example:

                - '450k+epic'  : CpG sites both in the 450k and EPIC arrays.
                - 'epic+epicv2': CpG sites both in the EPIC and EPICv2
                  arrays.

            5. 'auto': Automatically detects all array types from
            IDAT files in `analysis_dir` and returns the intersection of CpG
            sites. This process may take longer as all files need to be read
            and, if necessary, decompressed.

        cpg_blacklist: A list or set of CpG sites to exclude.

        n_cpgs: Number of CpG sites to select for UMAP.

        classifiers: Classifier model(s). Each classifier can
            be provided as:

            - A dictionary containing:

                - 'model': The classifier model object as defined
                  below (required).
                - 'name': A name for the classifier (default:
                    "Custom_Classifier_<index>").
                - 'cv': Cross-validation strategy (default: `self.cv_default`).
```

--------------------------------

### Initialize a Manifest object

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/manifests.html

Create a Manifest instance for a specific array type, such as the Illumina 450k array.

```python
>>> # To initialize a manifest object for Illumina 450k array:
>>> manifest = Manifest("450k")
>>> manifest
```

--------------------------------

### Print RawData overview

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Display a summary of the loaded raw data.

```python
>>> print(raw_data)
RawData():
**********

array_type: epic

manifest: epic

probes:
['200925700125_R07C01']

ids:
[ 1600101  1600111  1600115 ... 99810978 99810990 99810992]

green:
[[15629  8469  7015 ... 10228  7971   943]]

red:
[[ 4429  1575 24955 ...  6594 15010  5336]]

green_df:
          200925700125_R07C01
1600101                 15629
1600111                  8469
1600115                  7015
1600123                  7975
1600131                   938
...                       ...
99810958                 6292
99810970                  318
99810978                10228
99810990                 7971
99810992                  943

[1052641 rows x 1 columns]

red_df:
          200925700125_R07C01
1600101                  4429
1600111                  1575
1600115                 24955
1600123                 17707
1600131                  8967
...                       ...
99810958                 1881
99810970                 1936
99810978                 6594
99810990                15010
99810992                 5336

[1052641 rows x 1 columns]
```

--------------------------------

### Assemble NODE_DTYPE structured array

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/purity.html

Constructs a structured numpy array compatible with the installed scikit-learn version to ensure version-independent model loading.

```python
def _build_node_array(
    left: np.ndarray,
    right: np.ndarray,
    feature: np.ndarray,
    threshold: np.ndarray,
    n_nodes: int,
) -> np.ndarray:
    """Assemble a NODE_DTYPE structured array from primitive arrays.

    Called at load time so NODE_DTYPE always reflects the currently installed
    sklearn version, keeping the bundle version-independent.

    Args:
        left: Left child indices (sklearn sentinel TREE_LEAF for leaves).
        right: Right child indices (sklearn sentinel TREE_LEAF for leaves).
        feature: Split feature indices (TREE_UNDEFINED for leaves).
        threshold: Split thresholds (TREE_UNDEFINED for leaves).
        n_nodes: Total number of nodes.

    Returns:
        Structured numpy array with dtype NODE_DTYPE compatible with the
        currently installed sklearn version.
    """
    nodes = np.zeros(n_nodes, dtype=NODE_DTYPE)
    nodes["left_child"] = left
    nodes["right_child"] = right
    nodes["feature"] = feature
    nodes["threshold"] = threshold
    nodes["impurity"] = 0.0
    nodes["n_node_samples"] = 1
    nodes["weighted_n_node_samples"] = 1.0
    # missing_go_to_left added in newer sklearn; default 0 (False) is fine
    if "missing_go_to_left" in NODE_DTYPE.names:
        nodes["missing_go_to_left"] = 0
    return nodes
```

--------------------------------

### run_app

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Runs the mepylome Dash application server.

```APIDOC
## run_app(open_tab=False)

### Description
Starts the mepylome Dash application server on a free port.

### Parameters
- **open_tab** (bool) - Optional - Whether to automatically open a new browser tab with the application URL. Defaults to False.
```

--------------------------------

### Segmentation Algorithm Selection

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/cnv.html

Dynamically selects a segmentation function based on available libraries (ruptures, linear_segment, or cbseg). Returns None if no supported library is installed.

```python
def _get_segmentation_fn() -> Callable[[list[float]], list[list[int]]] | None:
    try:
        import ruptures

        def function(bin_values: list[float]) -> list[list[int]]:
            algo = ruptures.KernelCPD("linear", min_size=5).fit(bin_values)
            segments = algo.predict(pen=1.5)
            return [
                [start, end]
                for start, end in zip([0] + segments, segments, strict=False)
            ]

        return function
    except Exception:
        pass
    try:
        import linear_segment

        def function(bin_values: list[float]) -> list[list[int]]:
            segments = linear_segment.segment(
                bin_values, shuffles=1000, cutoff=0.3
            )
            return [[s.start, s.end] for s in segments]

        return function
    except Exception:
        pass
    try:
        import cbseg

        def function(bin_values: list[float]) -> list[list[int]]:
            segments = cbseg.segment(bin_values, shuffles=1000, p=0.0001)
            return [[s.start, s.end] for s in segments]

        return function
    except Exception:
        logger.warning(
            "**Warning**: Segmentation won't be calculated due to missing "
            "segmentation altorithm ('ruptures', 'linear_segment' or 'cbseg')."
        )
        return None
```

--------------------------------

### Initialize MethylAnalysis with Custom Parameters

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Advanced initialization using optional parameters like test directories, preprocessing methods, and CpG selection criteria.

```python
>>> TEST_DIR = DIR / "test_dir"
>>> methyl_analysis = MethylAnalysis(
>>>     analysis_dir=ANALYSIS_DIR,
>>>     reference_dir=REFERENCE_DIR,
>>>     output_dir=OUTPUT_DIR,
>>>     # New cases for validation, excluded from classifier training.
>>>     test_dir=TEST_DIR,
>>>     # Load beta values for all CpG sites into memory
>>>     load_full_betas=True,
>>>     # Use SWAN preprocessing method
>>>     prep="swan",
>>>     # Provide annotation file (if not already in analysis_dir)
>>>     annotation=ANALYSIS_DIR / "annotation.csv",
>>>     # Number of CpGs for UMAP analysis
>>>     n_cpgs=5000,
>>>     # Analyze the 'best' CpG sites
>>>     cpg_selection="top",
>>>     # Show segmentation intervals in CNV plot
>>>     do_seg=True,
>>> )
```

--------------------------------

### Initialize RawData from IDAT files

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Load raw intensity data from single or multiple IDAT files.

```python
>>> idat_file = ANALYSIS_DIR / "200925700125_R07C01_Red.idat"
>>> # or
>>> idat_file = ANALYSIS_DIR / "200925700125_R07C01_Grn.idat"
>>> # or
>>> idat_file = ANALYSIS_DIR / "200925700125_R07C01"
>>> raw_data = RawData(idat_file)
```

```python
>>> idat_file0 = ANALYSIS_DIR / "200925700125_R07C01_Grn.idat"
>>> idat_file1 = ANALYSIS_DIR / "200925700133_R02C01"

>>> raw_data_2 = RawData([idat_file0, idat_file1])
```

```python
>>> raw_data_all = RawData(REFERENCE_DIR)
```

--------------------------------

### Noob Preprocessing Method

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/beads.html

Performs normal-exponential background correction and dye-bias normalization.

```python
    def preprocess_noob(
        self,
        offset: float = 15,
        dye_method: str = "single",
    ) -> None:
        """The Noob preprocessing method.

        Description:
            Noob (normal-exponential out-of-band) is a background correction
            method with dye-bias normalization.

        Args:
            offset: An offset for the normexp background correction.

            dye_method: How should dye bias correction be done: "single" for
                single sample approach, or "reference" for a reference array.

        References:
            TJ Triche, DJ Weisenberger, D Van Den Berg, PW Laird and KD
            Siegmund. Low-level processing of Illumina Infinium DNA
            Methylation BeadArrays.  Nucleic Acids Res (2013) 41, e90.
            doi:10.1093/nar/gkt090.
        """
        self.prep = "noob"
        if self.array_type == ArrayType.ILLUMINA_27K:
            raise ValueError(f"{self.array_type} requires raw mode.")

        ci = MethylData._cached_indices(
            self.manifest, self.bead_addresses, "noob"
        )

        self._preprocess_raw(ci)

        grn_oob = np.concatenate(
            [
                self.green[:, ci.bead_t1r_a],
                self.green[:, ci.bead_t1r_b],
            ],
            axis=1,
        )
        red_oob = np.concatenate(
            [self.red[:, ci.bead_t1g_a], self.red[:, ci.bead_t1g_b]],
            axis=1,
        )

        methylated = self.methylated
        unmethylated = self.unmethylated
        methylated[methylated <= 0] = 1
        unmethylated[unmethylated <= 0] = 1

        grn_m = methylated[:, ci.meth_t1g]
        grn_u = unmethylated[:, ci.meth_t1g]
        grn_2 = methylated[:, ci.meth_t2]

        xf_grn = np.concatenate([grn_m, grn_u, grn_2], axis=1)
        xs_grn = normexp_get_xs(xf_grn, controls=grn_oob, offset=offset)

        cumsum = np.cumsum([0, grn_m.shape[1], grn_u.shape[1], grn_2.shape[1]])
        slice_grn_m = slice(cumsum[0], cumsum[1])
        slice_grn_u = slice(cumsum[1], cumsum[2])
        slice_grn_2 = slice(cumsum[2], cumsum[3])

        red_m = methylated[:, ci.meth_t1r]
        red_u = unmethylated[:, ci.meth_t1r]
        red_2 = unmethylated[:, ci.meth_t2]

        xf_red = np.concatenate([red_m, red_u, red_2], axis=1)
        xs_red = normexp_get_xs(xf_red, controls=red_oob, offset=offset)

        cumsum = np.cumsum([0, red_m.shape[1], red_u.shape[1], red_2.shape[1]])
        slice_red_m = slice(cumsum[0], cumsum[1])
        slice_red_u = slice(cumsum[1], cumsum[2])
        slice_red_2 = slice(cumsum[2], cumsum[3])

        methylated[:, ci.meth_t1g] = xs_grn["xs"][:, slice_grn_m]
        unmethylated[:, ci.meth_t1g] = xs_grn["xs"][:, slice_grn_u]

        methylated[:, ci.meth_t1r] = xs_red["xs"][:, slice_red_m]
        unmethylated[:, ci.meth_t1r] = xs_red["xs"][:, slice_red_u]

        methylated[:, ci.meth_t2] = xs_grn["xs"][:, slice_grn_2]
        unmethylated[:, ci.meth_t2] = xs_grn["xs"][:, slice_grn_2]

        # Dye correction

        grn_control = self.green[:, ci.ctrl_idx]
        red_control = self.red[:, ci.ctrl_idx]

        xcs_grn = normexp_get_xs(
            grn_control, param=xs_grn["param"], offset=offset
        )
        xcs_red = normexp_get_xs(
            red_control, param=xs_red["param"], offset=offset
        )
```

--------------------------------

### Combine sample columns into labels

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/utils.html

Demonstrates how to join multiple columns into a single string label per sample using a specified separator.

```python
>>> idat_handler.features(columns=["GEO", "CNVs"])
sample_1    SGT_103|Balanced
sample_2    SGT_056|Balanced
sample_3    SGT_276|Balanced
dtype: object
```

--------------------------------

### Run Mepylome with custom directories

Source: https://mepylome.readthedocs.io/en/latest/gui_cli.html

Specifies custom paths for analysis and reference directories using the -a and -r flags.

```bash
mepylome -a ~/mepylome/tutorial/tutorial_analysis/ -r ~/mepylome/tutorial/tutorial_reference/
```

--------------------------------

### Initialize MethylAnalysis

Source: https://mepylome.readthedocs.io/en/latest/tutorial.html

Configure the analysis environment by specifying custom output directories and optional parameters for validation, preprocessing, and data selection.

```python
>>> OUTPUT_DIR = DIR / "output_dir"
>>> OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
>>> methyl_analysis = MethylAnalysis(
>>>     analysis_dir=ANALYSIS_DIR,
>>>     reference_dir=REFERENCE_DIR,
>>>     output_dir=OUTPUT_DIR,
>>> )
```

```python
>>> TEST_DIR = DIR / "test_dir"
>>> methyl_analysis = MethylAnalysis(
>>>     analysis_dir=ANALYSIS_DIR,
>>>     reference_dir=REFERENCE_DIR,
>>>     output_dir=OUTPUT_DIR,
>>>     # New cases for validation, excluded from classifier training.
>>>     test_dir=TEST_DIR,
>>>     # Load beta values for all CpG sites into memory
>>>     load_full_betas=True,
>>>     # Use SWAN preprocessing method
>>>     prep="swan",
>>>     # Provide annotation file (if not already in analysis_dir)
>>>     annotation=ANALYSIS_DIR / "annotation.csv",
>>>     # Number of CpGs for UMAP analysis
>>>     n_cpgs=5000,
>>>     # Analyze the 'best' CpG sites
>>>     cpg_selection="top",
>>>     # Show segmentation intervals in CNV plot
>>>     do_seg=True,
>>> )
```

--------------------------------

### Initialize MethylAnalysis Object

Source: https://mepylome.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Creates an instance of MethylAnalysis using specified analysis and reference directories.

```python
>>> methyl_analysis = MethylAnalysis(
>>>     analysis_dir=ANALYSIS_DIR,
>>>     reference_dir=REFERENCE_DIR,
>>> )
```

--------------------------------

### Run Mepylome via CLI

Source: https://mepylome.readthedocs.io/en/latest/_sources/gui_cli.rst.txt

Execute the Mepylome tool using default settings or custom analysis and reference paths.

```sh
mepylome
```

```sh
mepylome -a ~/mepylome/tutorial/tutorial_analysis/ -r ~/mepylome/tutorial/tutorial_reference/
```

--------------------------------

### Print manifest data overview

Source: https://mepylome.readthedocs.io/en/latest/tutorial.html

Display the contents of the probe data frame.

```python
>>> print(probes_df)
            IlmnID  AddressA_ID  AddressB_ID  ...  N_CpG    End  Probe_Type
0       cg13869341     62703328     16661461  ...      2  15865           1
1       cg14008030     27651330           -1  ...      2  18827           2
2       cg12045430     25703424     34666387  ...      7  29407           1
3       cg20826792     61731400     14693326  ...      7  29425           1
4       cg00381604     26752380     50693408  ...      6  29435           1
...            ...          ...          ...  ...    ...    ...         ...
485572   rs1416770     28667385           -1  ...      0     -1           4
485573   rs1941955     33709340           -1  ...      0     -1           4
485574   rs2125573     25698376           -1  ...      0     -1           4
485575   rs2521373     12625304           -1  ...      0     -1           4
485576   rs4331560     10654345           -1  ...      0     -1           4

[485577 rows x 12 columns]
```

--------------------------------

### Parse an IDAT file using IdatParser

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/idat.html

Demonstrates how to initialize the IdatParser with a file path and access the parsed data.

```python
>>> filepath = "/path/to/idat/file_Grn.idat"
>>> idat_data = IdatParser(filepath)
>>> ids = idat_data.illumina_ids
>>> print(idat_data)
```

--------------------------------

### Initialize CNV object with set_all

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/cnv.html

Convenience method to create a CNV object and perform the full analysis pipeline in one call.

```python
>>> cnv = CNV.set_all(sample, reference, do_seg=do_seg)
>>> # Note: This command is equivalent to:
>>> cnv = CNV(sample, reference)
>>> cnv.set_bins()
>>> cnv.set_detail()
>>> if do_seg:
>>>     cnv.set_segments()
```

--------------------------------

### Initialize RawData

Source: https://mepylome.readthedocs.io/en/latest/tutorial.html

Initialize the RawData class using a base path, a specific IDAT file, or a partial path.

```python
>>> idat_file = ANALYSIS_DIR / "200925700125_R07C01_Red.idat"
>>> # or
>>> idat_file = ANALYSIS_DIR / "200925700125_R07C01_Grn.idat"
>>> # or
>>> idat_file = ANALYSIS_DIR / "200925700125_R07C01"
>>> raw_data = RawData(idat_file)
```

--------------------------------

### Manage Analysis Directories and Dependencies

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/analysis/core.html

Handles the creation of UMAP and classifier directories based on input hashes and resets cached beta values when dependencies change.

```python
# umap dir
cur_vars = self._get_vars_or_hashes()
umap_dir_hash_key = input_args_id(
    self.analysis_dir,
    "umap",
    self.annotation,
)
umap_plot_hash_key = input_args_id(
    self.analysis_dir,
    "umap",
    self.prep,
    self.n_cpgs,
    self.cpg_selection,
    self.balancing_feature,
    extra_hash=[
        cur_vars["cpgs"],
        cur_vars["test_files"],
        self.annotation,
    ],
)
self.umap_dir = self.output_dir / f"{umap_dir_hash_key}"
self.umap_plot_path = self.umap_dir / f"{umap_plot_hash_key}.csv"
ensure_directory_exists(self.umap_dir)

# clf dir
clf_label = re.sub(
    r"\W+", "", "_".join(self.idat_handler.selected_columns)
)
if self.feature_matrix is not None:
    features = self.feature_matrix
elif self.load_full_betas:
    features = self.load_full_betas
else:
    features = self.n_cpgs
clf_hash_key = input_args_id(
    self.analysis_dir,
    "clf",
    clf_label,
    features,
    self.prep,
    extra_hash=[
        cur_vars["cpgs"],
        self.annotation,
        self.analysis_ids,
    ],
)
self.clf_dir = self.output_dir / f"{clf_hash_key}"

# Reset betas_sel if necessary
dependencies = [
    "analysis_dir",
    "cpg_selection",
    "cpgs",
    "n_cpgs",
    "prep",
    "analysis_ids",
    "test_ids",
    "test_files",
]
if any(self._prev_vars[arg] != cur_vars[arg] for arg in dependencies):
    self.betas_sel = None

# Reset betas_all if necessary
dependencies = [
    "analysis_dir",
    "cpgs",
    "prep",
    "analysis_ids",
    "test_ids",
    "test_files",
]
if any(self._prev_vars[arg] != cur_vars[arg] for arg in dependencies):
    self.betas_all = None

# Update variables/hashes
self._prev_vars = cur_vars
```

--------------------------------

### Load Multiple IDAT Files

Source: https://mepylome.readthedocs.io/en/latest/tutorial.html

Initialize RawData with a list of multiple IDAT files or an entire directory for batch processing.

```python
>>> idat_file0 = ANALYSIS_DIR / "200925700125_R07C01_Grn.idat"
>>> idat_file1 = ANALYSIS_DIR / "200925700133_R02C01"

>>> raw_data_2 = RawData([idat_file0, idat_file1])
```

```python
>>> raw_data_all = RawData(REFERENCE_DIR)
```

--------------------------------

### CNV.__init__

Source: https://mepylome.readthedocs.io/en/latest/_modules/mepylome/dtypes/cnv.html

Initializes a new CNV object by comparing a sample against a reference dataset.

```APIDOC
## CNV(sample, reference, annotation=None)

### Description
Initializes the CNV analysis object. Validates that the sample contains exactly one ID and that the array types of the sample, reference, and annotation match.

### Parameters
- **sample** (MethylData) - Required - The sample data to analyze.
- **reference** (MethylData | ReferenceMethylData) - Required - The reference data for comparison.
- **annotation** (Annotation) - Optional - Genomic annotation information.
```
Tokens:
Raw