### Install Zram on Debian

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Installs the necessary zram-tools package on Debian-based Linux distributions using the apt package manager. This is the first step in setting up zram.

```bash
sudo apt install zram-tools
```

--------------------------------

### Install bblean from source (Bash)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/installing.rst

Installs the bblean package from source in editable mode within a conda environment. It requires Python 3.11 or newer and uses an environment.yaml file for setup. The `bb --help` command is used to verify the installation.

```bash
conda env create --file ./environment.yaml
conda activate bblean

pip install -e .

bb --help
```

--------------------------------

### Install BitBIRCH-Lean Package

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

Installs the BitBIRCH-Lean package from its GitHub repository using pip. This is a prerequisite for running the workflow.

```bash
git clone https://github.com/mqcomplab/bblean.git
cd bblean
pip install -v .
```

--------------------------------

### Install bblean with C++ extensions from source (Bash)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/installing.rst

Installs the bblean package from source in editable mode, enabling optional C++ extensions. This requires a Linux x86 system and is intended to provide a speedup of approximately 1.8-2.0x. The `BITBIRCH_BUILD_CPP=1` environment variable is used to trigger the build.

```bash
BITBIRCH_BUILD_CPP=1 pip install -e .
```

--------------------------------

### Install Libraries for Dataset Splitting

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Installs necessary Python packages for dataset splitting and analysis, including useful_rdkit_utils, tqdm, statsmodels, and lightgbm.

```python
import itertools
import time

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from tqdm.auto import tqdm
from rdkit import Chem
from rdkit.Chem import rdFingerprintGenerator, AllChem
from rdkit.DataStructs import BulkTanimotoSimilarity, ExplicitBitVect
from rdkit.ML.Cluster import Butina
from statsmodels.stats.multicomp import pairwise_tukeyhsd
from sklearn.metrics import r2_score, mean_absolute_error
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

from lightgbm import LGBMRegressor
import useful_rdkit_utils as uru
import bblean
import bblean.fingerprints
```

--------------------------------

### GPL Compliance Notice for Interactive Programs (Plaintext)

Source: https://github.com/mqcomplab/bblean/blob/main/LICENSES/GPL-3.0-only.txt

Example of a short notice to be displayed by a program when it starts in interactive mode, informing users about its licensing under the GPL. This notice should direct users to commands (e.g., 'show w' for warranty, 'show c' for conditions) that reveal the relevant sections of the license. This is a template and should be adapted to the program's specific commands or interface.

```plaintext
{{ project }}  Copyright (C) {{ year }}  {{ organization }}
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
```

--------------------------------

### Enable Zswap

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Enables zswap for the current session by writing '1' to the zswap enabled parameter. This utilizes the zswap memory compression feature.

```bash
echo 1 | sudo tee /sys/module/zswap/parameters/enabled
```

--------------------------------

### Create Zram Swap Device

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Creates a swap device on /dev/zram0 using the mkswap command. This prepares the zram device to be used as a swap area.

```bash
sudo mkswap /dev/zram0
```

--------------------------------

### Configure Zram System Settings

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Adds recommended zram-related kernel parameters to the sysctl configuration file for persistent settings. These parameters help optimize memory management with zram.

```bash
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
```

--------------------------------

### Verify Zram Activation

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Checks the active swap devices on the system using the 'swapon' command. The output should include an entry for /dev/zram0 with a high priority.

```bash
swapon
```

--------------------------------

### Configure Zswap Compressor

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Sets the compression algorithm for zswap by writing the desired algorithm (e.g., 'zstd') to the 'compressor' parameter. This affects how memory pages are compressed.

```bash
echo zstd | sudo tee /sys/module/zswap/parameters/compressor
```

--------------------------------

### Activate Zram Swap

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Activates the zram0 swap device with a high priority to ensure it's used before other swap files. The --discard option can help improve performance on SSDs.

```bash
sudo swapon --discard --priority 100 /dev/zram0
```

--------------------------------

### Configure Zswap Max Pool Percent

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Sets the maximum percentage of RAM that zswap is allowed to use for compression by writing the desired percentage (e.g., '30') to the 'max_pool_percent' parameter.

```bash
echo 30 | sudo tee /sys/module/zswap/parameters/max_pool_percent
```

--------------------------------

### Check Zswap Status

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Reads the 'enabled' parameter from the zswap module to determine if zswap is currently active. 'Y' indicates enabled, 'N' indicates disabled.

```bash
cat /sys/module/zswap/parameters/enabled
```

--------------------------------

### Get BitBirch Cluster Assignments

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_best_practices.ipynb

Retrieves the molecular IDs for each cluster generated by the BitBirch tree. This is used to analyze cluster sizes and count singletons.

```python
clusters = bb_tree.get_cluster_mol_ids()
print("Number of singletons", sum(1 for c in clusters if len(c) == 1))
```

--------------------------------

### Initialize, Fit, and Refine BitBirch Tree

Source: https://github.com/mqcomplab/bblean/blob/main/README.md

Demonstrates initializing a BitBirch tree with specified parameters, fitting it to fingerprint data (fps), and refining the tree in-place using a tolerance-diameter merge criterion. Assumes 'bblean', 'analysis', 'plotting', 'plt', 'pickle', and 'np' are imported and 'fps' and 'smiles' are defined.

```python
import bblean
import analysis
import plotting
import matplotlib.pyplot as plt
import pickle
import numpy as np

# Assume 'fps' and 'smiles' are defined numpy arrays or similar
# Example placeholder data (replace with actual data)
fps = np.random.rand(100, 2048).astype(np.uint8) 
smiles = [f'C{i}c1ccccc1' for i in range(100)]

# Initialize and fit the tree
tree = bblean.BitBirch(branching_factor=50, threshold=0.65, merge_criterion="diameter")
tree.fit(fps)

# Refine the tree
tree.set_merge(merge_criterion="tolerance-diameter", tolerance=0.0)
tree.refine_inplace(fps)

# Visualize the results
clusters = tree.get_cluster_mol_ids()
ca = analysis.cluster_analysis(clusters, fps, smiles)
plotting.summary_plot(ca, title="ChEMBL Sample")
plt.show()

# Save the results
with open("./clusters.pkl", "wb") as f:
    pickle.dump(clusters, f)
ca.dump_metrics("./metrics.csv")
np.save("./fps-packed-2048.npy", fps)
```

--------------------------------

### Disable Zram Swap

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/linux_memory_setup.rst

Deactivates the zram0 swap device using the 'swapoff' command. This is used to remove zram as an active swap area.

```bash
sudo swapoff /dev/zram0
```

--------------------------------

### Initialize and Fit BitBirch Clustering Model (Python)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

Initializes a BitBirch clustering object with specified parameters (branching factor, threshold, merge criterion) and then fits the model to the packed molecular fingerprints.

```python
# Initialize the BitBirch tree. In general, diameter is the best merge criterion for
# initial clustering. For more info consult the *refinement* paper.
# 0.5-0.65 is a good threshold range for "rdkit" fingerprints, for ecfp4 use 0.3-0.4 instead
bb_tree = bblean.BitBirch(branching_factor=50, threshold=0.65, merge_criterion="diameter")

# Cluster the packed fingerprints (By default all bblean functions take packed
# fingerprints)
bb_tree.fit(fps)
```

--------------------------------

### Visualize and Analyze Clustering Results (Python)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

Visualizes the clustering results using a summary plot and performs cluster analysis. It also shows how to save cluster metrics to a CSV file and inspect various cluster properties like the total number of clusters and singletons.

```python
# First we run a cluster analysis on the resulting ids
clusters = bb_tree.get_cluster_mol_ids()
ca = analysis.cluster_analysis(clusters, fps, smiles)

# Afterwards we can use the utility functions on the bblean.plotting module
plotting.summary_plot(ca, title="Diameter")

# Optionally we can save the cluster analysis metrics as a csv file
ca.dump_metrics("./diameter-metrics.csv")
plt.show()

# Total clusters
print("Number of clusters: ", ca.all_clusters_num)

# Clusters with more than 10 molecules
print("Number of clusters with more than 10 molecules: ", ca.all_clusters_num_with_size_above(10))

# Singletons (clusters with a single element)
print("Number of singletons", ca.all_singletons_num)
```

--------------------------------

### Load SMILES and Compute Fingerprints (Python)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

Loads molecular structures from a SMILES file and computes molecular fingerprints. It demonstrates the use of 'rdkit' kind fingerprints and shows the shape and data type of the resulting packed fingerprints.

```python
smiles = bblean.load_smiles("./chembl-33-natural-products-subset.smi")

# By default the fps created are of the "ecfp4" kind. Here we use "rdkit"
fps = bblean.fps_from_smiles(smiles, pack=True, n_features=2048, kind="rdkit")
print(f"Shape: {fps.shape}, DType: {fps.dtype}")
```

--------------------------------

### Pack and Unpack Fingerprints (Python)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

Demonstrates converting between packed and unpacked molecular fingerprint representations using bblean. This is crucial for efficient storage and manipulation of large fingerprint datasets.

```python
fps_unpacked = bblean.unpack_fingerprints(fps)
print(f"Shape unpacked: {fps_unpacked.shape}, DType unpacked: {fps_unpacked.dtype}")

fps = bblean.pack_fingerprints(fps_unpacked)
print(f"Shape re-packed: {fps.shape}, DType re-packed: {fps.dtype}")
```

--------------------------------

### Import BitBIRCH Modules (Python)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

Imports the main bblean package and essential submodules for plotting and analysis. These are necessary for all subsequent operations.

```python
import bblean
import bblean.plotting as plotting
import bblean.analysis as analysis
```

--------------------------------

### Visualize and Analyze Refined Clustering Results

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

This snippet shows how to obtain cluster assignments after refinement, perform cluster analysis, and generate a summary plot. It also includes saving the cluster analysis metrics to a CSV file and displaying the plot. This helps in visually assessing the quality of the refined clustering.

```python
clusters_refined = bb_tree.get_cluster_mol_ids()
ca_refined = analysis.cluster_analysis(clusters_refined, fps, smiles)
plotting.summary_plot(ca_refined, title="Refined")

# Again, we can save the cluster analysis metrics as a csv file
ca.dump_metrics("./refined-metrics.csv")
plt.show()
```

--------------------------------

### Inspect Cluster Features

Source: https://github.com/mqcomplab/bblean/blob/main/examples/bitbirch_quickstart.ipynb

Examines key features of the generated clusters from the cluster analysis object. This includes retrieving the total number of clusters, the number of clusters with more than a specified size, and the number of singleton clusters.

```python
# Total clusters
print("Number of clusters: ", ca.all_clusters_num)

# Clusters with more than 10 molecules
print("Number of clusters with more than 10 molecules: ", ca.all_clusters_num_with_size_above(10))

# Singletons (clusters with a single element)
print("Number of singletons", ca.all_singletons_num)
```

--------------------------------

### GPL License Header for Source Files (Various Languages)

Source: https://github.com/mqcomplab/bblean/blob/main/LICENSES/GPL-3.0-only.txt

Standard header to be included at the beginning of each source file when licensing a program under the GNU General Public License (GPL). This ensures clear attribution, copyright information, and a reference to the full license terms. It is recommended to include this in every file for maximum clarity on warranty exclusions.

```plaintext
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year>  <name of author>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
```

--------------------------------

### Count and Analyze Refined Clusters

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

This snippet provides Python code to print the total number of clusters, the number of clusters containing more than 10 molecules, and the number of singleton clusters (clusters with a single element) from the refined analysis results. This offers quantitative insights into the clustering structure.

```python
# Total clusters
print("Number of clusters: ", ca_refined.all_clusters_num)

# Clusters with more than 10 molecules
print("Number of clusters with more than 10 molecules: ", ca_refined.all_clusters_num_with_size_above(10))

# Singletons (clusters with a single element)
print("Number of singletons", ca_refined.all_singletons_num)
```

--------------------------------

### Visualize and Analyze Clustering Results

Source: https://github.com/mqcomplab/bblean/blob/main/examples/bitbirch_quickstart.ipynb

Visualizes the clustering results using bblean.plotting and performs cluster analysis with bblean.analysis. It generates a summary plot and can save cluster metrics to a CSV file. The results are displayed using matplotlib.

```python
# First we run a cluster analysis on the resulting ids
clusters = bb_tree.get_cluster_mol_ids()
ca = analysis.cluster_analysis(clusters, fps, smiles)

# Afterwards we can use the utility functions on the bblean.plotting module
plotting.summary_plot(ca, title="Diameter")

# Optionally we can save the cluster analysis metrics as a csv file
ca.dump_metrics("./diameter-metrics.csv")
plt.show()
```

--------------------------------

### BitBirch Clustering - Advanced Configuration - Python

Source: https://context7.com/mqcomplab/bblean/llms.txt

Demonstrates advanced configuration options for BitBIRCH clustering, including setting different merge criteria like 'tolerance-diameter' with a specified tolerance. It also shows how to modify these parameters after the initial fitting process.

```python
import bblean

smiles = bblean.load_smiles("examples/chembl-33-natural-products-subset.smi")
fps = bblean.fps_from_smiles(smiles, kind="ecfp4", n_features=2048, pack=True)

# Create tree with tolerance-diameter merge criterion
tree = bblean.BitBirch(
    threshold=0.35,
    branching_factor=50,
    merge_criterion="tolerance-diameter",
    tolerance=0.05
)

tree.fit(fps, input_is_packed=True, n_features=2048)
print(f"Initial clusters: {len(tree.get_cluster_mol_ids())}")

# Change merge criterion after fitting
tree.set_merge(
    criterion="tolerance-diameter",
    tolerance=0.0,
    threshold=0.40
)

```

--------------------------------

### Load Clustering Results in Python

Source: https://github.com/mqcomplab/bblean/blob/main/README.md

This Python snippet demonstrates how to load and inspect clustering results saved by the BitBIRCH-Lean CLI. It uses the `pickle` module to load the `clusters.pkl` file, which contains lists of molecule indices for each cluster. These indices can be used to link back to the original molecular data.

```python
import pickle

clusters = pickle.load(open("bb_run_outputs/504e40ef/clusters.pkl", "rb"))
clusters[:2]

```

--------------------------------

### Print Training and Test Set Sizes

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Iterates through a dictionary containing split information and prints the sizes of the training and testing sets for each split method. This helps in understanding the distribution of data across splits. It assumes the `split_dict` is already populated.

```python
for k, v in split_dict.items():
    print(k, len(v[0]), len(v[1]))
```

--------------------------------

### Fit BitBirch Tree

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_best_practices.ipynb

Fits a BitBirch tree model to the provided fingerprints. This is the initial step in the clustering process.

```python
bb_tree.fit(fps)
```

--------------------------------

### Prepare Data and Perform Cross-Validation

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

This code prepares a dataset by reading a CSV, generating molecular fingerprints, and then performs 5x5 cross-validation using different splitting strategies. It records the size of the test set for each split and strategy.

```python
size_df = pd.read_csv("biogen_logS.csv")
size_df["mol"] = size_df.SMILES.apply(Chem.MolFromSmiles)
fpgen = rdFingerprintGenerator.GetMorganGenerator()
size_df["fp"] = size_df.mol.apply(fpgen.GetCountFingerprintAsNumPy)
size_df["binary_fps"] = size_df.mol.apply(Chem.RDKFingerprint)

SPLIT_LIST = [
    "random_cluster",
    "butina_cluster",
    "umap_cluster",
    "scaffold_cluster",
    "bitbirch_cluster",
]
split_fn_dict = {
    "random_cluster": uru.get_random_clusters,
    "butina_cluster": uru.get_butina_clusters,
    "umap_cluster": uru.get_umap_clusters,
    "scaffold_cluster": uru.get_bemis_murcko_clusters,
    "bitbirch_cluster": get_bitbirch_clusters,
}

result_list = []
for split in SPLIT_LIST:
    for i in tqdm(range(0, 5), desc=split):
        cluster_list = split_fn_dict[split](size_df.SMILES)
        group_kfold_shuffle = uru.GroupKFoldShuffle(n_splits=5, shuffle=True)
        if split == "bitbirch_cluster":
            for train, test in group_kfold_shuffle.split(
                np.stack(size_df.binary_fps), size_df.logS, cluster_list
            ):
                result_list.append([split, len(test)])
        else:
            for train, test in group_kfold_shuffle.split(
                np.stack(size_df.fp), size_df.logS, cluster_list
            ):
                result_list.append([split, len(test)])
result_df = pd.DataFrame(result_list, columns=["split", "num_test"])
```

--------------------------------

### Generate Fingerprints using bblean CLI

Source: https://context7.com/mqcomplab/bblean/llms.txt

Demonstrates how to use the bblean command-line interface (CLI) to convert SMILES files into packed fingerprint arrays. This is a common preprocessing step before clustering. The command specifies input SMILES file, output file path, fingerprint kind, and other options.

```bash
bblean fps --input examples/chembl-33-natural-products-subset.smi --output fps.npz --kind ecfp4 --pack --n-features 2048
```

--------------------------------

### Visualize Clustering Results using CLI

Source: https://github.com/mqcomplab/bblean/blob/main/README.md

These CLI commands visualize the results of molecular clustering. `bb plot-summary` displays a summary of the largest clusters, optionally including Murcko scaffold analysis if a SMILES file is provided. `bb plot-tsne` generates a t-SNE visualization for exploring cluster structures, utilizing the openTSNE library for high performance.

```bash
bb plot-summary <output-path> --top 20
bb plot-tsne <output-path> -- top 20

```

--------------------------------

### Cluster Analysis and Summary Plot

Source: https://github.com/mqcomplab/bblean/blob/main/examples/bitbirch_best_practices.ipynb

Performs a cluster analysis on the fingerprints and clusters, then generates a summary plot. This involves using `analysis.cluster_analysis` and `plotting.summary_plot` from the bblean library.

```python
# First we run a cluster analysis on the resulting ids
clusters = bb_tree.get_cluster_mol_ids()
ca = analysis.cluster_analysis(clusters, fps, smiles)

# Afterwards we can use the utility functions on the bblean.plotting module
plotting.summary_plot(ca, title="Diameter")
plt.show()
```

--------------------------------

### Cluster Fingerprints using CLI (Serial Mode)

Source: https://github.com/mqcomplab/bblean/blob/main/README.md

This CLI command clusters pre-generated molecular fingerprints in serial mode. It takes the path to a fingerprint file or directory as input. The outputs, including cluster information, are stored in a uniquely identified directory. Users can control clustering parameters like branching factor and threshold, and optionally refine clusters.

```bash
bb run ./packed-fps-uint8-508e53ef.npy

```

--------------------------------

### Initialize BitBirch Clustering (Python)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_best_practices.ipynb

Initializes the BitBirch clustering algorithm with specified parameters. The threshold is typically set as the average similarity plus a multiple of the standard deviation. The 'diameter' is recommended as the merge criterion.

```python
# Initialize the BitBirch tree. In general, diameter is the best merge criterion for
optimal_threshold = average_sim + 4 * std
bb_tree = bblean.BitBirch(branching_factor=50, threshold=optimal_threshold, merge_criterion="diameter")

# Cluster the packed fingerprints (By default all bblean functions take packed
```

--------------------------------

### BitBirch Clustering - Basic Usage - Python

Source: https://context7.com/mqcomplab/bblean/llms.txt

Performs hierarchical clustering of molecular fingerprints using the BitBIRCH algorithm. This basic usage demonstrates fitting fingerprints to a tree with default parameters and retrieving cluster assignments and molecule IDs.

```python
import bblean
import numpy as np

# Load SMILES and generate fingerprints
smiles = bblean.load_smiles("examples/chembl-33-natural-products-subset.smi")
fps = bblean.fps_from_smiles(smiles, kind="rdkit", n_features=2048, pack=True)

# Create BitBirch tree with default parameters
# threshold: 0.5-0.65 for rdkit, 0.3-0.4 for ecfp4/ecfp6
tree = bblean.BitBirch(
    threshold=0.65,
    branching_factor=50,
    merge_criterion="diameter"
)

# Fit the fingerprints
tree.fit(fps, input_is_packed=True, n_features=2048)

# Get clustering results
clusters = tree.get_cluster_mol_ids()
print(f"Created {len(clusters)} clusters")
print(f"Largest cluster size: {len(clusters[0])}")
print(f"First cluster molecule indices: {clusters[0][:10]}")

# Get cluster assignments (label for each molecule)
assignments = tree.get_assignments()
print(f"Molecule 0 is in cluster {assignments[0]}")
```

--------------------------------

### Generate Images for Individual Clusters

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

This Python function call generates images for molecules within a specified cluster (cluster_idx=10). It uses the provided SMILES strings and cluster assignments to create visual representations, with each image containing up to 30 molecules by default. This is useful for detailed inspection of cluster contents.

```python
plotting.dump_mol_images(smiles, clusters_refined, cluster_idx=10)
```

--------------------------------

### Configure ML Model and Splitting Methods for Cross-Validation

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

This code block defines the machine learning model and the list of dataset splitting methods to be used in the cross-validation process. It sets up the necessary configurations before calling the cross_validate function.

```python
model_list = [("lgbm_morgan", LGBMMorganCountWrapper)]
group_list = [
    ("butina", uru.get_butina_clusters),
    ("random", uru.get_random_clusters),
    ("scaffold", uru.get_bemis_murcko_clusters),
    ("umap", uru.get_umap_clusters),
    ("bitbirch", get_bitbirch_clusters),
]
y = "logS"
```

--------------------------------

### Python API: Save and Load Clustering Results

Source: https://context7.com/mqcomplab/bblean/llms.txt

Demonstrates how to use the BitBirch-Lean Python API to perform clustering, save fingerprints and cluster assignments, and then load them back. This includes saving fingerprints as NumPy arrays, cluster assignments as pickle files or CSV, and the entire BitBirch tree object.

```python
import bblean
import pickle
import numpy as np

# Perform clustering
smiles = bblean.load_smiles("examples/chembl-33-natural-products-subset.smi")
fps = bblean.fps_from_smiles(smiles, pack=True)

tree = bblean.BitBirch(threshold=0.65, branching_factor=50)
tree.fit(fps)

# Save fingerprints
np.save("fingerprints.npy", fps)

# Save clusters
clusters = tree.get_cluster_mol_ids()
with open("clusters.pkl", "wb") as f:
    pickle.dump(clusters, f)

# Save cluster assignments as CSV
tree.dump_assignments("assignments.csv", smiles=smiles)

# Save the entire tree (warning: large files for big datasets)
import sys
sys.setrecursionlimit(100000)
with open("bitbirch_tree.pkl", "wb") as f:
    pickle.dump(tree, f)

# Load results later
with open("clusters.pkl", "rb") as f:
    loaded_clusters = pickle.load(f)

fps_loaded = np.load("fingerprints.npy")
print(f"Restored {len(loaded_clusters)} clusters and {len(fps_loaded)} fingerprints")
```

--------------------------------

### Generate Fingerprints from SMILES using CLI

Source: https://github.com/mqcomplab/bblean/blob/main/README.md

This command generates molecular fingerprints from a SMILES file using the `bb` CLI. It outputs a packed fingerprint array, optimized for memory efficiency. The output filename includes a unique identifier, and users can specify output directories and names. Parallel processing is supported by splitting the input into multiple parts.

```bash
bb fps-from-smiles examples/chembl-33-natural-products-sample.smi

```

--------------------------------

### Cluster Fingerprints using CLI (Parallel Mode)

Source: https://github.com/mqcomplab/bblean/blob/main/README.md

This CLI command clusters molecular fingerprints in parallel mode, suitable for large datasets. It iteratively merges sub-trees in intermediate rounds when pointed to a directory containing multiple fingerprint files. Outputs are typically written to a dedicated directory structure.

```bash
bb multiround ./file-or-dir

```

--------------------------------

### Import bblean Modules (Python)

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_best_practices.ipynb

Imports necessary modules from the bblean package and standard libraries for data manipulation and plotting. These are required for subsequent operations.

```python
import bblean
import bblean.plotting as plotting
import bblean.analysis as analysis
import bblean.similarity as iSIM
import numpy as np
import matplotlib.pyplot as plt
```

--------------------------------

### Data Preparation for Chembl Dataset Clustering

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Loads and prepares molecular fingerprint data from a subset of the Chembl library for clustering. It loads 20,000 SMILES strings, generates fingerprints, and converts them to RDKit ExplicitBitVects.

```python
chembl_smis = bblean.load_smiles("./chembl-33-natural-products-subset.smi")[:20_000]
fps = bblean.fingerprints

chembl_rdkit_fp_list = [numpy_to_explicit_bitvect(fp) for fp in chembl_fps]
```

--------------------------------

### Generate Fingerprints from SMILES (ECFP4)

Source: https://context7.com/mqcomplab/bblean/llms.txt

Generates Extended Connectivity Fingerprints (ECFP4) from a SMILES file. This command creates fingerprint files and packs them for efficient storage and processing. It can be configured with the desired number of features and an output directory.

```bash
bb fps-from-smiles examples/chembl-33-natural-products-subset.smi \
  --kind ecfp4 \
  --n-features 2048 \
  --out-dir ./fingerprints \
  --name chembl-subset \
  --pack
```

--------------------------------

### LGBM Morgan Count Wrapper Class for RDKit Utilities

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

A Python class that wraps a LightGBM regressor for use with RDKit utilities. It handles Morgan count fingerprint generation and provides fit, predict, and validate methods for machine learning tasks on chemical data.

```python
class LGBMMorganCountWrapper:
    def __init__(self, y_col):
        self.lgbm = LGBMRegressor(verbose=-1)
        self.y_col = y_col
        self.fp_name = "fp"
        self.fg = rdFingerprintGenerator.GetMorganGenerator(radius=2, fpSize=1024)

    def fit(self, train):
        train["mol"] = train.SMILES.apply(Chem.MolFromSmiles)
        train[self.fp_name] = train.mol.apply(self.fg.GetCountFingerprintAsNumPy)
        self.lgbm.fit(np.stack(train.fp), train[self.y_col])

    def predict(self, test):
        test["mol"] = test.SMILES.apply(Chem.MolFromSmiles)
        test[self.fp_name] = test.mol.apply(self.fg.GetCountFingerprintAsNumPy)
        pred = self.lgbm.predict(np.stack(np.stack(test[self.fp_name])))
        return pred

    def validate(self, train, test):
        self.fit(train)
        pred = self.predict(test)
        return pred
```

--------------------------------

### CLI Clustering (Serial and Refined)

Source: https://context7.com/mqcomplab/bblean/llms.txt

Performs hierarchical clustering on pre-generated fingerprints using the command-line interface. Supports serial processing with specified similarity thresholds, branching factors, and merge criteria. Can also include a refinement step to improve cluster quality.

```bash
# Serial clustering
bb run ./fingerprints/packed-fps-uint8-chembl-subset.npy \
  --threshold 0.35 \
  --branching 50 \
  --merge-criterion tolerance-diameter \
  --tolerance 0.05 \
  --out-dir ./clustering_results

# With refinement
bb run ./fingerprints/packed-fps-uint8-chembl-subset.npy \
  --threshold 0.35 \
  --branching 50 \
  --refine-num 1 \
  --refine-threshold-change 0.05
```

--------------------------------

### Refine BitBirch Tree with New Merge Criterion

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

This snippet demonstrates how to set a new merge criterion ('tolerance-diameter' with a tolerance of 0.0) for a BitBirch tree and then refine the tree in-place using the specified criterion and provided features (fps). This is useful for breaking apart large clusters and regenerating the tree.

```python
# Modify the merge criteria for the tree, from now on bb_tree will use this new criteria
bb_tree.set_merge(criterion="tolerance-diameter", tolerance=0.0)

# Refine the tree using the new merge criteria
bb_tree.refine_inplace(fps)
```

--------------------------------

### Visualize R-squared Distributions with Boxplot

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

This code generates a boxplot to visually compare the distributions of R-squared values across different splitting methods. This helps in identifying which splitting strategies yield more consistent or higher model performance.

```python
sns.set_style("whitegrid")
ax = sns.boxplot(x="split", y="r2", hue="split", data=out_df)
ax.set_ylabel("$R^2$")
ax.set_xlabel("Split")
```

--------------------------------

### CLI Visualization (Summary Plot)

Source: https://context7.com/mqcomplab/bblean/llms.txt

Generates a summary plot from clustering results, highlighting the top N clusters. This command can associate clusters with SMILES strings to provide representative structures and save the plot to a file. Useful for quickly understanding the main clusters in a dataset.

```bash
bb plot-summary ./clustering_results \
  --smiles examples/chembl-33-natural-products-subset.smi \
  --top 20 \
  --title "ChEMBL Analysis" \
  --save \
  --filename summary.pdf
```

--------------------------------

### Save Final Cluster Assignments

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_quickstart.ipynb

This Python function saves the final cluster assignments generated by BitBirch to a CSV file named 'smiles-assignments.csv'. It requires the BitBirch tree object and the list of SMILES strings as input, ensuring that the clustering results can be easily shared and used in subsequent analyses.

```python
bb_tree.dump_assignments("smiles-assignments.csv", smiles)
```

--------------------------------

### Generate and Pack Fingerprints in Python

Source: https://github.com/mqcomplab/bblean/blob/main/README.md

This Python code snippet shows how to use the `bblean` library to load SMILES data and generate molecular fingerprints. The fingerprints are then packed into a NumPy array for memory efficiency, which is the required format for most `bblean` functions. Users can specify the number of features and the fingerprinting algorithm.

```python
import bblean
import numpy as np

smiles = bblean.load_smiles("./examples/chembl-33-natural-products-sample.smi")
fps = bblean.fps_from_smiles(smiles, pack=True, n_features=2048, kind="rdkit")

```

--------------------------------

### Generate Bemis-Murcko Scaffold Clusters for Molecules

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Identifies Bemis-Murcko frameworks within molecules and assigns each unique scaffold to a cluster. This approach leverages RDKit's scaffold identification capabilities for clustering.

```python
df["scaffold_cluster"] = uru.get_bemis_murcko_clusters(df.SMILES)
```

--------------------------------

### Dump BitBirch Cluster Images

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_best_practices.ipynb

Generates and dumps images for molecules within a specified cluster. This allows for visual inspection of individual clusters, with options to control the number of molecules per image.

```python
plotting.dump_mol_images(smiles, clusters, cluster_idx=0)
```

--------------------------------

### Perform Cluster Analysis with bblean

Source: https://context7.com/mqcomplab/bblean/llms.txt

Analyzes clustering results using bblean's analysis module. It calculates various metrics such as total molecules, number and size of clusters, singletons, iSIM, and unique scaffold counts for the top N clusters. Requires loaded SMILES, fingerprints, and a fitted BitBirch tree. Outputs metrics to a CSV file.

```python
import bblean
import bblean.analysis as analysis

# Load data and cluster
smiles = bblean.load_smiles("examples/chembl-33-natural-products-subset.smi")
fps = bblean.fps_from_smiles(smiles, kind="rdkit", pack=True)

tree = bblean.BitBirch(threshold=0.65, branching_factor=50)
tree.fit(fps)
clusters = tree.get_cluster_mol_ids()

# Perform cluster analysis with top 20 largest clusters
ca = analysis.cluster_analysis(
    clusters=clusters,
    fps=fps,
    smiles=smiles,
    top=20,
    input_is_packed=True,
    n_features=2048
)

# Access analysis results
print(f"Total molecules: {ca.total_fps}")
print(f"Total clusters: {ca.all_clusters_num}")
print(f"Mean cluster size: {ca.all_clusters_mean_size:.2f}")
print(f"Median cluster size: {ca.all_clusters_median_size}")
print(f"Singletons: {ca.all_singletons_num}")

# Per-cluster metrics
print("\nTop 5 clusters:")
for i in range(min(5, len(ca.sizes))):
    print(f"Cluster {ca.labels[i]}: {ca.sizes[i]} molecules, "
          f"iSIM={ca.isims[i]:.3f}, "
          f"unique scaffolds={ca.unique_scaffolds_num[i]}")

# Export metrics to CSV
ca.dump_metrics("cluster_metrics.csv")
```

--------------------------------

### Load SMILES from File - Python

Source: https://context7.com/mqcomplab/bblean/llms.txt

Loads SMILES strings from a specified .smi file into a numpy array. Supports loading the entire file or a subset of molecules up to a maximum number. This is the first step in processing molecular data for clustering.

```python
import bblean

# Load all SMILES from a file
smiles = bblean.load_smiles("examples/chembl-33-natural-products-subset.smi")
print(f"Loaded {len(smiles)} molecules")

# Load only first 1000 molecules
smiles_subset = bblean.load_smiles("examples/chembl-33-natural-products-subset.smi", max_num=1000)
print(f"Loaded subset: {len(smiles_subset)} molecules")
```

--------------------------------

### Implement BitBIRCH Clustering Algorithm

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

A Python function to compute BitBIRCH clusters from a list of SMILES strings. It generates Morgan fingerprints, fits a BitBirch model, and returns cluster labels for each molecule. Dependencies include RDKit, NumPy, and bblean.

```python
def get_bitbirch_clusters(smiles_list):
    mols = [Chem.MolFromSmiles(smiles) for smiles in smiles_list]
    fps = np.array([Chem.RDKFingerprint(mol) for mol in mols])
    bitbirch = bblean.BitBirch(branching_factor=50, threshold=0.65)
    bitbirch.fit(fps)
    cluster_list = bitbirch.get_cluster_mol_ids()

    #     # Map each mol ID to its cluster ID
    n_molecules = len(fps)
    cluster_labels = [0] * n_molecules
    for cluster_id, indices in enumerate(cluster_list):
        for idx in indices:
            cluster_labels[idx] = cluster_id
    return cluster_labels
```

--------------------------------

### Recluster BitBirch Tree In-place

Source: https://github.com/mqcomplab/bblean/blob/main/docs/src/user-guide/notebooks/bitbirch_best_practices.ipynb

Reclusters the BitBirch tree in-place to reduce the number of singletons. It allows for iterative refinement by adjusting the threshold and number of iterations. Verbose output shows progress.

```python
bb_tree.recluster_inplace(iterations=5, extra_threshold=std, shuffle=False, verbose=True)
```

--------------------------------

### Generate Cluster Summary Plot with bblean

Source: https://context7.com/mqcomplab/bblean/llms.txt

Creates a comprehensive summary plot visualizing cluster sizes, scaffold diversity, and similarity metrics using bblean's plotting module. This requires pre-computed cluster analysis results. The plot is saved as a PNG file.

```python
import bblean
import bblean.analysis as analysis
import bblean.plotting as plotting
import matplotlib.pyplot as plt

# Prepare data and clustering
smiles = bblean.load_smiles("examples/chembl-33-natural-products-subset.smi")
fps = bblean.fps_from_smiles(smiles, kind="rdkit", pack=True)

tree = bblean.BitBirch(threshold=0.65, branching_factor=50)
tree.fit(fps)
clusters = tree.get_cluster_mol_ids()

# Create cluster analysis
ca = analysis.cluster_analysis(clusters, fps, smiles, top=20, input_is_packed=True)

# Generate summary plot
fig, axes = plotting.summary_plot(
    ca,
    title="ChEMBL Natural Products",
    annotate=True
)
plt.savefig("cluster_summary.png", dpi=300, bbox_inches="tight")
plt.show()
```

--------------------------------

### Split Dataset using GroupKFoldShuffle

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Splits a dataset using the GroupKFoldShuffle cross-validation strategy. It iterates through predefined splitting methods and performs one round of cross-validation to obtain training and testing indices. The results are stored in a dictionary. Dependencies include a custom `uru` module and pandas DataFrame.

```python
split_dict = {}
for split in SPLIT_LIST:
    kf = uru.GroupKFoldShuffle(n_splits=5, shuffle=True)
    for train_idx, test_idx in kf.split(df, groups=df[split]):
        split_dict[split] = [train_idx, test_idx]
        break
```

--------------------------------

### Data Preparation for Clustering

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Converts molecular fingerprint data into a format suitable for clustering algorithms. It takes binary fingerprints from a pandas DataFrame and transforms them into a list of RDKit ExplicitBitVects.

```python
data = np.array(size_df.binary_fps.values.tolist())
rdkit_fp_list = [numpy_to_explicit_bitvect(fp) for fp in data]
```

--------------------------------

### Perform Cross-Validation and Save Results

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

This snippet executes the cross-validation process using the defined model and splitting configurations. The results, including training/test sets and predictions, are stored in a DataFrame and then saved to a CSV file.

```python
result_df = uru.cross_validate(df, model_list, y, group_list, 5, 5)outfile_name = "biogen_logS_results.csv"
result_df.to_csv(outfile_name, index=False)
```

--------------------------------

### BitBirch and Butina Clustering Performance (Biogen Dataset)

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Measures the time taken by BitBirch and Butina algorithms to cluster 2,173 molecules. BitBirch is initialized with specific branching factor and threshold values. Butina calculates pairwise similarities and then clusters the data.

```python
s_time = time.time()
bitbirch = bblean.BitBirch(branching_factor=50, threshold=0.65)
bitbirch.fit(data)
bitbirch_time = time.time() - s_time
print("Time for BitBIRCH Clustering (s):", bitbirch_time)

dists = []
nfps = len(rdkit_fp_list)
s_time = time.time()
for i in range(1, nfps):
    sims = BulkTanimotoSimilarity(rdkit_fp_list[i], rdkit_fp_list[:i])
    dists.extend([1 - x for x in sims])
cluster_res = Butina.ClusterData(dists, nfps, 0.65, isDistData=True)
butina_time = time.time() - s_time
print("Time for Butina Clustering (s):", butina_time)
```

--------------------------------

### Add Split Labels to Plotting DataFrame

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Adds training and testing set labels to a temporary DataFrame based on predefined splitting methods. It initializes all entries as 'train' and then marks the indices corresponding to the test set for each split. Assumes `split_dict` and `SPLIT_LIST` are defined.

```python
for split in SPLIT_LIST:
    tmp_df[split] = "train"
    _, test_idx = split_dict[split]
    for t in test_idx:
        tmp_df.loc[t, split] = "test"
```

--------------------------------

### BitBirch and Butina Clustering Performance (Chembl Dataset)

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

Measures the time taken by BitBirch and Butina algorithms to cluster 20,000 molecules from the Chembl library. BitBirch is fitted with input_is_packed set to False, indicating pre-computed fingerprints. Butina calculates pairwise similarities and clusters the data.

```python
s_time = time.time()
bitbirch = bblean.BitBirch(branching_factor=50, threshold=0.65)
bitbirch.fit(chembl_fps, input_is_packed=False)
bitbirch_time = time.time() - s_time
print("Time for BitBIRCH Clustering (s):", bitbirch_time)


dists = []
nfps = len(chembl_rdkit_fp_list)
s_time = time.time()
for i in range(1, nfps):
    sims = BulkTanimotoSimilarity(chembl_rdkit_fp_list[i], chembl_rdkit_fp_list[:i])
    dists.extend([1 - x for x in sims])
cluster_res = Butina.ClusterData(dists, nfps, 0.65, isDistData=True)
butina_time = time.time() - s_time
print("Time for Butina Clustering (s):", butina_time)
```

--------------------------------

### Visualize Test Set Size Variability

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

This code generates a box plot to visualize the distribution of test set sizes across different dataset splitting strategies. It uses seaborn and matplotlib for plotting, helping to identify strategies with more or less variability.

```python
sns.set_style("whitegrid")
plt.figure(figsize=(8, 5))
ax = sns.boxplot(x="split", y="num_test", data=result_df)
ax.set_xlabel("Dataset Splitting Strategy")
ax.set_ylabel("Test Set Size")
plt.show()
```

--------------------------------

### CLI Clustering (Parallel Multi-round)

Source: https://context7.com/mqcomplab/bblean/llms.txt

Executes parallel multi-round clustering for large datasets. This approach divides the clustering process across multiple initial and midsection processes, allowing for efficient processing of split fingerprint files. Configuration includes the number of workers for each stage.

```bash
bb multiround ./fingerprints-split \
  --threshold 0.65 \
  --branching 50 \
  --num-initial-processes 8 \
  --num-midsection-processes 4 \
  --num-midsection-rounds 2 \
  --out-dir ./multiround_results
```

--------------------------------

### Parallel Clustering of Large Molecular Libraries

Source: https://context7.com/mqcomplab/bblean/llms.txt

Executes multi-round hierarchical clustering for very large molecular libraries in parallel. It processes pre-computed fingerprints stored in multiple files and merges them iteratively. This approach significantly speeds up clustering for massive datasets. Results are saved to a specified output directory.

```python
import bblean.multiround as multiround
from pathlib import Path
import numpy as np
import pickle

# Assume fingerprints are pre-computed and saved as multiple .npy files
# For example: fps_part_0.npy, fps_part_1.npy, ..., fps_part_9.npy
input_files = [Path(f"fps_part_{i}.npy") for i in range(10)]
output_dir = Path("multiround_output")
output_dir.mkdir(exist_ok=True)

# Run multi-round BitBirch with parallel processing
timer = multiround.run_multiround_bitbirch(
    input_files=input_files,
    out_dir=output_dir,
    n_features=2048,
    input_is_packed=True,
    num_initial_processes=8,
    num_midsection_processes=4,
    branching_factor=50,
    threshold=0.65,
    tolerance=0.05,
    initial_merge_criterion="diameter",
    midsection_merge_criterion="tolerance-diameter",
    refinement_before_midsection="full",
    num_midsection_rounds=2,
    save_centroids=True,
    verbose=True,
    cleanup=True
)

# Load results
with open(output_dir / "clusters.pkl", "rb") as f:
    clusters = pickle.load(f)

print(f"Parallel clustering completed in {timer.timings['total']:.2f} seconds")
print(f"Created {len(clusters)} clusters from multiple input files")
```

--------------------------------

### Visualize Test Set Size vs. Number of Clusters

Source: https://github.com/mqcomplab/bblean/blob/main/examples/dataset_splitting.ipynb

This code generates a box plot to visualize the relationship between the number of clusters and the test set size. It helps to determine the point at which the test set size stabilizes as the number of clusters increases.

```python
sns.set_style("whitegrid")
ax = sns.boxplot(x="num_clus", y="num_test", data=urc_result_df)
ax.set_xlabel("Number of Clusters")
ax.set_ylabel("Number of Test Set Molecules")
```

--------------------------------

### Pack and Unpack Fingerprints - Python

Source: https://context7.com/mqcomplab/bblean/llms.txt

Provides functionality to convert between packed and unpacked binary fingerprint representations. Packing significantly reduces memory usage by compressing 8 bits into 1 byte. This is crucial for handling very large datasets.

```python
import bblean
import numpy as np

# Create unpacked binary fingerprints (0s and 1s only)
fps_unpacked = np.random.randint(0, 2, size=(10, 2048), dtype=np.uint8)

# Pack fingerprints for memory efficiency (8x compression)
fps_packed = bblean.pack_fingerprints(fps_unpacked)
print(f"Unpacked: {fps_unpacked.nbytes} bytes -> Packed: {fps_packed.nbytes} bytes")

# Unpack back to binary representation
fps_restored = bblean.unpack_fingerprints(fps_packed, n_features=2048)
assert np.array_equal(fps_unpacked, fps_restored)
```