### Install HDBSCAN from Source

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Download the package, install its requirements, and manually run the installer. This method is useful if direct pip installation is not feasible.

```bash
wget https://github.com/scikit-learn-contrib/hdbscan/archive/master.zip
unzip master.zip
rm master.zip
cd hdbscan-master

pip install -r requirements.txt

python setup.py install
```

--------------------------------

### Install HDBSCAN from GitHub

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Install the HDBSCAN package directly from its GitHub repository using pip. This is the recommended installation method.

```bash
pip install --upgrade git+https://github.com/scikit-learn-contrib/hdbscan.git#egg=hdbscan
```

--------------------------------

### Run HDBSCAN Tests

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Execute the package tests after installation using the 'nosetests' command. Ensure 'nose' is installed.

```bash
nosetests -s hdbscan
```

```bash
python -m nose -s hdbscan
```

--------------------------------

### Import necessary libraries for benchmarking

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/performance_and_scalability.md

Imports essential libraries for clustering, data generation, timing, and plotting. Ensure these are installed before running benchmarks.

```python
import hdbscan
import debacl
import fastcluster
import sklearn.cluster
import scipy.cluster
import sklearn.datasets
import numpy as np
import pandas as pd
import time
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set_context('poster')
sns.set_palette('Paired', 10)
sns.set_color_codes()
```

--------------------------------

### Import Libraries and Setup Plotting

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/soft_clustering_explanation.md

Imports necessary libraries for data manipulation, clustering, and visualization. Sets up plotting aesthetics for a poster-like context.

```python
import hdbscan
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import matplotlib as mpl

from scipy.spatial.distance import cdist

%matplotlib inline
sns.set_context('poster')
sns.set_style('white')
sns.set_color_codes()

plot_kwds={'alpha':0.25, 's':60, 'linewidths':0}
palette = sns.color_palette('deep', 12)
```

--------------------------------

### Import Libraries and Setup Plotting

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/how_hdbscan_works.md

Imports necessary libraries for numerical operations, plotting, and data generation. Sets up plotting aesthetics for better visualization.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.datasets as data
%matplotlib inline
sns.set_context('poster')
sns.set_style('white')
sns.set_color_codes()
plot_kwds = {'alpha' : 0.5, 's' : 80, 'linewidths':0}
```

--------------------------------

### Install HDBSCAN using Pip

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Installs the hdbscan package using pip. It is recommended to upgrade pip first.

```bash
pip install hdbscan
```

--------------------------------

### Branch Detection Example

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Illustrates how to use the BranchDetector class to identify branching structures in clusters after fitting HDBSCAN. Requires importing hdbscan and make_blobs.

```python
import hdbscan
from sklearn.datasets import make_blobs

data, _ = make_blobs(1000)

clusterer = hdbscan.HDBSCAN(branch_detection_data=True).fit(data)
branch_detector = hdbscan.BranchDetector().fit(clusterer)
branch_detector.cluster_approximation_graph_.plot(edge_width=0.1)
```

--------------------------------

### Import Libraries and Setup Plotting

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Looking at cluster consistency.ipynb

Imports necessary libraries for data manipulation, clustering, and visualization. Sets up plotting aesthetics using seaborn and matplotlib.

```python
import pandas as pd
import numpy as np
import hdbscan
from scipy.spatial.distance import cdist
#Some plotting libraries
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib notebook

sns.set_context('poster')
sns.set_color_codes()
plot_kwds = {'alpha' : 0.25, 's' : 40, 'linewidths':0}
```

--------------------------------

### Install HDBSCAN using Conda

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Installs the hdbscan package using the conda package manager from the conda-forge channel.

```bash
conda install -c conda-forge hdbscan
```

--------------------------------

### Create Conda Environment

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/oryx-build-commands.txt

Use this command to create a new Conda environment from a specified environment file. The --prefix flag specifies the installation directory, and --quiet suppresses verbose output during creation.

```bash
conda env create --file environment.yml --prefix ./venv --quiet
```

--------------------------------

### Generate Dataset Sizes for Benchmarking

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/performance_and_scalability.md

Creates an array of dataset sizes for performance testing, starting with small increments and increasing spacing for larger datasets.

```python
dataset_sizes = np.hstack([np.arange(1, 6) * 500, np.arange(3,7) * 1000, np.arange(4,17) * 2000])
```

--------------------------------

### Initialize HDBSCAN with Soft Clustering Enabled

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/soft_clustering.md

Initializes the HDBSCAN algorithm with a specified minimum cluster size and enables the generation of prediction data necessary for soft clustering. This setup is crucial for accessing membership probabilities.

```python
import hdbscan
```

```python
clusterer = hdbscan.HDBSCAN(min_cluster_size=10, prediction_data=True).fit(data)
color_palette = sns.color_palette('Paired', 12)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.5, 0.5, 0.5)
                  for x in clusterer.labels_]
cluster_member_colors = [sns.desaturate(x, p) for x, p in
                         zip(cluster_colors, clusterer.probabilities_)]
plt.scatter(*projection.T, s=50, linewidth=0, c=cluster_member_colors, alpha=0.25)
```

--------------------------------

### Install HDBSCAN Dependencies Manually

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Installs necessary dependencies (cython, numpy, scipy, scikit-learn) using conda before installing hdbscan via pip.

```bash
conda install cython
conda install numpy scipy
conda install scikit-learn
pip install hdbscan
```

--------------------------------

### Robust Single Linkage Clustering Example

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Demonstrates the usage of the RobustSingleLinkage class for clustering data and extracting cluster hierarchy. Requires importing hdbscan and make_blobs.

```python
import hdbscan
from sklearn.datasets import make_blobs

data, _ = make_blobs(1000)

clusterer = hdbscan.RobustSingleLinkage(cut=0.125, k=7)
cluster_labels = clusterer.fit_predict(data)
hierarchy = clusterer.cluster_hierarchy_
alt_labels = hierarchy.get_clusters(0.100, 5)
hierarchy.plot()
```

--------------------------------

### HDBSCAN with Precomputed Distance Matrices

Source: https://context7.com/scikit-learn-contrib/hdbscan/llms.txt

Demonstrates using HDBSCAN with both dense and sparse precomputed distance matrices, including an example for text clustering using cosine distance. Requires numpy, hdbscan, sklearn, and scipy.

```python
import numpy as np
import hdbscan
from sklearn.metrics import pairwise_distances
from sklearn.datasets import make_blobs
from scipy.sparse import csr_matrix

X, _ = make_blobs(n_samples=200, centers=3, random_state=42)

# Dense precomputed matrix
D = pairwise_distances(X, metric='cosine')
labels_dense = hdbscan.HDBSCAN(
    min_cluster_size=10,
    metric='precomputed'
).fit_predict(D)
print("Dense precomputed labels:", np.unique(labels_dense))
```

```python
# Sparse precomputed matrix: keep only k-NN distances
from sklearn.neighbors import kneighbors_graph
knn_graph = kneighbors_graph(X, n_neighbors=15, mode='distance')
sparse_D = (knn_graph + knn_graph.T)  # symmetrize

labels_sparse = hdbscan.HDBSCAN(
    min_cluster_size=10,
    metric='precomputed',
    algorithm='generic',
).fit_predict(sparse_D)
print("Sparse precomputed labels:", np.unique(labels_sparse))
```

```python
# String / text clustering via custom distance
from sklearn.feature_extraction.text import TfidfVectorizer
docs = ["cat dog", "cat cat", "python code", "code debug", "dog cat", "python run"]
tfidf = TfidfVectorizer()
D_text = pairwise_distances(tfidf.fit_transform(docs).toarray(), metric='cosine')
print("Text cluster labels:", hdbscan.HDBSCAN(
    min_cluster_size=2, metric='precomputed'
).fit_predict(D_text))
```

--------------------------------

### Import necessary libraries for HDBSCAN

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/advanced_hdbscan.md

Import the required libraries for data loading, clustering, visualization, and color mapping. Ensure these are installed before running.

```python
import hdbscan
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
```

--------------------------------

### K-Means clustering with known number of clusters

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/comparing_clustering_algorithms.md

Applies the K-Means algorithm to the dataset, specifying the number of clusters. This example assumes prior knowledge of the optimal number of clusters for the dataset.

```python
plot_clusters(data, cluster.KMeans, (), {'n_clusters':6})
```

--------------------------------

### Generate Datasize Table Based on Time Constraints

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Benchmarking scalability of clustering implementations 2D v0.7.ipynb

Defines a helper function `get_size` to find the maximum data size that can be processed within a given time limit. This function is then applied to the `timing_data` DataFrame for different time constraints (Interactive, Get Coffee, Over Lunch, Overnight) to create a table summarizing achievable dataset sizes for each clustering implementation.

```python
def get_size(series, max_time):
    return series.index[series < max_time].max()

datasize_table = pd.concat([
                            timing_data.apply(get_size, max_time=30),
                            timing_data.apply(get_size, max_time=300),
                            timing_data.apply(get_size, max_time=3600),
                            timing_data.apply(get_size, max_time=8*3600)
                            ], axis=1)
datasize_table.columns=('Interactive', 'Get Coffee', 'Over Lunch', 'Overnight')
datasize_table
```

--------------------------------

### Upgrade Pip

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Upgrades the pip package installer to the latest version.

```bash
pip install --upgrade pip
```

--------------------------------

### Instantiate RankedPoints

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Looking at cluster consistency.ipynb

Instantiate the `RankedPoints` class with data, a clusterer, and specify the metric and selection method.

```python
examples = RankedPoints(data, clusterer, metric='euclidean', selection_method='medoid')
```

--------------------------------

### Get Number of Nodes in NetworkX Graph

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/advanced_hdbscan.md

Determine the total number of nodes in the NetworkX representation of the condensed tree.

```python
g = clusterer.condensed_tree_.to_networkx()
g.number_of_nodes()
```

--------------------------------

### Display First Few Data Samples

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/basic_hdbscan.md

Shows the first few rows of the generated data in a pandas DataFrame for inspection.

```python
pd.DataFrame(blobs).head()
```

--------------------------------

### Get Closest Samples for a Cluster

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Looking at cluster consistency.ipynb

Retrieves a specified number of points that are closest to the cluster's representative point (centroid or medoid).

```python
def get_closest_samples_for_cluster(self, cluster_id, n_samples=5):
    """Get the N closest points to the cluster centroid/medoid"""

```

--------------------------------

### Generate Sample Data

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/basic_hdbscan.md

Generates a dataset with 2000 samples and 10 features using make_blobs.

```python
blobs, labels = make_blobs(n_samples=2000, n_features=10)
```

--------------------------------

### Create and visualize a synthetic dataset

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Flat clustering.ipynb

Generates a dataset with multiple circular clusters and moon shapes, then scales and splits it into training and testing sets. Visualizes both sets to show the data distribution.

```python
centers = [(0, 2), (-0.2, 0), (0.2, 0),
           (1.5, 0), (2., 1.), (2.5, 0.)]
std = [0.5, 0.08, 0.06, 0.25, 0.25, 0.25]
X, y = make_blobs(n_samples=[700, 300, 800, 1000, 400, 1500],
                  centers=centers,
                  cluster_std=std)
X1, y1 = make_moons(n_samples=5000, noise=0.07)
X1 += 3.
y1 += len(centers)
X = np.vstack((X, X1))
y = np.concatenate((y, y1))
scaler = StandardScaler()
X = scaler.fit_transform(X)
X, X_test, y, y_test = train_test_split(X, y, test_size=0.2,
                                        random_state=42)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].set_title("Training set")
axes[0].scatter(X[:, 0], X[:, 1], c=y, s=5)
axes[1].set_title("Test set")
axes[1].scatter(X_test[:, 0], X_test[:, 1], c=y_test, s=5)
plt.suptitle("Dataset used for illustration")
plt.show()
```

--------------------------------

### Rank points by distance to cluster center

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Looking at cluster consistency.ipynb

Get a pandas DataFrame of points ranked by their distance to the cluster center. This requires `calculate_all_distances_to_center` to have been called previously.

```python
cluster_1_ranked = examples.rank_cluster_points_by_distance(1)
```

```python
cluster_1_ranked
```

--------------------------------

### Basic HDBSCAN Clustering with Scikit-learn API

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/README.rst

Demonstrates the basic usage of HDBSCAN by initializing the clusterer and fitting it to sample data. This snippet assumes data is already generated and ready for clustering.

```python
import hdbscan
from sklearn.datasets import make_blobs

data, _ = make_blobs(1000)

clusterer = hdbscan.HDBSCAN(min_cluster_size=10)
cluster_labels = clusterer.fit_predict(data)
```

--------------------------------

### Import Libraries for Data Generation

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/basic_hdbscan.md

Imports necessary libraries for creating sample data for clustering.

```python
from sklearn.datasets import make_blobs
import pandas as pd
```

--------------------------------

### Initialize HDBSCAN Clusterer

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/basic_hdbscan.md

Creates an instance of the HDBSCAN clustering algorithm with default parameters.

```python
clusterer = hdbscan.HDBSCAN()
```

--------------------------------

### Spectral Clustering with Scikit-learn

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/comparing_clustering_algorithms.md

Demonstrates how to apply Spectral Clustering from scikit-learn to a dataset. Requires the 'plot_clusters' function and specifies the number of clusters.

```python
plot_clusters(data, cluster.SpectralClustering, (), {"n_clusters":6})
```

--------------------------------

### Predict Cluster Labels for New Points

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/prediction_tutorial.md

Use `hdbscan.approximate_predict()` to get cluster labels and strengths for new data points. Ensure the input is an array, even for a single point.

```python
test_labels, strengths = hdbscan.approximate_predict(clusterer, test_points)
test_labels
```

--------------------------------

### Load and Visualize Sample Data

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/soft_clustering_explanation.md

Loads sample data from a .npy file and visualizes it using a scatter plot. This step is for initial data exploration before clustering.

```python
data = np.load('clusterable_data.npy')
fig = plt.figure()
ax = fig.add_subplot(111)
plt.scatter(data.T[0], data.T[1], **plot_kwds)
ax.set_xticks([])
ax.set_yticks([]);
```

--------------------------------

### Get all clusters larger than min_cluster_size using 'leaf' method

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Flat clustering.ipynb

Use `HDBSCAN_flat` with `cluster_selection_method='leaf'` and `n_clusters=None` to extract all clusters larger than the specified `min_cluster_size`. This method is useful for finding more homogeneous clusters.

```python
clusterer = HDBSCAN_flat(X,
                         cluster_selection_method='leaf',
                         n_clusters=None, min_cluster_size=10)
labels = clusterer.labels_
proba = clusterer.probabilities_

plt.scatter(X[labels>=0, 0], X[labels>=0, 1], c=labels[labels>=0], s=5,
            cmap=plt.cm.jet)
plt.scatter(X[labels<0, 0], X[labels<0, 1], c='k', s=3, marker='x', alpha=0.2)
plt.show()
print(f"Unique labels (-1 for outliers): {np.unique(labels)}")
```

--------------------------------

### Prepare Data for Timing Analysis

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Python vs Java.ipynb

Prepares and combines timing data from different implementations for analysis. It resets indices, assigns implementation labels, renames columns, concatenates DataFrames, and calculates logarithmic scales for plotting.

```python
reference_series = pd.DataFrame(reference_timing_series.copy()).reset_index()
reference_series['implementation'] = 'Reference Implemetation'

hdbscan_series = pd.DataFrame(hdbscan_v03_timing_series.copy()).reset_index()
hdbscan_series['implementation'] = 'hdbscan library'
hdbscan_series.columns = ('dim', 'size', 'time', 'implementation')

combined_data = pd.concat([reference_series, hdbscan_series])
combined_data['log(time)'] = np.log10(combined_data.time)
combined_data['log(size)'] = np.log10(combined_data['size'])
```

--------------------------------

### Get N Furthest Samples for a Cluster

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Looking at cluster consistency.ipynb

Retrieves the N samples furthest from the representative point of a specified cluster. Returns a DataFrame with original data indices, cluster labels, and distance to the representative point.

```python
examples.get_furthest_samples_for_cluster(2, n_samples=5)
```

--------------------------------

### Get N Closest Samples for a Cluster

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Looking at cluster consistency.ipynb

Retrieves the N samples closest to the representative point of a specified cluster. Returns a DataFrame with original data indices, cluster labels, and distance to the representative point.

```python
examples.get_closest_samples_for_cluster(2, n_samples=5)
```

--------------------------------

### Load Datasets and Libraries

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/soft_clustering.md

Imports necessary libraries for data loading, dimensionality reduction, plotting, and numerical operations. This is a prerequisite for the subsequent clustering steps.

```python
from sklearn import datasets
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
```

--------------------------------

### Generate Sample Clustering Data

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/how_hdbscan_works.md

Creates a dataset with 100 data points, combining two distinct cluster shapes (moons and blobs) with some noise. Useful for illustrating clustering algorithms.

```python
moons, _ = data.make_moons(n_samples=50, noise=0.05)
blobs, _ = data.make_blobs(n_samples=50, centers=[(-0.75,2.25), (1.0, 2.0)], cluster_std=0.25)
test_data = np.vstack([moons, blobs])
plt.scatter(test_data.T[0], test_data.T[1], color='b', **plot_kwds)
```

--------------------------------

### HDBSCAN Clustering with min_cluster_size=60 (default min_samples)

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/parameter_selection.md

Performs HDBSCAN clustering with `min_cluster_size` set to 60. When `min_samples` is not explicitly set, it defaults to `min_cluster_size`, potentially leading to more noise points and fewer clusters. Visualizes the results.

```python
clusterer = hdbscan.HDBSCAN(min_cluster_size=60).fit(data)
color_palette = sns.color_palette('Paired', 12)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.5, 0.5, 0.5)
                  for x in clusterer.labels_]
cluster_member_colors = [sns.desaturate(x, p) for x, p in
                         zip(cluster_colors, clusterer.probabilities_)]
plt.scatter(*projection.T, s=50, linewidth=0, c=cluster_member_colors, alpha=0.25)
```

--------------------------------

### Generate and Analyze Clustering Timings

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/performance_and_scalability.md

Applies the timing model to various datasets and concatenates results. It then calculates the maximum dataset size that can be processed within specified time limits (Interactive, Get Coffee, Over Lunch, Overnight).

```python
ap_timings = get_timing_series(ap_data)
spectral_timings = get_timing_series(spectral_data)
agg_timings = get_timing_series(agg_data)
debacl_timings = get_timing_series(debacl_data)
fastclust_timings = get_timing_series(large_fastclust_data.ix[:10,:].copy())
scipy_single_timings = get_timing_series(large_scipy_single_data.ix[:10,:].copy())
hdbscan_boruvka = get_timing_series(huge_hdbscan_data, quadratic=True)
#scipy_k_means_timings = get_timing_series(huge_scipy_k_means_data, quadratic=False)
dbscan_timings = get_timing_series(huge_dbscan_data, quadratic=True)
k_means_timings = get_timing_series(huge_k_means_data, quadratic=False)

timing_data = pd.concat([ap_timings, spectral_timings, agg_timings, debacl_timings,
                            scipy_single_timings, fastclust_timings, hdbscan_boruvka,
                            dbscan_timings, k_means_timings
                           ], axis=1)
timing_data.columns=['AffinityPropagation', 'Spectral', 'Agglomerative',
                                       'DeBaCl', 'ScipySingleLinkage', 'Fastcluster',
                                       'HDBSCAN', 'DBSCAN', 'SKLearn KMeans'
                                      ]
def get_size(series, max_time):
    return series.index[series < max_time].max()

datasize_table = pd.concat([
                            timing_data.apply(get_size, max_time=30),
                            timing_data.apply(get_size, max_time=300),
                            timing_data.apply(get_size, max_time=3600),
                            timing_data.apply(get_size, max_time=8*3600)
                            ], axis=1)
datasize_table.columns=('Interactive', 'Get Coffee', 'Over Lunch', 'Overnight')
datasize_table
```

--------------------------------

### Train HDBSCAN flat instance with 5 clusters

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Flat clustering.ipynb

Initializes and trains an HDBSCAN_flat model with a specified number of clusters. Visualizes the resulting clustering on the training data, marking outliers with 'x'.

```python
clusterer = HDBSCAN_flat(X,
                         cluster_selection_method='eom',
                         n_clusters=5, min_cluster_size=10)
labels = clusterer.labels_
proba = clusterer.probabilities_

plt.title("Trained clustering:")
plt.scatter(X[labels>=0, 0], X[labels>=0, 1], c=labels[labels>=0], s=5,
            cmap=plt.cm.jet)
plt.scatter(X[labels<0, 0], X[labels<0, 1], c='k', s=3, marker='x', alpha=0.2)
plt.show()
print(f"Unique labels (-1 for outliers): {np.unique(labels)}")
```

--------------------------------

### Get membership vectors and derive labels for training set

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Flat clustering.ipynb

Extracts membership vectors for all points in the training set using a trained HDBSCAN_flat instance. Then, it uses the `clusters_from_membership` helper function to derive hard cluster labels and visualizes the result, including outliers.

```python
n_clusters = 4
clusterer = HDBSCAN_flat(X,
                         cluster_selection_method='eom',
                         n_clusters=n_clusters, min_cluster_size=10)
memberships = all_points_membership_vectors_flat(clusterer)
labels, proba = clusters_from_membership(memberships)

plt.scatter(X[labels>=0, 0], X[labels>=0, 1], c=labels[labels>=0], s=5,
            cmap=plt.cm.jet)
plt.scatter(X[labels<0, 0], X[labels<0, 1], c='k', s=3, marker='x', alpha=0.2)
plt.show()
print(f"Unique labels (-1 for outliers): {np.unique(labels)}")
```

--------------------------------

### Generate and aggregate reference timings

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Performance data generation .ipynb

Iterates through various dataset dimensions and sizes, generating data, and collecting external and internal timings from the reference implementation.

```python
internal_timing = {}
external_timing = {}

for dataset_dimension in (2,5,10,25,50):
    for dataset_size in np.arange(1,17) * 8000:
        data, _ = sklearn.datasets.make_blobs(dataset_size, 
                                              n_features=dataset_dimension, 
                                              centers=dataset_dimension)
        (external_timing[(dataset_dimension, dataset_size)], 
         internal_timing[(dataset_dimension, dataset_size)]) = get_reference_timings(data)
    
internal_timing_df = pd.DataFrame(internal_timing).T
external_timing_series = pd.Series(external_timing)
```

--------------------------------

### Initialize and Fit HDBSCAN

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/how_hdbscan_works.md

Initializes the HDBSCAN clustering algorithm with a minimum cluster size of 5 and enables the generation of the minimum spanning tree. Then, it fits the model to the provided test data.

```python
clusterer = hdbscan.HDBSCAN(min_cluster_size=5, gen_min_span_tree=True)
clusterer.fit(test_data)
```

--------------------------------

### Plot performance comparison of clustering algorithms

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/performance_and_scalability.rst

Plots runtime vs. data size for multiple clustering algorithms using seaborn regression plots. Expects precomputed results as pandas DataFrames with columns 'x' (data size) and 'y' (time in seconds). The example shows comparing ten algorithms across dataset sizes up to approximately 32,000 points.

```python
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

# Plot performance comparison across multiple clustering results
sns.regplot(x='x', y='y', data=k_means_data, order=2,
            label='Sklearn K-Means', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=dbscan_data, order=2,
            label='Sklearn DBSCAN', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=scipy_k_means_data, order=2,
            label='Scipy K-Means', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=hdbscan_data, order=2,
            label='HDBSCAN', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=fastclust_data, order=2,
            label='Fastcluster Single Linkage', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=scipy_single_data, order=2,
            label='Scipy Single Linkage', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=debacl_data, order=2,
            label='DeBaCl Geom Tree', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=spectral_data, order=2,
            label='Sklearn Spectral', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=agg_data, order=2,
            label='Sklearn Agglomerative', x_estimator=np.mean)
sns.regplot(x='x', y='y', data=ap_data, order=2,
            label='Sklearn Affinity Propagation', x_estimator=np.mean)

plt.gca().axis([0, 34000, 0, 120])
plt.gca().set_xlabel('Number of data points')
plt.gca().set_ylabel('Time taken to cluster (s)')
plt.title('Performance Comparison of Clustering Implementations')
plt.legend()
plt.show()
```

--------------------------------

### Display data content

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/advanced_hdbscan.md

View the first few rows of the loaded data to understand its structure and values.

```python
data
```

--------------------------------

### Flat Clustering for Training Data (n_clusters=5)

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Flat clustering.ipynb

Generates flat clusters for the training data with 5 clusters. Visualizes the clustered points and outliers, and prints the unique labels.

```python
n_clusters = 5
memberships = all_points_membership_vectors_flat(clusterer, n_clusters=n_clusters)
labels, proba = clusters_from_membership(memberships)

plt.scatter(X[labels>=0, 0], X[labels>=0, 1], c=labels[labels>=0], s=5,
            cmap=plt.cm.jet)
plt.scatter(X[labels<0, 0], X[labels<0, 1], c='k', s=3, marker='x', alpha=0.2)
plt.show()
print(f"Unique labels (-1 for outliers): {np.unique(labels)}")
```

--------------------------------

### Prepare Data for Plotting (hdbscan v0.6)

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Python vs Java.ipynb

Prepares timing data for plotting for the hdbscan v0.6 release. Similar to the v0.5 preparation, it creates pandas DataFrames, combines them, and calculates logarithmic scales for size and time for performance visualization.

```Python
reference_series = pd.DataFrame(reference_timing_series.copy()).reset_index()
reference_series['implementation'] = 'Reference Implemetation'

hdbscan_series = pd.DataFrame(hdbscan_v06_timing_series.copy()).reset_index()
hdbscan_series['implementation'] = 'hdbscan library'
hdbscan_series.columns = ('dim', 'size', 'time', 'implementation')

combined_data = pd.concat([reference_series, hdbscan_series])
combined_data['log(time)'] = np.log10(combined_data.time)
combined_data['log(size)'] = np.log10(combined_data['size'])
```

--------------------------------

### HDBSCAN Clustering with min_cluster_size=60 and min_samples=15

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/parameter_selection.md

Performs HDBSCAN clustering with `min_cluster_size` set to 60 and `min_samples` explicitly set to 15. This demonstrates how setting `min_samples` independently can recover more clusters than when it defaults to `min_cluster_size`. Visualizes the results.

```python
clusterer = hdbscan.HDBSCAN(min_cluster_size=60, min_samples=15).fit(data)
color_palette = sns.color_palette('Paired', 12)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.5, 0.5, 0.5)
                  for x in clusterer.labels_]
cluster_member_colors = [sns.desaturate(x, p) for x, p in
                         zip(cluster_colors, clusterer.probabilities_)]
plt.scatter(*projection.T, s=50, linewidth=0, c=cluster_member_colors, alpha=0.25)
```

--------------------------------

### Plotting Clusters with Mean Shift

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/comparing_clustering_algorithms.md

Demonstrates how to use Mean Shift for clustering and visualize the results. Requires data and a plotting function. The 'cluster_all' parameter controls whether all points are assigned to a cluster or if noise points are allowed.

```python
plot_clusters(data, cluster.MeanShift, (0.175,), {'cluster_all':False})
```

--------------------------------

### Initialize HDBSCAN_flat for Novel Point Prediction

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Flat clustering.ipynb

Initializes the HDBSCAN_flat model with the training data (X) and specifies 'eom' as the cluster selection method with no predefined number of clusters.

```python
n_clusters = None
clusterer = HDBSCAN_flat(X,
                         cluster_selection_method='eom',
                         n_clusters=n_clusters, min_cluster_size=10)
```

--------------------------------

### Import HDBSCAN Library

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/basic_hdbscan.md

Imports the HDBSCAN clustering library.

```python
import hdbscan
```

--------------------------------

### DBSCAN Clustering with Plotting

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/comparing_clustering_algorithms.md

Demonstrates how to apply DBSCAN clustering to a dataset and visualize the results. Requires the 'plot_clusters' function and the 'DBSCAN' class from scikit-learn. Parameter 'eps' controls the maximum distance between samples for one to be considered as in the neighborhood of the other.

```python
plot_clusters(data, cluster.DBSCAN, (), {"eps":0.025})
```

--------------------------------

### Fit Timing Models with Statsmodels

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/docs/performance_and_scalability.md

Fits linear models to timing data, supporting quadratic (O(n^2)) or n*log(n) complexity. Use this to predict algorithm runtimes for various dataset sizes.

```python
import statsmodels.formula.api as sm

time_samples = [1000, 2000, 5000, 10000, 25000, 50000, 75000, 100000, 250000, 500000, 750000,
               1000000, 2500000, 5000000, 10000000, 50000000, 100000000, 500000000, 1000000000]

def get_timing_series(data, quadratic=True):
    if quadratic:
        data['x_squared'] = data.x**2
        model = sm.ols('y ~ x + x_squared', data=data).fit()
        predictions = [model.params.dot([1.0, i, i**2]) for i in time_samples]
        return pd.Series(predictions, index=pd.Index(time_samples))
    else: # assume n log(n)
        data['xlogx'] = data.x * np.log(data.x)
        model = sm.ols('y ~ x + xlogx', data=data).fit()
        predictions = [model.params.dot([1.0, i, i*np.log(i)]) for i in time_samples]
        return pd.Series(predictions, index=pd.Index(time_samples))
```

--------------------------------

### Prepare Data for Plotting (hdbscan v0.5)

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Python vs Java.ipynb

Prepares timing data for plotting by creating pandas DataFrames, concatenating them, and calculating logarithmic scales for size and time. This is used for visualizing performance comparisons.

```Python
reference_series = pd.DataFrame(reference_timing_series.copy()).reset_index()
reference_series['implementation'] = 'Reference Implemetation'

hdbscan_series = pd.DataFrame(hdbscan_v05_timing_series.copy()).reset_index()
hdbscan_series['implementation'] = 'hdbscan library'
hdbscan_series.columns = ('dim', 'size', 'time', 'implementation')

combined_data = pd.concat([reference_series, hdbscan_series])
combined_data['log(time)'] = np.log10(combined_data.time)
combined_data['log(size)'] = np.log10(combined_data['size'])
```

--------------------------------

### Prepare Data for Timing Analysis in Python

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Python vs Java.ipynb

Prepares pandas DataFrames for timing analysis by copying, resetting indices, and adding implementation labels. It also calculates logarithmic transformations of 'size' and 'time' for plotting.

```python
reference_series = pd.DataFrame(reference_timing_series.copy()).reset_index()
reference_series['implementation'] = 'Reference Implemetation'

hdbscan_series = pd.DataFrame(hdbscan_v04_timing_series.copy()).reset_index()
hdbscan_series['implementation'] = 'hdbscan library'
hdbscan_series.columns = ('dim', 'size', 'time', 'implementation')

combined_data = pd.concat([reference_series, hdbscan_series])
combined_data['log(time)'] = np.log10(combined_data.time)
combined_data['log(size)'] = np.log10(combined_data['size'])
```

--------------------------------

### Import necessary libraries

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Performance data generation .ipynb

Imports required modules for data generation, manipulation, and subprocess execution.

```python
import sklearn.datasets
import numpy as np
import pandas as pd
import subprocess
import time
```

--------------------------------

### Time reference Java implementation

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Performance data generation .ipynb

Generates a CSV from input data, runs the Java reference implementation via subprocess, and parses internal timing information.

```python
def get_reference_timings(data, filename='tmp_data.csv',
                          jarfile='/Users/leland/Source/HDBSCAN_Star/HDBSCAN_Star.jar',
                          min_points=5, min_cluster_size=5):
    
    # Create the required csv file
    pd.DataFrame(data).to_csv('tmp_data.csv', header=False, index=False)
    
    # Run the clustering via a subprocess call and grab the output as it
    # has timing information to be parsed
    start_time = time.time()
    internal_timing = subprocess.check_output(['java', '-jar', jarfile,
                                               'file={}'.format(filename),
                                               'minPts={}'.format(min_points),
                                               'minClSize={}'.format(min_cluster_size),
                                               'compact=true'])
    time_taken = time.time() - start_time
    
    # Parse internal timing info into a pandas series for later use
    result_dict = {}
    for line in internal_timing.split('\n'):
        if ':' in line:
            key, value = line.split(':')
            key = key.replace(' (ms)', '')
            key = key.replace('Time to ', '')
            key = key.replace('Overall ', '')
            value = int(value)
            result_dict[key] = value
            
    internal_timing = pd.Series(result_dict)
    
    return time_taken, internal_timing
```

--------------------------------

### Prepare Data for Plotting

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Python vs Java.ipynb

Combines timing data from reference and hdbscan implementations, calculates logarithmic scales for time and dataset size, and prepares the data for plotting with Seaborn's lmplot.

```python
reference_series = pd.DataFrame(reference_timing_series.copy()).reset_index()
reference_series['implementation'] = 'Reference Implemetation'

hdbscan_series = pd.DataFrame(hdbscan_v01_timing_series.copy()).reset_index()
hdbscan_series['implementation'] = 'hdbscan library'
hdbscan_series.columns = ('dim', 'size', 'time', 'implementation')

combined_data = pd.concat([reference_series, hdbscan_series])
combined_data['log(time)'] = np.log10(combined_data.time)
combined_data['log(size)'] = np.log10(combined_data['size'])
```

--------------------------------

### Perform HDBSCAN Clustering

Source: https://github.com/scikit-learn-contrib/hdbscan/blob/master/notebooks/Looking at cluster consistency.ipynb

Initializes and fits an HDBSCAN model to the loaded data. The `min_cluster_size` parameter is set to 15.

```python
clusterer = hdbscan.HDBSCAN(min_cluster_size=15)
clusterer.fit(data)
labels = clusterer.labels_
```