pyclustertend

https://pyclustertend.readthedocs.io/en/latest/
Admin
pyclustertend is a Python toolkit for assessing cluster tendency in datasets using methods like the...
Tokens:3,652
Snippets:20
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### vat(data, return_odm=False, figure_size=(10, 10))

Source: https://pyclustertend.readthedocs.io/en/latest/pyclustertend.html

Performs VAT visualization.

```APIDOC
## vat(data, return_odm=False, figure_size=(10, 10))

### Description
VAT (Visual Assessment of Tendency) allows assessing cluster tendency through a map based on the dissimilarity matrix.

### Parameters
- **data** (matrix) - Required - numpy array
- **return_odm** (boolean) - Optional - Whether to return the Ordered Dissimilarity Matrix (default: False)
- **figure_size** (tuple) - Optional - Size of the VAT (default: (10, 10))

### Returns
- **ODM** (matrix) - The ordered dissimilarity matrix plotted.
```

--------------------------------

### assess_tendency_by_mean_metric_score(dataset, n_cluster=10, random_state=None)

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/metric.html

Assess the clusterability of a dataset using KMeans and the mean of silhouette, calinski-harabasz, and davies-bouldin scores.

```APIDOC
## assess_tendency_by_mean_metric_score

### Description
Assess the clusterability of a dataset using KMeans algorithm and the silhouette, calinski, and davies bouldin scores. The best cluster number is the mean of the result of the three methods.

### Parameters
- **dataset** (numpy array, DataFrame) - The input dataset.
- **n_cluster** (int) - The maximum number of clusters to consider.
- **random_state** (int) - Optional, default to None.

### Returns
- **n_clusters** (float) - The mean of the best number of cluster scores calculated across the three methods.
```

--------------------------------

### assess_tendency_by_metric(dataset, metric='silhouette', n_cluster=10, random_state=None)

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/metric.html

Assess the clusterability of a dataset using the KMeans algorithm and a specified metric score. Returns the optimal number of clusters and the scores for each cluster count.

```APIDOC
## assess_tendency_by_metric

### Description
Assess the clusterability of a dataset using KMeans algorithm and a metric score. The best cluster number is determined by the highest (or lowest for davies_bouldin) score.

### Parameters
- **dataset** (numpy array, DataFrame) - The input dataset.
- **metric** (string) - The method to assess cluster quality ('silhouette', 'calinski_harabasz', 'davies_bouldin'), default to 'silhouette'.
- **n_cluster** (int) - The maximum number of clusters to consider.
- **random_state** (int) - Optional, default to None.

### Returns
- **(n_clusters, value)** (tuple) - n_clusters is the number of clusters that best scored on the chosen metric. value is the array of scores for each number of clusters.
```

--------------------------------

### ivat(data, return_odm=False, figure_size=(10, 10))

Source: https://pyclustertend.readthedocs.io/en/latest/pyclustertend.html

Performs iVAT visualization.

```APIDOC
## ivat(data, return_odm=False, figure_size=(10, 10))

### Description
iVAT returns a visualization based on the VAT, which is more reliable and easier to interpret.

### Parameters
- **data** (matrix) - Required - numpy array
- **return_odm** (boolean) - Optional - Whether to return the Ordered Dissimilarity Matrix (default: False)
- **figure_size** (tuple) - Optional - Size of the VAT (default: (10, 10))

### Returns
- **D_prim** (matrix) - The iVAT ordered dissimilarity matrix.
```

--------------------------------

### Visualize Cluster Tendency with VAT

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/visual_assessment_of_tendency.html

Generates a VAT plot based on the dissimilarity matrix of the input data.

```python
def vat(data, return_odm=False, figure_size=(10, 10)):
    """VAT means Visual assesement of tendency. basically, it allow to asses cluster tendency
    through a map based on the dissimiliraty matrix.


    Parameters
    ----------

    data : matrix
        numpy array

    return_odm : return the Ordered Dissimalirity Matrix
        boolean (default to False)

    figure_size : size of the VAT.
        tuple (default to (10,10))


    Return
    -------

    ODM : matrix
        the ordered dissimalarity matrix plotted.

    """

    ordered_dissimilarity_matrix = compute_ordered_dissimilarity_matrix(data)

    _, ax = plt.subplots(figsize=figure_size)
    ax.imshow(ordered_dissimilarity_matrix, cmap='gray', vmin=0, vmax=np.max(ordered_dissimilarity_matrix))

    if return_odm is True:
        return ordered_dissimilarity_matrix
```

--------------------------------

### vat(data, return_odm=False, figure_size=(10, 10))

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/visual_assessment_of_tendency.html

Performs Visual Assessment of Tendency (VAT) to assess cluster tendency through a map based on the dissimilarity matrix.

```APIDOC
## vat(data, return_odm=False, figure_size=(10, 10))

### Description
Allows assessment of cluster tendency through a map based on the dissimilarity matrix.

### Parameters
- **data** (matrix) - Required - numpy array
- **return_odm** (boolean) - Optional - Whether to return the Ordered Dissimilarity Matrix (default: False)
- **figure_size** (tuple) - Optional - Size of the VAT plot (default: (10, 10))

### Returns
- **ODM** (matrix) - The ordered dissimilarity matrix plotted.
```

--------------------------------

### compute_ordered_dissimilarity_matrix(X)

Source: https://pyclustertend.readthedocs.io/en/latest/pyclustertend.html

Computes the ordered dissimilarity matrix used by VAT.

```APIDOC
## compute_ordered_dissimilarity_matrix(X)

### Description
Computes the ordered dissimilarity matrix used by visual assessment of tendency.

### Parameters
- **X** (matrix) - Required - numpy array

### Returns
- **ODM** (matrix) - The ordered dissimilarity matrix.
```

--------------------------------

### Compute Ordered Dissimilarity Matrix

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/visual_assessment_of_tendency.html

Calculates the reordered dissimilarity matrix required for VAT analysis.

```python
def compute_ordered_dissimilarity_matrix(X):
    """The ordered dissimilarity matrix is used by visual assesement of tendency. It is a just a a reordering
    of the dissimilarity matrix.


    Parameters
    ----------

    X : matrix
        numpy array

    Return
    -------

    ODM : matrix
        the ordered dissimalarity matrix .

    """

    # Step 1 :

    observation_path = []

    matrix_of_pairwise_distance = pairwise_distances(X)
    list_of_int = np.zeros(matrix_of_pairwise_distance.shape[0], dtype="int")

    index_of_maximum_value = np.argmax(matrix_of_pairwise_distance)

    column_index_of_maximum_value = index_of_maximum_value // matrix_of_pairwise_distance.shape[1]

    list_of_int[0] = column_index_of_maximum_value
    observation_path.append(column_index_of_maximum_value)

    K = np.linspace(0, matrix_of_pairwise_distance.shape[0] - 1, matrix_of_pairwise_distance.shape[0], dtype="int")
    J = np.delete(K, column_index_of_maximum_value)

    # Step 2 :

    for r in range(1, matrix_of_pairwise_distance.shape[0]):

        p, q = (-1, -1)

        mini = np.max(matrix_of_pairwise_distance)

        for candidate_p in observation_path:
            for candidate_j in J:
                if matrix_of_pairwise_distance[candidate_p, candidate_j] < mini:
                    p = candidate_p
                    q = candidate_j
                    mini = matrix_of_pairwise_distance[p, q]

        list_of_int[r] = q
        observation_path.append(q)

        ind_q = np.where(np.array(J) == q)[0][0]
        J = np.delete(J, ind_q)

    # Step 3

    ordered_matrix = np.zeros(matrix_of_pairwise_distance.shape)

    for column_index_of_maximum_value in range(ordered_matrix.shape[0]):
        for j in range(ordered_matrix.shape[1]):
            ordered_matrix[column_index_of_maximum_value, j] = matrix_of_pairwise_distance[
                list_of_int[column_index_of_maximum_value], list_of_int[j]]

    # Step 4 :

    return ordered_matrix
```

--------------------------------

### Assess cluster tendency using a specific metric

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/metric.html

Evaluates dataset clusterability by finding the optimal number of clusters based on a chosen metric score.

```python
>>> from sklearn import datasets
>>> from pyclustertend import assess_tendency_by_metric
>>> from sklearn.preprocessing import scale
>>> X = scale(datasets.load_boston().data)
>>> assess_tendency_by_metric(X, n_cluster=10)
(2, array([0.36011769, 0.25740335, 0.28098046, 0.28781574, 0.26746932,
    0.26975514, 0.27155699, 0.28883395, 0.29028124]))
```

--------------------------------

### compute_ivat_ordered_dissimilarity_matrix(X)

Source: https://pyclustertend.readthedocs.io/en/latest/pyclustertend.html

Computes the ordered dissimilarity matrix used by iVAT.

```APIDOC
## compute_ivat_ordered_dissimilarity_matrix(X)

### Description
Computes the ordered dissimilarity matrix used by iVAT, which is a reordering of the dissimilarity matrix.

### Parameters
- **X** (matrix) - Required - numpy array

### Returns
- **D_prim** (matrix) - The ordered dissimilarity matrix.
```

--------------------------------

### Visualize Cluster Tendency with iVAT

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/visual_assessment_of_tendency.html

Generates an iVAT plot, which is generally more reliable and easier to interpret than standard VAT.

```python
def ivat(data, return_odm=False, figure_size=(10, 10)):
    """iVat return a visualisation based on the Vat but more reliable and easier to
    interpret.


    Parameters
    ----------

    data : matrix
        numpy array

    return_odm : return the Ordered Dissimalirity Matrix
            boolean (default to False)

    figure_size : size of the VAT.
        tuple (default to (10,10))


    Return
    -------

    D_prim : matrix
        the ivat ordered dissimalarity matrix.

    """

    ordered_matrix = compute_ivat_ordered_dissimilarity_matrix(data)

    _, ax = plt.subplots(figsize=figure_size)
    ax.imshow(ordered_matrix, cmap='gray', vmin=0, vmax=np.max(ordered_matrix))

    if return_odm is True:
        return ordered_matrix
```

--------------------------------

### ivat(data, return_odm=False, figure_size=(10, 10))

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/visual_assessment_of_tendency.html

Performs Improved Visual Assessment of Tendency (iVAT) which provides a more reliable and easier to interpret visualization than standard VAT.

```APIDOC
## ivat(data, return_odm=False, figure_size=(10, 10))

### Description
Returns a visualization based on the VAT but more reliable and easier to interpret.

### Parameters
- **data** (matrix) - Required - numpy array
- **return_odm** (boolean) - Optional - Whether to return the Ordered Dissimilarity Matrix (default: False)
- **figure_size** (tuple) - Optional - Size of the VAT plot (default: (10, 10))

### Returns
- **D_prim** (matrix) - The iVAT ordered dissimilarity matrix.
```

--------------------------------

### Hopkins Function Implementation

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/hopkins.html

The source code implementation of the hopkins function.

```python
from sklearn.neighbors import BallTree
import numpy as np
import pandas as pd



[docs]def hopkins(data_frame, sampling_size):
    """Assess the clusterability of a dataset. A score between 0 and 1, a score around 0.5 express
    no clusterability and a score tending to 0 express a high cluster tendency.

    Parameters
    ----------
    data_frame : numpy array
        The input dataset
    sampling_size : int
        The sampling size which is used to evaluate the number of DataFrame.

    Returns
    ---------------------
    score : float
        The hopkins score of the dataset (between 0 and 1)

    Examples
    --------
    >>> from sklearn import datasets
    >>> from pyclustertend import hopkins
    >>> X = datasets.load_iris().data
    >>> hopkins(X,150)
    0.16
    """

    if type(data_frame) == np.ndarray:
        data_frame = pd.DataFrame(data_frame)

    # Sample n observations from D : P

    if sampling_size > data_frame.shape[0]:
        raise Exception(
            'The number of sample of sample is bigger than the shape of D')

    data_frame_sample = data_frame.sample(n=sampling_size)

    # Get the distance to their neirest neighbors in D : X

    tree = BallTree(data_frame, leaf_size=2)
    dist, _ = tree.query(data_frame_sample, k=2)
    data_frame_sample_distances_to_nearest_neighbours = dist[:, 1]

    # Randomly simulate n points with the same variation as in D : Q.

    max_data_frame = data_frame.max()
    min_data_frame = data_frame.min()

    uniformly_selected_values_0 = np.random.uniform(min_data_frame[0], max_data_frame[0], sampling_size)
    uniformly_selected_values_1 = np.random.uniform(min_data_frame[1], max_data_frame[1], sampling_size)

    uniformly_selected_observations = np.column_stack((uniformly_selected_values_0, uniformly_selected_values_1))
    if len(max_data_frame) >= 2:
        for i in range(2, len(max_data_frame)):
            uniformly_selected_values_i = np.random.uniform(min_data_frame[i], max_data_frame[i], sampling_size)
            to_stack = (uniformly_selected_observations, uniformly_selected_values_i)
            uniformly_selected_observations = np.column_stack(to_stack)

    uniformly_selected_observations_df = pd.DataFrame(uniformly_selected_observations)

    # Get the distance to their neirest neighbors in D : Y

    tree = BallTree(data_frame, leaf_size=2)
    dist, _ = tree.query(uniformly_selected_observations_df, k=1)
    uniformly_df_distances_to_nearest_neighbours = dist

    # return the hopkins score

    x = sum(data_frame_sample_distances_to_nearest_neighbours)
    y = sum(uniformly_df_distances_to_nearest_neighbours)

    if x + y == 0:
        raise Exception('The denominator of the hopkins statistics is null')

    return x / (x + y)[0]
```

--------------------------------

### Assess cluster tendency using mean metric scores

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/metric.html

Calculates the mean of the optimal cluster counts determined by silhouette, Calinski-Harabasz, and Davies-Bouldin scores.

```python
>>> from sklearn import datasets
>>> from pyclustertend import assess_tendency_by_mean_metric_score
>>> from sklearn.preprocessing import scale
>>> X = scale(datasets.load_boston().data)
>>> assess_tendency_by_mean_metric_score(X,10)
2.6666666666666665
```

--------------------------------

### Calculate Hopkins Statistic

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/hopkins.html

Assess the clusterability of a dataset using the hopkins function.

```python
from sklearn import datasets
from pyclustertend import hopkins
X = datasets.load_iris().data
hopkins(X,150)
```

--------------------------------

### hopkins(data_frame, sampling_size)

Source: https://pyclustertend.readthedocs.io/en/latest/_modules/pyclustertend/hopkins.html

Calculates the Hopkins score for a given dataset to determine its clusterability.

```APIDOC
## hopkins(data_frame, sampling_size)

### Description
Assess the clusterability of a dataset. A score between 0 and 1 is returned, where a score around 0.5 expresses no clusterability and a score tending to 0 expresses a high cluster tendency.

### Parameters
- **data_frame** (numpy array) - The input dataset.
- **sampling_size** (int) - The sampling size used to evaluate the number of observations.

### Returns
- **score** (float) - The hopkins score of the dataset (between 0 and 1).

### Example
```python
from sklearn import datasets
from pyclustertend import hopkins
X = datasets.load_iris().data
hopkins(X, 150)
# Returns: 0.16
```
```
Tokens:
Raw