PyTDigest

https://protivinsky.github.io/pytdigest/
Admin
PyTDigest is a fast Python package with a C implementation designed for streaming calculation of...
Tokens:6,491
Snippets:65
Trust Score:9.7
Update:5 days ago
Show doc for...
Tokens:
Raw
### Basic TDigest computation and usage

Source: https://protivinsky.github.io/pytdigest/index.html

Creates a TDigest from weighted data using TDigest.compute, then demonstrates mean, variance, inverse CDF (quantiles), copying, and addition of TDigests. Requires numpy and pandas. The example uses random data and shows that results are close to numpy's unweighted quantiles.

```python
from pytdigest import TDigest
import numpy as np
import pandas as pd

rng = np.random.default_rng(12354)
n = 100_000
x = rng.normal(loc=0, scale=10, size=n)
w = rng.exponential(scale=1, size=n)

# estimation from data is simple:
td = TDigest.compute(x, w, compression=1_000)
# td now contains "compressed" distribution: centroids with their means and weights

# TDigest can be used to provide mean, variance or approximate quantiles (i.e. inverse CDF):
td.mean
quantiles = [0., 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1.]
td.inverse_cdf(quantiles)
td.variance   # new in 0.1.4

# these results are close to numpy values (note that numpy provides only unweighted quantiles)
np.average(x, weights=w)  # mean should be exact
np.quantile(x, quantiles)

# TDigest can be copied
td2 = td.copy()

# and multiple TDigests can be added together to provide approximate quantiles for the overall dataset
td + td2

```

--------------------------------

### TDigest.get_centroids()

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Gets all centroids as a two-dimensional array. Based on the centroids, the TDigest can be fully reconstructed (for a given compression). Conversion to centroids and back can be used for serialization.

```APIDOC
## TDigest.get_centroids()

### Description
Gets all centroids as a two-dimensional array. Based on the centroids, the TDigest can be fully reconstructed (for a given compression). Conversion to centroids and back can be used for serialization.

### Method
get_centroids

### Parameters
None

### Returns
- A two-dimensional array of centroids.
```

--------------------------------

### TDigest.get_centroid(i)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Gets the centroid (i.e., a tuple of mean and weight) at a given index of the underlying representation.

```APIDOC
## TDigest.get_centroid(i)

### Description
Gets the centroid (i.e., a tuple of mean and weight) at a given index of the underlying representation.

### Method
get_centroid

### Parameters
- **i** (int) - Required - Index of the centroid to retrieve.

### Returns
- A tuple of mean and weight for the centroid at the given index.
```

--------------------------------

### TDigest.compute(x, w=None, handling_invalid=HandlingInvalid.Drop, compression=100)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Estimates a TDigest directly from data.

```APIDOC
## TDigest.compute(x, w=None, handling_invalid=HandlingInvalid.Drop, compression=100)

### Description
Estimates a TDigest directly from data.

### Method
compute (static)

### Parameters
- **x** (Number | ndarray | Series) - Required - Values to calculate the distribution of.
- **w** (Number | ndarray | Series | None) - Optional - Optional weight for the values. If it is np.ndarray or pd.Series, it has to have the same (one-dimensional) shape as x.
- **handling_invalid** (HandlingInvalid) - Optional - How to handle invalid values in calculation ['drop', 'raise'], default value 'drop'. Provided either as enum or its string representation.
- **compression** (int) - Optional - Higher compression value leads to more precise results and larger memory requirements. Compression gives a maximum number of centroids of the internal representation after merging of the TDigest. Unmerged representation can have up to roughly six times as many centroids.

### Returns
- New TDigest object estimated based on (possibly weighted) data.

### Raises
- **TypeError** - If x or w are not of permitted types or of different types.
- **ValueError** - If handling_invalid is 'raise' and there are invalid values in data (nan, infinity, negative weight).
```

--------------------------------

### TDigest.combine(first, second=None)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Combines multiple TDigests together.

```APIDOC
## TDigest.combine(first, second=None)

### Description
Combines multiple TDigests together.

### Method
combine (static)

### Parameters
- **first** (TDigest | Iterable[TDigest]) - Required - The first TDigest or an iterable of TDigests to combine.
- **second** (TDigest | None) - Optional - The second TDigest to combine, if first is a single TDigest.

### Returns
- A new TDigest object representing the combined distribution.
```

--------------------------------

### Compare PyTDigest with pure Python tdigest package

Source: https://protivinsky.github.io/pytdigest/index.html

Benchmarks PyTDigest against the pure Python tdigest package. The output shows PyTDigest at 0.0246 seconds and TDigest at 7.26 seconds for 100,000 elements. Note that tdigest only supports different weights via the update method for single observations.

```python
import numpy as np
from pytdigest import TDigest
from tdigest import TDigest as TDigestPython
import time

rng = np.random.Generator(np.random.PCG64(12345))
n = 100_000
x = rng.normal(size=n)
w = rng.exponential(size=n)

start = time.time()
td = TDigest.compute(x, w)
end = time.time()
print(f'PyTDigest: {end - start:.3g} seconds')
# PyTDigest: 0.0246 seconds

tdp = TDigestPython()
start = time.time()
tdp.batch_update(x)
end = time.time()
print(f'TDigest: {end - start:.3g} seconds')
# TDigest: 7.26 seconds
```

--------------------------------

### pytdigest.TDigest.combine()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Static method to combine multiple TDigest instances into one.

```APIDOC
## combine()

### Description
Static method to combine multiple TDigest instances into one.

### Method
combine()

### Parameters
- **digests** (list of TDigest) - Required - The TDigest instances to combine.

### Returns
- **TDigest** - The combined TDigest.
```

--------------------------------

### TDigest.weight (property)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Total weight of the data.

```APIDOC
## TDigest.weight (property)

### Description
Total weight of the data.

### Method
weight (property)

### Parameters
None

### Returns
- The total weight of the data.
```

--------------------------------

### Benchmark PyTDigest performance on numpy arrays

Source: https://protivinsky.github.io/pytdigest/index.html

Measures PyTDigest's compute time for varying array sizes. The output shows approximately 0.0222, 0.21, and 2.02 seconds for 100k, 1M, and 10M elements respectively.

```python
import numpy as np
from pytdigest import TDigest
import time

rng = np.random.Generator(np.random.PCG64(12345))

for n in [100_000, 1_000_000, 10_000_000]:
    x = rng.normal(size=n)
    w = rng.exponential(size=n)

    start = time.time()
    td = TDigest.compute(x, w)
    end = time.time()
    print(f'PyTDigest, n = {n:,}: {end - start:.3g} seconds')

# PyTDigest, n = 100,000: 0.0222 seconds
# PyTDigest, n = 1,000,000: 0.21 seconds
# PyTDigest, n = 10,000,000: 2.02 seconds
```

--------------------------------

### TDigest.update(x, w=None, handling_invalid=HandlingInvalid.Drop)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Adds new data to the TDigest.

```APIDOC
## TDigest.update(x, w=None, handling_invalid=HandlingInvalid.Drop)

### Description
Adds new data to the TDigest.

### Method
update

### Parameters
- **x** (Number | ndarray | Series) - Required - Values to calculate the distribution of.
- **w** (Number | ndarray | Series | None) - Optional - Optional weight for the values. If it is np.ndarray or pd.Series, it has to have the same (one-dimensional) shape as x.
- **handling_invalid** (HandlingInvalid) - Optional - How to handle invalid values in calculation ['drop', 'raise'], default value 'drop'. Provided either as enum or its string representation.

### Raises
- **TypeError** - If x or w are not of permitted types or of different types.
- **ValueError** - If handling_invalid is 'raise' and there are invalid values in data (nan, infinity, negative weight).
```

--------------------------------

### Benchmark t-digest CFFI package

Source: https://protivinsky.github.io/pytdigest/index.html

Benchmarks the t-digest CFFI package, which requires iterating over the array in Python because it does not pass batch updates to the C layer. The output shows 0.479 seconds for 100,000 elements, and the package only supports integer weights.

```python
import numpy as np
from tdigest import TDigest as TDigestCFFI
import time

rng = np.random.Generator(np.random.PCG64(12345))
n = 100_000
x = rng.normal(size=n)

tdcffi = TDigestCFFI()
start = time.time()
for xx in x:
    tdcffi.insert(xx)
end = time.time()
print(f'TDigest-CFFI: {end - start:.3g} seconds')
# TDigest-CFFI: 0.479 seconds
```

--------------------------------

### TDigest.inverse_cdf(quantile)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Calculates quantiles at given points (quantiles are just inverse of cumulative distribution function).

```APIDOC
## TDigest.inverse_cdf(quantile)

### Description
Calculates quantiles at given points (quantiles are just inverse of cumulative distribution function).

### Method
inverse_cdf

### Parameters
- **quantile** (Number | List | ndarray) - Required - The values where the inverse CDF should be calculated. Multiple values are passed to C library for fast estimation of larger number of quantiles.

### Returns
- Estimated approximate quantiles (float | ndarray).

### Raises
- **TypeError** - If quantile is of invalid type.
```

--------------------------------

### TDigest.of_centroids(centroids, compression=100)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Reconstructs a TDigest from centroids and a given compression.

```APIDOC
## TDigest.of_centroids(centroids, compression=100)

### Description
Reconstructs a TDigest from centroids and a given compression.

### Method
of_centroids (static)

### Parameters
- **centroids** (ndarray) - Required - The centroids to reconstruct the TDigest from.
- **compression** (float) - Optional - The compression parameter for the reconstructed TDigest.

### Returns
- A TDigest object reconstructed from the given centroids.
```

--------------------------------

### TDigest.mean (property)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Exact mean of the data.

```APIDOC
## TDigest.mean (property)

### Description
Exact mean of the data.

### Method
mean (property)

### Parameters
None

### Returns
- The exact mean of the data.
```

--------------------------------

### TDigest.std_dev (property)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Standard deviation, not corrected for doff.

```APIDOC
## TDigest.std_dev (property)

### Description
Standard deviation, not corrected for doff.

### Method
std_dev (property)

### Parameters
None

### Returns
- The standard deviation of the data.
```

--------------------------------

### TDigest.cdf(at)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Calculates approximate values of the empirical cumulative distribution function (CDF) at given points.

```APIDOC
## TDigest.cdf(at)

### Description
Calculates approximate values of the empirical cumulative distribution function (CDF) at given points.

### Method
cdf

### Parameters
- **at** (Number | List | ndarray) - Required - Values at which the CDF should be calculated.

### Returns
- Values of CDF calculated at given points (float | ndarray).

### Raises
- **TypeError** - If at is of invalid type.
```

--------------------------------

### pytdigest.TDigest.mean (property)

Source: https://protivinsky.github.io/pytdigest/genindex.html

Returns the mean of the TDigest.

```APIDOC
## mean (property)

### Description
Returns the mean of the TDigest.

### Returns
- **float** - The mean.
```

--------------------------------

### pytdigest.TDigest.compute()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Static method to compute a TDigest from a list of values.

```APIDOC
## compute()

### Description
Static method to compute a TDigest from a list of values.

### Method
compute()

### Parameters
- **values** (list of float) - Required - The values to compute the TDigest from.

### Returns
- **TDigest** - The computed TDigest.
```

--------------------------------

### pytdigest.TDigest.weight (property)

Source: https://protivinsky.github.io/pytdigest/genindex.html

Returns the total weight of the TDigest.

```APIDOC
## weight (property)

### Description
Returns the total weight of the TDigest.

### Returns
- **float** - The total weight.
```

--------------------------------

### pytdigest.TDigest.inverse_cdf()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Computes the inverse cumulative distribution function (quantile) for a given probability.

```APIDOC
## inverse_cdf()

### Description
Computes the inverse cumulative distribution function (quantile) for a given probability.

### Method
inverse_cdf()

### Parameters
- **probability** (float) - Required - The probability for which to compute the quantile.

### Returns
- **float** - The quantile value.
```

--------------------------------

### pytdigest.TDigest.scale_weight()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Scales the weight of a centroid.

```APIDOC
## scale_weight()

### Description
Scales the weight of a centroid.

### Method
scale_weight()

### Parameters
- **index** (int) - Required - The index of the centroid.
- **factor** (float) - Required - The scaling factor.

### Returns
- **None**
```

--------------------------------

### pytdigest.TDigest.std_dev (property)

Source: https://protivinsky.github.io/pytdigest/genindex.html

Returns the standard deviation of the TDigest.

```APIDOC
## std_dev (property)

### Description
Returns the standard deviation of the TDigest.

### Returns
- **float** - The standard deviation.
```

--------------------------------

### TDigest.var (property)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Alias for variance.

```APIDOC
## TDigest.var (property)

### Description
Alias for variance.

### Method
var (property)

### Parameters
None

### Returns
- The variance of the data.
```

--------------------------------

### TDigest

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

The TDigest class represents a t-digest data structure for accurate quantile estimation. It provides methods to update with data, compute quantiles, combine digests, and access centroids and statistics.

```APIDOC
## TDigest

### Description
The TDigest class is the main entry point for creating and manipulating t-digest data structures. It supports updating with data points, computing cumulative distribution and inverse CDF, combining multiple digests, and accessing centroids and statistical properties.

### Constructor
- **TDigest()** - Creates a new empty TDigest instance.

### Methods
- **cdf()** - Computes the cumulative distribution function (CDF) for a given value.
- **combine()** - Combines this digest with another digest.
- **compute()** - Computes the quantile for a given cumulative probability.
- **force_merge()** - Merges another digest into this one, forcing a merge even if not optimal.
- **get_centroid()** - Retrieves a specific centroid by index.
- **get_centroids()** - Returns a list of all centroids.
- **inverse_cdf()** - Computes the inverse CDF (quantile) for a given probability.
- **of_centroids()** - Creates a TDigest from a list of centroids.
- **scale_weight()** - Scales the weight of a centroid.
- **update()** - Adds a data point to the digest.

### Properties
- **mean** - The mean of the data in the digest.
- **std_dev** - The standard deviation of the data.
- **var** - The variance of the data.
- **variance** - The variance of the data (alias for var).
- **weight** - The total weight of all data points.

### Usage Example
```python
from pytdigest import TDigest

digest = TDigest()
digest.update(10)
digest.update(20)
print(digest.cdf(15))
print(digest.inverse_cdf(0.5))
```
```

--------------------------------

### pytdigest.TDigest.of_centroids()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Static method to create a TDigest from a list of centroids.

```APIDOC
## of_centroids()

### Description
Static method to create a TDigest from a list of centroids.

### Method
of_centroids()

### Parameters
- **centroids** (list of Centroid) - Required - The centroids to create the TDigest from.

### Returns
- **TDigest** - The created TDigest.
```

--------------------------------

### pytdigest.TDigest.cdf()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Computes the cumulative distribution function (CDF) for a given value.

```APIDOC
## cdf()

### Description
Computes the cumulative distribution function (CDF) for a given value.

### Method
cdf()

### Parameters
- **value** (float) - Required - The value for which to compute the CDF.

### Returns
- **float** - The CDF value.
```

--------------------------------

### TDigest.scale_weight(factor)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Scales weight by a certain factor. Can be used for instance to estimate exponentially decaying approximate quantiles.

```APIDOC
## TDigest.scale_weight(factor)

### Description
Scales weight by a certain factor. Can be used for instance to estimate exponentially decaying approximate quantiles.

### Method
scale_weight

### Parameters
- **factor** (Number) - Required - The factor by which to scale the weight.

### Returns
None
```

--------------------------------

### pytdigest.TDigest.force_merge()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Merges another TDigest into this one, forcing a merge.

```APIDOC
## force_merge()

### Description
Merges another TDigest into this one, forcing a merge.

### Method
force_merge()

### Parameters
- **other** (TDigest) - Required - The TDigest to merge into this one.

### Returns
- **None**
```

--------------------------------

### pytdigest.TDigest.get_centroids()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Returns a list of all centroids in the TDigest.

```APIDOC
## get_centroids()

### Description
Returns a list of all centroids in the TDigest.

### Method
get_centroids()

### Returns
- **list of Centroid** - The list of centroids.
```

--------------------------------

### pytdigest.TDigest.var (property)

Source: https://protivinsky.github.io/pytdigest/genindex.html

Returns the variance of the TDigest.

```APIDOC
## var (property)

### Description
Returns the variance of the TDigest.

### Returns
- **float** - The variance.
```

--------------------------------

### pytdigest.TDigest.update()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Updates the TDigest with a new value.

```APIDOC
## update()

### Description
Updates the TDigest with a new value.

### Method
update()

### Parameters
- **value** (float) - Required - The value to add.
- **weight** (float) - Optional - The weight of the value (default 1).

### Returns
- **None**
```

--------------------------------

### TDigest.force_merge()

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Forces merging of centroids of the underlying representation.

```APIDOC
## TDigest.force_merge()

### Description
Forces merging of centroids of the underlying representation.

### Method
force_merge

### Parameters
None

### Returns
None
```

--------------------------------

### pytdigest.TDigest.variance (property)

Source: https://protivinsky.github.io/pytdigest/genindex.html

Returns the variance of the TDigest.

```APIDOC
## variance (property)

### Description
Returns the variance of the TDigest.

### Returns
- **float** - The variance.
```

--------------------------------

### TDigest.variance (property)

Source: https://protivinsky.github.io/pytdigest/api/tdigest.html

Exact variance of the data (biased, i.e. normalized by total weight without doff correction).

```APIDOC
## TDigest.variance (property)

### Description
Exact variance of the data (biased, i.e. normalized by total weight without doff correction).

### Method
variance (property)

### Parameters
None

### Returns
- The exact variance of the data.
```

--------------------------------

### pytdigest.TDigest.get_centroid()

Source: https://protivinsky.github.io/pytdigest/genindex.html

Returns the centroid at a given index.

```APIDOC
## get_centroid()

### Description
Returns the centroid at a given index.

### Method
get_centroid()

### Parameters
- **index** (int) - Required - The index of the centroid.

### Returns
- **Centroid** - The centroid at the specified index.
```
Tokens:
Raw