### Basic TDigest computation and usage Source: https://protivinsky.github.io/pytdigest/index.html Creates a TDigest from weighted data using TDigest.compute, then demonstrates mean, variance, inverse CDF (quantiles), copying, and addition of TDigests. Requires numpy and pandas. The example uses random data and shows that results are close to numpy's unweighted quantiles. ```python from pytdigest import TDigest import numpy as np import pandas as pd rng = np.random.default_rng(12354) n = 100_000 x = rng.normal(loc=0, scale=10, size=n) w = rng.exponential(scale=1, size=n) # estimation from data is simple: td = TDigest.compute(x, w, compression=1_000) # td now contains "compressed" distribution: centroids with their means and weights # TDigest can be used to provide mean, variance or approximate quantiles (i.e. inverse CDF): td.mean quantiles = [0., 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1.] td.inverse_cdf(quantiles) td.variance # new in 0.1.4 # these results are close to numpy values (note that numpy provides only unweighted quantiles) np.average(x, weights=w) # mean should be exact np.quantile(x, quantiles) # TDigest can be copied td2 = td.copy() # and multiple TDigests can be added together to provide approximate quantiles for the overall dataset td + td2 ``` -------------------------------- ### TDigest.get_centroids() Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Gets all centroids as a two-dimensional array. Based on the centroids, the TDigest can be fully reconstructed (for a given compression). Conversion to centroids and back can be used for serialization. ```APIDOC ## TDigest.get_centroids() ### Description Gets all centroids as a two-dimensional array. Based on the centroids, the TDigest can be fully reconstructed (for a given compression). Conversion to centroids and back can be used for serialization. ### Method get_centroids ### Parameters None ### Returns - A two-dimensional array of centroids. ``` -------------------------------- ### TDigest.get_centroid(i) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Gets the centroid (i.e., a tuple of mean and weight) at a given index of the underlying representation. ```APIDOC ## TDigest.get_centroid(i) ### Description Gets the centroid (i.e., a tuple of mean and weight) at a given index of the underlying representation. ### Method get_centroid ### Parameters - **i** (int) - Required - Index of the centroid to retrieve. ### Returns - A tuple of mean and weight for the centroid at the given index. ``` -------------------------------- ### TDigest.compute(x, w=None, handling_invalid=HandlingInvalid.Drop, compression=100) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Estimates a TDigest directly from data. ```APIDOC ## TDigest.compute(x, w=None, handling_invalid=HandlingInvalid.Drop, compression=100) ### Description Estimates a TDigest directly from data. ### Method compute (static) ### Parameters - **x** (Number | ndarray | Series) - Required - Values to calculate the distribution of. - **w** (Number | ndarray | Series | None) - Optional - Optional weight for the values. If it is np.ndarray or pd.Series, it has to have the same (one-dimensional) shape as x. - **handling_invalid** (HandlingInvalid) - Optional - How to handle invalid values in calculation ['drop', 'raise'], default value 'drop'. Provided either as enum or its string representation. - **compression** (int) - Optional - Higher compression value leads to more precise results and larger memory requirements. Compression gives a maximum number of centroids of the internal representation after merging of the TDigest. Unmerged representation can have up to roughly six times as many centroids. ### Returns - New TDigest object estimated based on (possibly weighted) data. ### Raises - **TypeError** - If x or w are not of permitted types or of different types. - **ValueError** - If handling_invalid is 'raise' and there are invalid values in data (nan, infinity, negative weight). ``` -------------------------------- ### TDigest.combine(first, second=None) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Combines multiple TDigests together. ```APIDOC ## TDigest.combine(first, second=None) ### Description Combines multiple TDigests together. ### Method combine (static) ### Parameters - **first** (TDigest | Iterable[TDigest]) - Required - The first TDigest or an iterable of TDigests to combine. - **second** (TDigest | None) - Optional - The second TDigest to combine, if first is a single TDigest. ### Returns - A new TDigest object representing the combined distribution. ``` -------------------------------- ### Compare PyTDigest with pure Python tdigest package Source: https://protivinsky.github.io/pytdigest/index.html Benchmarks PyTDigest against the pure Python tdigest package. The output shows PyTDigest at 0.0246 seconds and TDigest at 7.26 seconds for 100,000 elements. Note that tdigest only supports different weights via the update method for single observations. ```python import numpy as np from pytdigest import TDigest from tdigest import TDigest as TDigestPython import time rng = np.random.Generator(np.random.PCG64(12345)) n = 100_000 x = rng.normal(size=n) w = rng.exponential(size=n) start = time.time() td = TDigest.compute(x, w) end = time.time() print(f'PyTDigest: {end - start:.3g} seconds') # PyTDigest: 0.0246 seconds tdp = TDigestPython() start = time.time() tdp.batch_update(x) end = time.time() print(f'TDigest: {end - start:.3g} seconds') # TDigest: 7.26 seconds ``` -------------------------------- ### pytdigest.TDigest.combine() Source: https://protivinsky.github.io/pytdigest/genindex.html Static method to combine multiple TDigest instances into one. ```APIDOC ## combine() ### Description Static method to combine multiple TDigest instances into one. ### Method combine() ### Parameters - **digests** (list of TDigest) - Required - The TDigest instances to combine. ### Returns - **TDigest** - The combined TDigest. ``` -------------------------------- ### TDigest.weight (property) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Total weight of the data. ```APIDOC ## TDigest.weight (property) ### Description Total weight of the data. ### Method weight (property) ### Parameters None ### Returns - The total weight of the data. ``` -------------------------------- ### Benchmark PyTDigest performance on numpy arrays Source: https://protivinsky.github.io/pytdigest/index.html Measures PyTDigest's compute time for varying array sizes. The output shows approximately 0.0222, 0.21, and 2.02 seconds for 100k, 1M, and 10M elements respectively. ```python import numpy as np from pytdigest import TDigest import time rng = np.random.Generator(np.random.PCG64(12345)) for n in [100_000, 1_000_000, 10_000_000]: x = rng.normal(size=n) w = rng.exponential(size=n) start = time.time() td = TDigest.compute(x, w) end = time.time() print(f'PyTDigest, n = {n:,}: {end - start:.3g} seconds') # PyTDigest, n = 100,000: 0.0222 seconds # PyTDigest, n = 1,000,000: 0.21 seconds # PyTDigest, n = 10,000,000: 2.02 seconds ``` -------------------------------- ### TDigest.update(x, w=None, handling_invalid=HandlingInvalid.Drop) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Adds new data to the TDigest. ```APIDOC ## TDigest.update(x, w=None, handling_invalid=HandlingInvalid.Drop) ### Description Adds new data to the TDigest. ### Method update ### Parameters - **x** (Number | ndarray | Series) - Required - Values to calculate the distribution of. - **w** (Number | ndarray | Series | None) - Optional - Optional weight for the values. If it is np.ndarray or pd.Series, it has to have the same (one-dimensional) shape as x. - **handling_invalid** (HandlingInvalid) - Optional - How to handle invalid values in calculation ['drop', 'raise'], default value 'drop'. Provided either as enum or its string representation. ### Raises - **TypeError** - If x or w are not of permitted types or of different types. - **ValueError** - If handling_invalid is 'raise' and there are invalid values in data (nan, infinity, negative weight). ``` -------------------------------- ### Benchmark t-digest CFFI package Source: https://protivinsky.github.io/pytdigest/index.html Benchmarks the t-digest CFFI package, which requires iterating over the array in Python because it does not pass batch updates to the C layer. The output shows 0.479 seconds for 100,000 elements, and the package only supports integer weights. ```python import numpy as np from tdigest import TDigest as TDigestCFFI import time rng = np.random.Generator(np.random.PCG64(12345)) n = 100_000 x = rng.normal(size=n) tdcffi = TDigestCFFI() start = time.time() for xx in x: tdcffi.insert(xx) end = time.time() print(f'TDigest-CFFI: {end - start:.3g} seconds') # TDigest-CFFI: 0.479 seconds ``` -------------------------------- ### TDigest.inverse_cdf(quantile) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Calculates quantiles at given points (quantiles are just inverse of cumulative distribution function). ```APIDOC ## TDigest.inverse_cdf(quantile) ### Description Calculates quantiles at given points (quantiles are just inverse of cumulative distribution function). ### Method inverse_cdf ### Parameters - **quantile** (Number | List | ndarray) - Required - The values where the inverse CDF should be calculated. Multiple values are passed to C library for fast estimation of larger number of quantiles. ### Returns - Estimated approximate quantiles (float | ndarray). ### Raises - **TypeError** - If quantile is of invalid type. ``` -------------------------------- ### TDigest.of_centroids(centroids, compression=100) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Reconstructs a TDigest from centroids and a given compression. ```APIDOC ## TDigest.of_centroids(centroids, compression=100) ### Description Reconstructs a TDigest from centroids and a given compression. ### Method of_centroids (static) ### Parameters - **centroids** (ndarray) - Required - The centroids to reconstruct the TDigest from. - **compression** (float) - Optional - The compression parameter for the reconstructed TDigest. ### Returns - A TDigest object reconstructed from the given centroids. ``` -------------------------------- ### TDigest.mean (property) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Exact mean of the data. ```APIDOC ## TDigest.mean (property) ### Description Exact mean of the data. ### Method mean (property) ### Parameters None ### Returns - The exact mean of the data. ``` -------------------------------- ### TDigest.std_dev (property) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Standard deviation, not corrected for doff. ```APIDOC ## TDigest.std_dev (property) ### Description Standard deviation, not corrected for doff. ### Method std_dev (property) ### Parameters None ### Returns - The standard deviation of the data. ``` -------------------------------- ### TDigest.cdf(at) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Calculates approximate values of the empirical cumulative distribution function (CDF) at given points. ```APIDOC ## TDigest.cdf(at) ### Description Calculates approximate values of the empirical cumulative distribution function (CDF) at given points. ### Method cdf ### Parameters - **at** (Number | List | ndarray) - Required - Values at which the CDF should be calculated. ### Returns - Values of CDF calculated at given points (float | ndarray). ### Raises - **TypeError** - If at is of invalid type. ``` -------------------------------- ### pytdigest.TDigest.mean (property) Source: https://protivinsky.github.io/pytdigest/genindex.html Returns the mean of the TDigest. ```APIDOC ## mean (property) ### Description Returns the mean of the TDigest. ### Returns - **float** - The mean. ``` -------------------------------- ### pytdigest.TDigest.compute() Source: https://protivinsky.github.io/pytdigest/genindex.html Static method to compute a TDigest from a list of values. ```APIDOC ## compute() ### Description Static method to compute a TDigest from a list of values. ### Method compute() ### Parameters - **values** (list of float) - Required - The values to compute the TDigest from. ### Returns - **TDigest** - The computed TDigest. ``` -------------------------------- ### pytdigest.TDigest.weight (property) Source: https://protivinsky.github.io/pytdigest/genindex.html Returns the total weight of the TDigest. ```APIDOC ## weight (property) ### Description Returns the total weight of the TDigest. ### Returns - **float** - The total weight. ``` -------------------------------- ### pytdigest.TDigest.inverse_cdf() Source: https://protivinsky.github.io/pytdigest/genindex.html Computes the inverse cumulative distribution function (quantile) for a given probability. ```APIDOC ## inverse_cdf() ### Description Computes the inverse cumulative distribution function (quantile) for a given probability. ### Method inverse_cdf() ### Parameters - **probability** (float) - Required - The probability for which to compute the quantile. ### Returns - **float** - The quantile value. ``` -------------------------------- ### pytdigest.TDigest.scale_weight() Source: https://protivinsky.github.io/pytdigest/genindex.html Scales the weight of a centroid. ```APIDOC ## scale_weight() ### Description Scales the weight of a centroid. ### Method scale_weight() ### Parameters - **index** (int) - Required - The index of the centroid. - **factor** (float) - Required - The scaling factor. ### Returns - **None** ``` -------------------------------- ### pytdigest.TDigest.std_dev (property) Source: https://protivinsky.github.io/pytdigest/genindex.html Returns the standard deviation of the TDigest. ```APIDOC ## std_dev (property) ### Description Returns the standard deviation of the TDigest. ### Returns - **float** - The standard deviation. ``` -------------------------------- ### TDigest.var (property) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Alias for variance. ```APIDOC ## TDigest.var (property) ### Description Alias for variance. ### Method var (property) ### Parameters None ### Returns - The variance of the data. ``` -------------------------------- ### TDigest Source: https://protivinsky.github.io/pytdigest/api/tdigest.html The TDigest class represents a t-digest data structure for accurate quantile estimation. It provides methods to update with data, compute quantiles, combine digests, and access centroids and statistics. ```APIDOC ## TDigest ### Description The TDigest class is the main entry point for creating and manipulating t-digest data structures. It supports updating with data points, computing cumulative distribution and inverse CDF, combining multiple digests, and accessing centroids and statistical properties. ### Constructor - **TDigest()** - Creates a new empty TDigest instance. ### Methods - **cdf()** - Computes the cumulative distribution function (CDF) for a given value. - **combine()** - Combines this digest with another digest. - **compute()** - Computes the quantile for a given cumulative probability. - **force_merge()** - Merges another digest into this one, forcing a merge even if not optimal. - **get_centroid()** - Retrieves a specific centroid by index. - **get_centroids()** - Returns a list of all centroids. - **inverse_cdf()** - Computes the inverse CDF (quantile) for a given probability. - **of_centroids()** - Creates a TDigest from a list of centroids. - **scale_weight()** - Scales the weight of a centroid. - **update()** - Adds a data point to the digest. ### Properties - **mean** - The mean of the data in the digest. - **std_dev** - The standard deviation of the data. - **var** - The variance of the data. - **variance** - The variance of the data (alias for var). - **weight** - The total weight of all data points. ### Usage Example ```python from pytdigest import TDigest digest = TDigest() digest.update(10) digest.update(20) print(digest.cdf(15)) print(digest.inverse_cdf(0.5)) ``` ``` -------------------------------- ### pytdigest.TDigest.of_centroids() Source: https://protivinsky.github.io/pytdigest/genindex.html Static method to create a TDigest from a list of centroids. ```APIDOC ## of_centroids() ### Description Static method to create a TDigest from a list of centroids. ### Method of_centroids() ### Parameters - **centroids** (list of Centroid) - Required - The centroids to create the TDigest from. ### Returns - **TDigest** - The created TDigest. ``` -------------------------------- ### pytdigest.TDigest.cdf() Source: https://protivinsky.github.io/pytdigest/genindex.html Computes the cumulative distribution function (CDF) for a given value. ```APIDOC ## cdf() ### Description Computes the cumulative distribution function (CDF) for a given value. ### Method cdf() ### Parameters - **value** (float) - Required - The value for which to compute the CDF. ### Returns - **float** - The CDF value. ``` -------------------------------- ### TDigest.scale_weight(factor) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Scales weight by a certain factor. Can be used for instance to estimate exponentially decaying approximate quantiles. ```APIDOC ## TDigest.scale_weight(factor) ### Description Scales weight by a certain factor. Can be used for instance to estimate exponentially decaying approximate quantiles. ### Method scale_weight ### Parameters - **factor** (Number) - Required - The factor by which to scale the weight. ### Returns None ``` -------------------------------- ### pytdigest.TDigest.force_merge() Source: https://protivinsky.github.io/pytdigest/genindex.html Merges another TDigest into this one, forcing a merge. ```APIDOC ## force_merge() ### Description Merges another TDigest into this one, forcing a merge. ### Method force_merge() ### Parameters - **other** (TDigest) - Required - The TDigest to merge into this one. ### Returns - **None** ``` -------------------------------- ### pytdigest.TDigest.get_centroids() Source: https://protivinsky.github.io/pytdigest/genindex.html Returns a list of all centroids in the TDigest. ```APIDOC ## get_centroids() ### Description Returns a list of all centroids in the TDigest. ### Method get_centroids() ### Returns - **list of Centroid** - The list of centroids. ``` -------------------------------- ### pytdigest.TDigest.var (property) Source: https://protivinsky.github.io/pytdigest/genindex.html Returns the variance of the TDigest. ```APIDOC ## var (property) ### Description Returns the variance of the TDigest. ### Returns - **float** - The variance. ``` -------------------------------- ### pytdigest.TDigest.update() Source: https://protivinsky.github.io/pytdigest/genindex.html Updates the TDigest with a new value. ```APIDOC ## update() ### Description Updates the TDigest with a new value. ### Method update() ### Parameters - **value** (float) - Required - The value to add. - **weight** (float) - Optional - The weight of the value (default 1). ### Returns - **None** ``` -------------------------------- ### TDigest.force_merge() Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Forces merging of centroids of the underlying representation. ```APIDOC ## TDigest.force_merge() ### Description Forces merging of centroids of the underlying representation. ### Method force_merge ### Parameters None ### Returns None ``` -------------------------------- ### pytdigest.TDigest.variance (property) Source: https://protivinsky.github.io/pytdigest/genindex.html Returns the variance of the TDigest. ```APIDOC ## variance (property) ### Description Returns the variance of the TDigest. ### Returns - **float** - The variance. ``` -------------------------------- ### TDigest.variance (property) Source: https://protivinsky.github.io/pytdigest/api/tdigest.html Exact variance of the data (biased, i.e. normalized by total weight without doff correction). ```APIDOC ## TDigest.variance (property) ### Description Exact variance of the data (biased, i.e. normalized by total weight without doff correction). ### Method variance (property) ### Parameters None ### Returns - The exact variance of the data. ``` -------------------------------- ### pytdigest.TDigest.get_centroid() Source: https://protivinsky.github.io/pytdigest/genindex.html Returns the centroid at a given index. ```APIDOC ## get_centroid() ### Description Returns the centroid at a given index. ### Method get_centroid() ### Parameters - **index** (int) - Required - The index of the centroid. ### Returns - **Centroid** - The centroid at the specified index. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.