### Import libraries Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Initial setup importing numpy, pandas, and the benford module. ```python %matplotlib inline import numpy as np import pandas as pd ``` ```python import benford as bf ``` -------------------------------- ### Install Benford_py with pip Source: https://github.com/milcent/benford_py/blob/master/README-pypi.md Use pip to install the benford-py package. This is the recommended method for most users. ```bash pip install benford_py ``` ```bash pip install benford-py ``` -------------------------------- ### Clone repository Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Alternative installation method by cloning the source code directly. ```bash $ git clone http://github.com/milcent/benford_py.git ``` -------------------------------- ### Install Benford_py with Git Clone Source: https://github.com/milcent/benford_py/blob/master/README-pypi.md Clone the repository directly into your Python site-packages directory for development or specific use cases. ```bash git clone https://github.com/milcent/benford_py ``` -------------------------------- ### GET /duplicates Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Performs a duplicates test on the data source and returns the count of duplicated entries in descending order. ```APIDOC ## GET /duplicates ### Description Performs a duplicates test and maps the duplicates count in descending order. ### Parameters #### Query Parameters - **top_Rep** (int or None) - Optional - Chooses how many duplicated entries will be shown with the top repetitions. Defaults to 20. If None, returns all ordered repetitions. - **inform** (Any) - Optional - Additional information parameter. ### Response #### Success Response (200) - **DataFrame** (object) - DataFrame with the duplicated records and their occurrence counts, in descending order. ``` -------------------------------- ### Instantiate Benford Object with Sample Size Limit Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Initialize the Benford analysis with a custom sample size limit to improve test robustness. ```python benf = bf.Benford((sp, 'l_r'), decimals=8, limit_N=1800) ``` -------------------------------- ### Constructor: Initialize Benford Analysis Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Initializes the Benford analysis object with a sequence of numbers and configuration parameters for data filtering and processing. ```APIDOC ## __init__(data, decimals=2, sign='all', sec_order=False, verbose=True, inform=None) ### Description Initializes the analysis sequence, validates input types, and applies filtering based on sign and second-order test requirements. ### Parameters #### Request Body - **data** (array/Series) - Required - Sequence of numbers (int or float). - **decimals** (int/str) - Optional - Number of decimal places to consider. Defaults to 2. Use 'infer' for automatic detection. - **sign** (str) - Optional - Portion of data to consider: 'all', 'pos', or 'neg'. Defaults to 'all'. - **sec_order** (bool) - Optional - Whether to perform the Second Order Test. Defaults to False. - **verbose** (bool) - Optional - Whether to print initialization details. Defaults to True. ``` -------------------------------- ### Perform Summation test Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Performs the Summation test on a Benford series, checking if sums of entries starting with the same digits are consistent. ```python def summation(data, digs=2, decimals=2, sign='all', top=20, verbose=True, show_plot=True, save_plot=None, save_plot_kwargs=None, inform=None): """Performs the Summation test. In a Benford series, the sums of the entries begining with the same digits tends to be the same. Works only with the First Digits (1, 2 or 3) test. Args: digs: tells the first digits to use: 1- first; 2- first two; 3- first three. Defaults to 2. decimals: number of decimal places to consider. Defaluts to 2. If integers, set to 0. If set to -infer-, it will remove the zeros and consider up to the fifth decimal place to the right, but will loose performance. top: choses how many top values to show. Defaults to 20. show_plot: plots the results. Defaults to True. save_plot (str): string with the path/name of the file in which the generated plot will be saved. Uses matplotlib.pyplot.savefig(). File format is infered by the file name extension. Only available when plot=True. save_plot_kwargs (dict): any of the kwargs accepted by matplotlib.pyplot.savefig() https://matplotlib.org/api/_as_gen/matplotlib.pyplot.savefig.html Only available when plot=True and save_plot is a string with the figure file path/name. Returns: DataFrame with the Summation test, whether sorted in descending order (if verbose == True) or not. """ verbose = _deprecate_inform_(verbose, inform) if not isinstance(data, Source): data = Source(data, sign=sign, decimals=decimals, verbose=verbose) data = data.summation(digs=digs, top=top, show_plot=show_plot, save_plot=save_plot, save_plot_kwargs=save_plot_kwargs, ret_df=True) if verbose: return data.sort_values('AbsDif', ascending=False) else: return data ``` -------------------------------- ### Source.sec_order() Method Source: https://github.com/milcent/benford_py/blob/master/docs/source/api.md Runs the Second Order tests, which are Benford’s tests performed on the differences between the ordered sample. ```APIDOC ## Method sec_order() ### Description Runs the Second Order tests, which are the Benford’s tests performed on the differences between the ordered sample (a value minus the one before it, and so on). If the original series is Benford-compliant, this new sequence should also follow Benford. The Second Order can also be called separately, through the method sec_order(). ``` -------------------------------- ### Get Mantissas Statistics Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Calculates and returns key statistics for the mantissas, including mean, variance, skewness, kurtosis, and Kolmogorov-Smirnov test results against critical values. ```python @property def stats(self): # (dict): Dictionary with the mantissas statistics ks, crit_ks = _mantissas_ks_(self.data.Mantissa.values, self.confidence, self.limit_N) return {'Mean': self.data.Mantissa.mean(), 'Var': self.data.Mantissa.var(), 'Skew': self.data.Mantissa.skew(), 'Kurt': self.data.Mantissa.kurt(), 'KS': ks, 'KS_critical': crit_ks} ``` -------------------------------- ### Summation Test API Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/api.html Performs the Summation test to check if sums of entries starting with the same digits tend to be the same, following Benford's Law principles. Works with First Digits (1, 2, or 3) tests. ```APIDOC ## benford.benford.summation ### Description Performs the Summation test. In a Benford series, the sums of the entries beginning with the same digits tend to be the same. Works only with the First Digits (1, 2 or 3) test. ### Method `benford.benford.summation` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body * **data** (sequence) - Required - The data to perform the summation test on. * **digs** (int) - Optional - Specifies the first digits to use: 1 for the first digit, 2 for the first two digits, 3 for the first three digits. Defaults to 2. * **decimals** (int) - Optional - Number of decimal places to consider. Defaults to 2. * **sign** (string) - Optional - Specifies which portion of the data to consider: 'pos' for positive entries, 'neg' for negative entries, 'all' for all entries except zeros. Defaults to 'all'. * **top** (int) - Optional - Sets the limit for the number of top entries to consider in the summation. Defaults to 20. * **verbose** (bool) - Optional - If True, displays detailed information about the analysis. Defaults to True. * **show_plot** (bool) - Optional - If True, draws the test plot. Defaults to True. * **save_plot** (str) - Optional - String with the path/name of the file to save the generated plot. File format is inferred from the extension. Only available when `show_plot` is True. * **save_plot_kwargs** (dict) - Optional - Any keyword arguments accepted by `matplotlib.pyplot.savefig()`. Only available when `show_plot` is True and `save_plot` is specified. ### Request Example ```json { "data": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "digs": 1, "top": 10, "show_plot": true } ``` ### Response #### Success Response (200) - **DataFrame** (pandas DataFrame) - Results of the summation test. #### Response Example ```json { "Digit": [1, 2, 3, 4, 5, 6, 7, 8, 9], "Sum": [100, 50, 30, 20, 15, 10, 8, 6, 5] } ``` ``` -------------------------------- ### Initialize Benford Analysis Object Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html The constructor processes input data and initializes test DataFrames for various digit positions based on the provided configuration. ```python def __init__(self, data, decimals=2, sign='all', confidence=95, mantissas=True, sec_order=False, summation=False, limit_N=None, verbose=True): self.data, self.chosen = input_data(data) self.decimals = decimals self.sign = sign self.confidence = _check_confidence_(confidence) self.limit_N = limit_N self.verbose = verbose self.base = Base(self.chosen, decimals, sign) self.tests = [] # Create a DatFrame for each Test for key, val in DIGS.items(): test = Test(self.base.loc[self.base[val] != -1], digs=key, confidence=self.confidence, limit_N=self.limit_N) setattr(self, val, test) self.tests.append(val) # dict with the numbers of discarded entries for each test column self._discarded = {key: val for (key, val) in zip(DIGS.values(), [len(self.base[col].loc[self.base[col] == -1]) for col in DIGS.values()])} if self.verbose: print('\n', ' Benford Object Instantiated '.center(50, '#'), '\n') print(f'Initial sample size: {len(self.chosen)}.\n') ``` -------------------------------- ### Load and process data Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Loading S&P500 data and calculating simple and log returns. ```python sp = pd.read_csv('data/SPY.csv', index_col='Date', parse_dates=True) ``` ```python #adding '_' to facilitate handling the column sp.rename(columns={'Adj Close':'Adj_Close'}, inplace=True) sp['p_r'] = sp.Close/sp.Close.shift()-1 #simple returns sp['l_r'] = np.log(sp.Close/sp.Close.shift()) #log returns sp.tail() ``` -------------------------------- ### Get Benford Test Critical Values Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Returns a dictionary containing critical values for the current test, based on the confidence level. Includes values for Z-score, Kolmogorov-Smirnov (KS), Chi-square, and Mean Absolute Deviation (MAD). ```python @property def critical_values(self): """dict: a dictionary with the critical values for the test at hand, according to the current confidence level.""" crit_ks = CRIT_KS[self.confidence] / (self.limit_N ** 0.5) if self.confidence\ else None return {'Z': CONFS[self.confidence], 'KS': crit_ks, 'chi2': CRIT_CHI2[self.ddf][self.confidence], 'MAD': MAD_CONFORM[self.digs]} ``` -------------------------------- ### POST /Benford Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Initializes a new Benford analysis object with the provided dataset and configuration parameters. ```APIDOC ## POST /Benford ### Description Initializes a Benford Analysis object and computes the proportions for the digits based on the provided data. ### Parameters #### Request Body - **data** (array/Series) - Required - Sequence of numbers to be evaluated. - **decimals** (int) - Optional - Number of decimal places to consider. Defaults to 2. - **sign** (str) - Optional - Portion of data to consider (pos, neg, all). Defaults to 'all'. - **confidence** (int/float) - Optional - Confidence level for limits and critical values. Defaults to 95. - **mantissas** (bool) - Optional - Whether to run the mantissas test. Defaults to True. - **sec_order** (bool) - Optional - Whether to run second order tests. Defaults to False. - **summation** (bool) - Optional - Whether to create summation DataFrames. Defaults to False. - **limit_N** (int) - Optional - Sample size limit for Z score calculations. Defaults to None. - **verbose** (bool) - Optional - Whether to print information about the process. Defaults to True. ``` -------------------------------- ### Initialize Benford Analysis Class Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Initializes the class with input data, performing validation on the sign and data type, and optionally applying a Second Order Test. ```python def __init__(self, data, decimals=2, sign='all', sec_order=False, verbose=True, inform=None): if sign not in ['all', 'pos', 'neg']: raise ValueError("The -sign- argument must be " "'all','pos' or 'neg'.") DataFrame.__init__(self, {'seq': data}) if self.seq.dtype != 'float' and self.seq.dtype != 'int': raise TypeError('The sequence dtype was neither int nor float.\n' 'Convert it to whether int or float, and try again.') if sign == 'pos': self.seq = self.seq.loc[self.seq > 0] elif sign == 'neg': self.seq = self.seq.loc[self.seq < 0] else: self.seq = self.seq.loc[self.seq != 0] self.dropna(inplace=True) #: (bool): verbose or not self.verbose = _deprecate_inform_(verbose, inform) if self.verbose: print(f"\nInitialized sequence with {len(self)} registries.") if sec_order: self.seq = subtract_sorted(self.seq.copy()) self.dropna(inplace=True) self.reset_index(inplace=True) if verbose: print('Second Order Test. Initial series reduced ' f'to {len(self.seq)} entries.') ab = self.seq.abs() if self.seq.dtype == 'int': self['ZN'] = ab else: if decimals == 'infer': # There is some numerical issue with Windows that required # implementing it differently (and slower) self['ZN'] = ab.astype(str)\ .str.replace('.', '', regex=False)\ .str.lstrip('0').str[:5]\ .astype(int) else: self['ZN'] = (ab * (10 ** decimals)).astype(int) ``` -------------------------------- ### Benford Analysis Initialization Source: https://github.com/milcent/benford_py/blob/master/docs/source/api.md Initializes a Benford Analysis object and computes the proportions for the digits. Various parameters can be set to customize the analysis. ```APIDOC ## class benford.benford.Benford(data, decimals=2, sign='all', confidence=95, mantissas=True, sec_order=False, summation=False, limit_N=None, verbose=True) ### Description Initializes a Benford Analysis object and computes the proportions for the digits. The tets dataFrames are atributes, i.e., obj.F1D is the First Digit DataFrame, the obj.F2D,the First Two Digits one, and so one, F3D for First Three Digits, SD for Second Digit and L2D for Last Two Digits. ### Parameters #### Parameters - **data** – sequence of numbers to be evaluated. Must be a numpy 1D array, a pandas Series or a tuple with a pandas DataFrame and the name (str) of the chosen column. Values must be integers or floats. - **decimals** – number of decimal places to consider. Defaluts to 2. If integers, set to 0. If set to -infer-, it will remove the zeros and consider up to the fifth decimal place to the right, but will loose performance. - **sign** – tells which portion of the data to consider. pos: only the positive entries; neg: only negative entries; all: all entries but zeros. Defaults to all. - **confidence** (*int* *,* *float*) – confidence level to draw lower and upper limits when plotting and to limit the top deviations to show, as well as to calculate critical values for the tests’ statistics. Defaults to 95. - **mantissas** (*bool*) – opts for also running the mantissas Test. Defaulst to True - **sec_order** – runs the Second Order tests, which are the Benford’s tests performed on the differences between the ordered sample (a value minus the one before it, and so on). If the original series is Benford-compliant, this new sequence should aldo follow Beford. The Second Order can also be called separately, through the method sec_order(). - **summation** – creates the Summation DataFrames for the First, First Two, and First Three Digits. The summation tests can also be called separately, through the method summation(). - **limit_N** (*int*) – sets a limit to N as the sample size for the calculation of the Z scores if the sample is too big. Defaults to None. - **verbose** – gives some information about the data and the registries used and discarded for each test. ### Attributes #### data the raw data provided for the analysis #### chosen the column of the DataFrame to be analysed or the data itself #### sign which number sign(s) to include in the analysis * **Type:** str #### confidence current confidence level #### limit_N sample size to use in computations * **Type:** int #### verbose verbose or not * **Type:** bool #### base the Base, pre-processed object #### tests keeps track of the tests the instance has * **Type:** `list` of `str` ``` -------------------------------- ### Source Class Initialization Source: https://github.com/milcent/benford_py/blob/master/docs/source/api.md Initializes the Source object, preparing data for Benford's Law analysis. It's a pandas DataFrame subclass. ```APIDOC ## Class benford.benford.Source ### Description Prepares the data for Analysis. pandas DataFrame subclass. ### Parameters * **data** (sequence of numbers) - Required - Sequence of numbers to be evaluated. Must be a numpy 1D array, a pandas Series or a pandas DataFrame column, with values being integers or floats. * **decimals** (int or 'infer') - Optional - Number of decimal places to consider. Defaults to 2. If integers, set to 0. If set to -infer-, it will remove the zeros and consider up to the fifth decimal place to the right, but will loose performance. * **sign** (str) - Optional - Tells which portion of the data to consider. 'pos': only the positive entries; 'neg': only negative entries; 'all': all entries but zeros. Defaults to 'all'. * **sec_order** (bool) - Optional - Choice for the Second Order Test, which computes the differences between the ordered entries before running the Tests. * **verbose** (bool) - Optional - Tells the number of registries that are being subjected to the analysis; defaults to True. * **inform** (any) - Optional - Not described in the source text. ### Raises * **ValueError** - if the sign arg is not in ['all', 'pos', 'neg'] * **TypeError** - if not receiving int or float as input. ``` -------------------------------- ### Initialize Base Class for Benford's Law Analysis Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Initializes the Base class, preparing data for Benford's Law analysis. Handles data type checks, filtering by sign (positive, negative, or all non-zero), and calculating normalized values (ZN) based on the specified number of decimals or inferring them. ```python class Base(DataFrame): """Internalizes and prepares the data for Analysis. Args: data: sequence of numbers to be evaluated. Must be a numpy 1D array, a pandas Series or a pandas DataFrame column, with values being integers or floats. decimals: number of decimal places to consider. Defaluts to 2. If integers, set to 0. If set to -infer-, it will remove the zeros and consider up to the fifth decimal place to the right, but will loose performance. sign: tells which portion of the data to consider. pos: only the positive entries; neg: only negative entries; all: all entries but zeros. Defaults to all. Raises: TypeError: if not receiving `int` or `float` as input. """ def __init__(self, data, decimals, sign='all', sec_order=False): DataFrame.__init__(self, {'seq': data}) if (self.seq.dtype != 'float') & (self.seq.dtype != 'int'): raise TypeError("The sequence dtype was neither int nor " "float. Convert it to whether int of float, " "and try again.") if sign == 'all': self.seq = self.seq.loc[self.seq != 0] elif sign == 'pos': self.seq = self.seq.loc[self.seq > 0] else: self.seq = self.seq.loc[self.seq < 0] self.dropna(inplace=True) ab = self.seq.abs() if self.seq.dtype == 'int': self['ZN'] = ab else: if decimals == 'infer': self['ZN'] = ab.astype(str).str\ .replace('.', '', regex=False)\ .str.lstrip('0')\ .str[:5].astype(int) else: self['ZN'] = (ab * (10 ** decimals)).astype(int) # First digits for col in ['F1D', 'F2D', 'F3D']: temp = self.ZN.loc[self.ZN >= 10 ** (REV_DIGS[col] - 1)] self[col] = (temp // 10 ** (log10(temp).astype(int)) - (REV_DIGS[col] - 1))) # fill NANs with -1, which is a non-usable value for digits, # to be discarded later. self[col] = self[col].fillna(-1).astype(int) # Second digit temp_sd = self.loc[self.ZN >= 10] self['SD'] = (temp_sd.ZN // 10**((log10(temp_sd.ZN)).astype(int) - 1)) % 10 self['SD'] = self['SD'].fillna(-1).astype(int) # Last two digits temp_l2d = self.loc[self.ZN >= 1000] self['L2D'] = temp_l2d.ZN % 100 self['L2D'] = self['L2D'].fillna(-1).astype(int) ``` -------------------------------- ### Benford Analysis Initialization Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/api.html Initializes a Benford Analysis object and computes the proportions for the digits. Various test DataFrames are available as attributes. ```APIDOC ## Benford Analysis Initialization ### Description Initializes a Benford Analysis object and computes the proportions for the digits. The test DataFrames are attributes, i.e., obj.F1D is the First Digit DataFrame, obj.F2D is the First Two Digits DataFrame, and so on. SD is the Second Digit DataFrame, and L2D is the Last Two Digits DataFrame. ### Parameters #### Request Body - **data** (sequence) - Required - Sequence of numbers to be evaluated. Must be a numpy 1D array, a pandas Series, or a tuple with a pandas DataFrame and the name (str) of the chosen column. Values must be integers or floats. - **decimals** (int) - Optional - Number of decimal places to consider. Defaults to 2. If integers, set to 0. If set to -infer-, it will remove the zeros and consider up to the fifth decimal place to the right, but will lose performance. - **sign** (str) - Optional - Specifies which portion of the data to consider. Options: 'pos' (only positive entries), 'neg' (only negative entries), 'all' (all entries but zeros). Defaults to 'all'. - **confidence** (int) - Optional - Confidence level for tests. Defaults to 95. - **mantissas** (bool) - Optional - Whether to consider mantissas for analysis. Defaults to True. - **sec_order** (bool) - Optional - Whether to consider second-order analysis. Defaults to False. - **summation** (bool) - Optional - Whether to consider summation for analysis. Defaults to False. - **limit_N** (int) - Optional - Limit for N values. Defaults to None. - **verbose** (bool) - Optional - Whether to display verbose output. Defaults to True. ``` -------------------------------- ### benford.stats.kolmogorov_smirnov_2 (simplified) Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/api.html Computes the Kolmogorov-Smirnov test of the found distributions. ```APIDOC ## GET /benford/stats/kolmogorov_smirnov_2_simplified ### Description Computes the Kolmogorov-Smirnov test of the found distributions. ### Method GET ### Endpoint /benford/stats/kolmogorov_smirnov_2_simplified ### Parameters #### Query Parameters - **frame** (DataFrame) - Required - DataFrame with Foud and Expected distributions. ### Response #### Success Response (200) - **Suprem** (float) - The greatest absolute difference between the Found and the expected proportions. #### Response Example ```json { "Suprem": 0.05 } ``` ``` -------------------------------- ### Instantiate Benford Object Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Initializes a Benford object with a DataFrame and a specific column name for analysis. ```python benf = bf.Benford((sp, 'l_r'), decimals=8) ``` -------------------------------- ### Analyze Data with Source Class Source: https://context7.com/milcent/benford_py/llms.txt Uses the Source class for granular control and method chaining across multiple Benford tests. ```python import benford as bf import pandas as pd # Load data df = pd.read_csv('data/SPY.csv') data = df['Close'].values # Create Source object source = bf.Source( data, decimals=2, sign='all', # 'all', 'pos', 'neg' sec_order=False, # set True for second order analysis verbose=True ) # Output: Initialized sequence with 5282 registries. # Run individual tests on the source source.first_digits( digs=1, confidence=95, high_Z='pos', MAD=True, MSE=True, chi_square=True, KS=True, show_plot=True ) # Run second digit test source.second_digit(confidence=95, MAD=True, show_plot=True) # Run last two digits test source.last_two_digits(confidence=95, show_plot=True) # Run summation test source.summation(digs=2, top=20, show_plot=True) # Run mantissas analysis source.mantissas(report=True, show_plot=True) # Find duplicates source.duplicates(top_Rep=20) ``` -------------------------------- ### Summ Class Initialization Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Initializes the Summation test object by grouping data and calculating statistical metrics like MAD and MSE. ```python def __init__(self, base, test): super(Summ, self).__init__(base.abs() .groupby(test)[['seq']] .sum()) self['Percent'] = self.seq / self.seq.sum() self.columns.values[0] = 'Sum' self.expected = 1 / len(self) self['AbsDif'] = (self.Percent - self.expected).abs() self.index = self.index.astype(int) #: Mean Absolute Deviation for the test self.MAD = self.AbsDif.mean() self.MSE = (self.AbsDif ** 2).mean() #: Confidence level to consider when setting some critical values self.confidence = None # (int): numerical representation of the test at hand self.digs = REV_DIGS[test] # (str): the name of the Summation test. self.name = TEST_NAMES[f'{test}_Summ'] ``` -------------------------------- ### Mantissas Class Initialization Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Initializes the Mantissas object with data and confidence level. ```APIDOC ## __init__ Mantissas Class ### Description Initializes the Mantissas object with a sequence of numbers (data) and an optional confidence level. It processes the data to remove zeros and negative values, calculates mantissas, and stores them in a pandas DataFrame. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **data** (numpy 1D array, pandas Series, or pandas DataFrame column) - Required - Sequence to compute mantissas from. - **confidence** (int) - Optional - Confidence level for computing critical values. Defaults to 95. - **limit_N** (int) - Optional - Limit for the number of data points to consider. ``` -------------------------------- ### Perform Statistical Tests with Chi-Square and KS Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Execute Chi-Square and Kolmogorov-Smirnov tests by setting their respective arguments to True. ```python sd = bf.second_digit(sp.l_r, decimals=8, MAD=True, confidence=95, chi_square=True, KS=True, show_plot=False) ``` -------------------------------- ### Show Plot Method Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Generates a plot of the ordered mantissas. ```APIDOC ## show_plot ### Description Plots the ordered mantissas against an expected inclination line. Allows customization of figure size and saving the plot to a file. ### Method GET ### Endpoint /mantissas/show_plot ### Parameters #### Path Parameters None #### Query Parameters - **figsize** (tuple) - Optional - Dimensions of the figure (width, height). Defaults to (12, 6). - **save_plot** (str) - Optional - File path to save the generated plot. If provided, the plot will be saved to this file. - **save_plot_kwargs** (dict) - Optional - Additional keyword arguments to pass to the plot saving function (e.g., `matplotlib.pyplot.savefig`). ### Request Example ```json { "figsize": [10, 5], "save_plot": "ordered_mantissas.png" } ``` ### Response #### Success Response (200) Displays or saves the plot of ordered mantissas. No specific data is returned in the response body. #### Response Example (No response body for success) ``` -------------------------------- ### Generate Expected Benford Distributions Source: https://context7.com/milcent/benford_py/llms.txt Creates and visualizes theoretical Benford distributions for first, second, and last two digits. ```python from benford.expected import First, Second, LastTwo # First Digit expected distribution f1d_expected = First(digs=1, plot=True, save_plot='expected_f1d.png') print(f1d_expected) # Expected # First_1_Dig # 1 0.301030 # 2 0.176091 # 3 0.124939 # 4 0.096910 # 5 0.079181 # 6 0.066947 # 7 0.057992 # 8 0.051153 # 9 0.045757 # First Two Digits expected distribution f2d_expected = First(digs=2, plot=True) print(f2d_expected.head(10)) # Expected # First_2_Dig # 10 0.041393 # 11 0.037827 # 12 0.034678 # ... # Second Digit expected distribution sd_expected = Second(plot=True) print(sd_expected) # Expected # Sec_Dig # 0 0.119679 # 1 0.113890 # 2 0.108821 # ... # Last Two Digits (uniform distribution) l2d_expected = LastTwo(plot=True) print(l2d_expected.head()) ``` -------------------------------- ### Generate First Two Digits Test Report Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Execute the `report` method on the F2D object to obtain a comprehensive analysis of the first two digits distribution. The output details conformity metrics and highlights entries with notable deviations. ```python benf.F2D.report() ``` -------------------------------- ### Generate First Digit Test Report Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Use the `report` method on the F1D object to generate a detailed conformity report for the first digit distribution. This report includes statistical test results and significant deviations. ```python benf.F1D.report() ``` -------------------------------- ### Class: benford.benford.Base Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/api.html Initializes and prepares a sequence of numbers for Benford's Law analysis. ```APIDOC ## Class: benford.benford.Base ### Description Internalizes and prepares the data for Analysis. ### Parameters - **data** (sequence) - Required - Sequence of numbers to be evaluated (numpy 1D array, pandas Series, or DataFrame column). - **decimals** (int) - Optional - Number of decimal places to consider. Defaults to 2. - **sign** (str) - Optional - Portion of data to consider: 'pos', 'neg', or 'all'. Defaults to 'all'. ### Errors - **TypeError** - Raised if input is not int or float. ``` -------------------------------- ### Perform First and Second Digit Tests with MAD Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Calculate conformity limits using the Mean Absolute Deviation (MAD) parameter. Requires verbose mode to be enabled for output. ```python f2d = bf.first_digits(sp.l_r, digs=2, decimals=8, MAD=True, show_plot=False) ``` ```python sd = bf.second_digit(sp.l_r, decimals=8, MAD=True, show_plot=False) ``` -------------------------------- ### Summ.show_plot Method Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Generates and displays the Summation test plot, with optional file saving capabilities using matplotlib. ```python def show_plot(self, save_plot=None, save_plot_kwargs=None): """Draws the Summation test plot Args: save_plot (str): string with the path/name of the file in which the generated plot will be saved. Uses matplotlib.pyplot.savefig(). File format is infered by the file name extension. save_plot_kwargs (dict): any of the kwargs accepted by matplotlib.pyplot.savefig() https://matplotlib.org/api/_as_gen/matplotlib.pyplot.savefig.html Only available when save_plot is a string with the figure file path/name. """ figsize=(2 * (self.digs ** 2 + 5), 1.5 * (self.digs ** 2 + 5)) plot_sum(self, figsize, self.expected, save_plot=save_plot, save_plot_kwargs=save_plot_kwargs) ``` -------------------------------- ### Benford Package Modules Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_sources/api.rst.txt Overview of the available modules within the benford package for statistical analysis and visualization. ```APIDOC ## Module Overview ### benford.benford Provides core functionality for Benford's Law analysis. ### benford.expected Contains utilities for calculating expected distributions based on Benford's Law. ### benford.stats Provides statistical methods to evaluate data against Benford's Law distributions. ### benford.viz Provides visualization tools to plot and compare data distributions. ``` -------------------------------- ### Source Class Attributes and Methods Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/genindex.html Details on attributes and methods of the Source class. ```APIDOC ## second_digit() (Source method) ### Description Calculates the second digit distribution for the Source. ### Method Method of the Source class. ### Endpoint N/A ### Parameters None explicitly listed. ### Request Example N/A ### Response Second digit distribution. ## summation() (Source method) ### Description Performs summation calculations for the Source. ### Method Method of the Source class. ### Endpoint N/A ### Parameters None explicitly listed. ### Request Example N/A ### Response Summation results. ## verbose (Source attribute) ### Description Attribute controlling the verbosity of the Source class output. ### Method N/A (Attribute) ### Endpoint N/A ### Parameters None explicitly listed. ### Request Example N/A ### Response Verbosity setting. ``` -------------------------------- ### Run Summation Test Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Generates Summation test DataFrames for F1D, F2D, and F3D tests based on the Base object. ```python def summation(self): """Creates Summation test DataFrames from Base object""" for test in ['F1D', 'F2D', 'F3D']: t = f'{test}_Summ' setattr(self, t, Summ(self.base, test)) self.tests.append(t) if self.verbose: print('\nAdded Summation DataFrames to F1D, F2D and F3D Tests.') ``` -------------------------------- ### Source Class Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Prepares the data for Analysis. This class is a subclass of pandas DataFrame. ```APIDOC ## Class: Source ### Description Prepares the data for Analysis. pandas DataFrame subclass. ### Parameters #### Args: - **data** (pandas.DataFrame or similar) - The input data to be processed. - **decimals** (int, optional) - The number of decimal places to consider. Defaults to 1. - **sign** (str, optional) - Specifies the sign to consider ('positive' or 'negative'). Defaults to 'positive'. ### Methods This class inherits methods from pandas.DataFrame and may have additional methods specific to data preparation for Benford's Law analysis. ``` -------------------------------- ### Show Plot Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/api.html Plots the ordered mantissas and the expected inclination line. ```APIDOC ## POST /show_plot ### Description Plots the ordered mantissas and a line representing the expected inclination. ### Method POST ### Endpoint /show_plot ### Parameters #### Request Body - **figsize** (tuple) - Optional - Figure size dimensions. Defaults to (12, 6). - **save_plot** (str) - Optional - Path/name of the file to save the generated plot. File format is inferred by the file name extension. - **save_plot_kwargs** (dict) - Optional - Additional keyword arguments accepted by matplotlib.pyplot.savefig(). Only available when save_plot is provided. ``` -------------------------------- ### Initialize Test Class for Benford's Law Analysis Source: https://github.com/milcent/benford_py/blob/master/docs/build/html/_modules/benford/benford.html Initializes the Test class, which transforms the data prepared by the Base class into a DataFrame suitable for Benford's Law analysis. It computes columns based on the specified digits to test (first, second, first two, first three, or last two digits). ```python class Test(DataFrame): """Transforms the original number sequence into a DataFrame reduced by the ocurrences of the chosen digits, creating other computed columns Args: base: The Base object with the data prepared for Analysis digs: Tells which test to perform: 1: first digit; 2: first two digits; 3: furst three digits; 22: second digit; -2: last two digits. confidence (int, float): confidence level to draw lower and upper limits when plotting and to limit the top deviations to show. limit_N (int): sets a limit to N as the sample size for the calculation of ``` -------------------------------- ### First Three Digits Test on Log Returns Source: https://github.com/milcent/benford_py/blob/master/Demo.ipynb Performs the first three digits test on log returns with a 99% confidence level. Use 'digs=3' for the first three digits and 'decimals=8' for high precision. ```python # First Three Digits Test, now with 99% confidence level # digs=3 for the first three digits f3d = bf.first_digits(sp.l_r, digs=3, decimals=8, confidence=99) ``` -------------------------------- ### Benford Class - Comprehensive Analysis Source: https://context7.com/milcent/benford_py/llms.txt Instantiate the Benford class to perform all digit tests at once on sample data. Access test results as attributes and view statistics. The confidence level can be updated, and summation tests can be added. ```python import numpy as np import pandas as pd import benford as bf # Generate sample data (financial transactions) np.random.seed(42) data = np.random.exponential(scale=1000, size=5000) # Create Benford analysis object with all tests ben = bf.Benford( data, decimals=2, # decimal places to consider sign='all', # 'all', 'pos', or 'neg' confidence=95, # confidence level for statistical tests mantissas=True, # include mantissas test sec_order=False, # second order test summation=False, # summation test verbose=True # print analysis info ) # Output: # Benford Object Instantiated # Initial sample size: 5000. # Test performed on 5000 registries. # Number of discarded entries for each test: {'F1D': 0, 'F2D': 0, 'F3D': 183, 'SD': 0, 'L2D': 2127} # Access test DataFrames print(ben.F1D) # First Digit test results # Expected Counts Found Dif AbsDif Z_score # F1D # 1 0.301030 1469 0.293800 -0.007230 0.007230 0.779876 # 2 0.176091 873 0.174600 -0.001491 0.001491 0.082461 # ... # View test statistics print(f"Chi-square: {ben.F1D.chi_square:.4f}") print(f"KS statistic: {ben.F1D.KS:.4f}") print(f"MAD: {ben.F1D.MAD:.6f}") print(f"MSE: {ben.F1D.MSE:.8f}") # Get critical values for comparison print(ben.F1D.critical_values) # {'Z': 1.96, 'KS': 0.019206, 'chi2': 15.507, 'MAD': [0.006, 0.012, 0.015]} # Generate report with visualization ben.F1D.report(high_Z='pos', show_plot=True, save_plot='f1d_test.png') # Update confidence level ben.update_confidence(99) # Add summation tests ben.summation() print(ben.F1D_Summ) # First Digit Summation test ``` -------------------------------- ### Perform Mantissa Analysis Source: https://context7.com/milcent/benford_py/llms.txt Analyzes the decimal parts of logarithms, which should be uniformly distributed in Benford-compliant data. ```python import benford as bf import numpy as np # Generate test data np.random.seed(42) data = np.random.exponential(scale=500, size=3000) # Perform mantissa analysis mant = bf.mantissas( data, report=True, # print statistics show_plot=True, # ordered mantissas plot arc_test=True # circular arc test plot ) # Output: # Mantissas Test # # Statistics: # Statistic Expected Reference Found # 0 Mean 0.5000 0.4983 # 1 Variance 0.0833 0.0821 # 2 Skewness 0.0000 0.0127 # 3 Kurtosis -1.2000 -1.1934 # # Kolmogorov-Smirnov statistic: 0.0089 # Critical K-S value at 95% confidence: 0.0248 # Access mantissa statistics print(mant.stats) # {'Mean': 0.4983, 'Var': 0.0821, 'Skew': 0.0127, # 'Kurt': -1.1934, 'KS': 0.0089, 'KS_critical': 0.0248} # Access raw mantissa data print(mant.data.head()) # Mantissa # 0 0.746857 # 1 0.234521 # 2 0.890234 ``` -------------------------------- ### Source.summation() Method Source: https://github.com/milcent/benford_py/blob/master/docs/source/api.md Creates Summation test DataFrames from the Base object. ```APIDOC ## Method summation() ### Description Creates Summation test DataFrames from Base object. ```