### Import libraries and setup path Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/semisupervised.ipynb Sets up the notebook environment, imports required libraries, and adds the project root to the path so that ppi_py and baseline_utils can be imported. ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append( os.path.abspath(os.path.join(os.getcwd(), os.pardir, os.pardir)) ) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_mean_ci, semisupervised_mean_ci, classical_mean_ci from tqdm import tqdm from scipy.optimize import brentq from baseline_utils import * ``` -------------------------------- ### Install ppi-python with pip Source: https://github.com/aangelopoulos/ppi_py/blob/main/docs/source/index.rst Install the ppi_py package using the pip install command. This is the first step to using the library. ```python pip install ppi-python ``` -------------------------------- ### Import libraries and set up path Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/alphafold.ipynb Imports required libraries and sets up the path to include the parent directory for the `utils` module. This is the initial setup for the notebook. ```python import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_mean_ci, classical_mean_ci from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### Setup: imports and autoreload Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/plankton.ipynb Loads autoreload extensions and imports necessary libraries including ppi_py functions and utility functions. ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from scipy.stats import binom, norm from ppi_py.datasets import load_dataset from ppi_py import ppi_distribution_label_shift_ci, classical_mean_ci from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### Install ppi-python with pip Source: https://github.com/aangelopoulos/ppi_py/blob/main/docs/source/README.rst Install the ppi-python package using pip. Run this command in your terminal or command prompt. ```python pip install ppi-python ``` -------------------------------- ### Example signature for ppi_mean_ci Source: https://github.com/aangelopoulos/ppi_py/blob/main/docs/source/README.rst Example signature for mean estimation, showing that X and X_unlabeled are omitted when not needed. ```python ppi_mean_ci(Y, Yhat, Yhat_unlabeled, alpha=0.1) ``` -------------------------------- ### Setup and imports for conformal baseline comparison Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/conformal.ipynb Sets up the notebook environment, imports required libraries (numpy, pandas, ppi_py, tqdm, scipy), and adds the project root to the path so that the baseline_utils module can be imported. ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append( os.path.abspath(os.path.join(os.getcwd(), os.pardir, os.pardir)) ) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_mean_ci, conformal_mean_ci from tqdm import tqdm from scipy.optimize import brentq from baseline_utils import * ``` -------------------------------- ### Import libraries and set up path Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_income.ipynb Imports required libraries and adds the parent directory to the path to access the utils module. Needed before running the census income example. ```python import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_ols_ci, classical_ols_ci from statsmodels.regression.linear_model import OLS from scipy.optimize import brentq from tqdm import tqdm from utils import * ``` -------------------------------- ### Expected output for ppi_mean_ci example Source: https://github.com/aangelopoulos/ppi_py/blob/main/assets/PyPI-README.md Expected console output from the prediction-powered mean estimation example. Produced with numpy=1.26.0; results may differ slightly due to randomness in other environments. ```text theta=0.259, CPP=(0.235677274705698, 0.265952239707548) ``` -------------------------------- ### Setup path and imports for AlphaFold low-N experiment Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/alphafold_lowN.ipynb Adds the project root and parent directory to the module search path, then imports NumPy, pandas, ppi_py functions, tqdm, scipy's brentq, and utilities needed for the experiment. ```python sys.path.append( os.path.abspath(os.path.join(os.getcwd(), os.pardir, os.pardir)) ) sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_mean_ci, classical_mean_ci from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### Progress bar output Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_income.ipynb Example output from the interval construction loop, showing a tqdm progress bar completing 10 iterations in about 14 minutes and 25 seconds. ```text Output: 100%|███████████████████████████████████████████| 10/10 [14:25<00:00, 86.53s/it] ``` -------------------------------- ### Import libraries and setup for census education analysis Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_education.ipynb Imports necessary libraries including ppi_py, sklearn, and utility functions. Sets up the environment by adding the parent directory to the path. ```python import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_poisson_ci, classical_poisson_ci from sklearn.linear_model import PoissonRegressor from sklearn.preprocessing import StandardScaler from scipy.optimize import brentq from tqdm import tqdm from utils import * ``` -------------------------------- ### Import libraries and setup for post-prediction inference Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/postprediction.ipynb Sets up the environment for the post-prediction inference comparison. Imports required libraries and the ppi_py functions for confidence intervals, including postprediction_ols_ci. ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append( os.path.abspath(os.path.join(os.getcwd(), os.pardir, os.pardir)) ) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_ols_ci, classical_ols_ci, postprediction_ols_ci from tqdm import tqdm from scipy.optimize import brentq from statsmodels.regression.linear_model import OLS from baseline_utils import * ``` -------------------------------- ### Setup imports and path for forest_badmodel notebook Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/forest_badmodel.ipynb Sets up the environment for the notebook: enables autoreload, adds parent directories to the path, and imports the required libraries (numpy, pandas, ppi_py, tqdm, scipy, and local utils). ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append( os.path.abspath(os.path.join(os.getcwd(), os.pardir, os.pardir)) ) sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_mean_ci, classical_mean_ci from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### load_dataset Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/galaxies.ipynb Loads a dataset from a specified folder. In this example, it loads the 'galaxies' dataset containing human-annotated labels and predicted labels. ```APIDOC ## `ppi_py.datasets.load_dataset` ### Description Loads a dataset from the specified folder. In this example, it loads the 'galaxies' dataset containing human-annotated labels (`Y`) and predicted labels (`Yhat`). ### Method Function call ### Endpoint `load_dataset(dataset_folder, dataset_name)` ### Parameters - **dataset_folder** (str) - Required - Path to the folder containing the dataset. - **dataset_name** (str) - Required - Name of the dataset to load (e.g., 'galaxies'). ### Returns - **data** (dict) - Dictionary containing the dataset arrays. For 'galaxies', keys include `Y` (human labels) and `Yhat` (predicted labels). ### Example ```python from ppi_py.datasets import load_dataset data = load_dataset("./data/", "galaxies") Y_total = data["Y"] Yhat_total = data["Yhat"] ``` ``` -------------------------------- ### load_dataset Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_income_covshift.ipynb Loads a dataset from the ppi_py package. In this example, it loads the census income dataset. ```APIDOC ## ppi_py.datasets.load_dataset ### Description Loads a dataset from the ppi_py package. In this example, it loads the census income dataset. ### Method Function call ### Endpoint `load_dataset(dataset_folder, dataset_name)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from ppi_py.datasets import load_dataset data = load_dataset("./data/", "census_income") ``` ### Response #### Success Response - **data** (dict) - A dictionary containing the dataset with keys `Y`, `Yhat`, and `X`. #### Response Example ```python { "Y": array([...]), "Yhat": array([...]), "X": array([...]) } ``` ``` -------------------------------- ### Set problem parameters Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_income_covshift.ipynb Sets up the problem parameters: error level `alpha`, coordinate to test, sampling weights for the new population, in-distribution weights, total labeled examples, in-distribution size, range of labeled data sizes (`ns`), and number of trials. ```python alpha = 0.05 coordinate = 0 weights = np.array([0.8, 0.2]) # Women, men in the new population inD_weights = np.array( [1 - Z_total.mean(), Z_total.mean()] ) # Women, men in the old population n_total = Y_total.shape[0] # Total number of labeled examples n_inD = 10000 ns = np.linspace(100, 2000, 10).astype( int ) # Test for different numbers of labeled ballots num_trials = 100 ``` -------------------------------- ### PPBoot and classical bootstrap confidence intervals Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare_ppboot.ipynb This code demonstrates the use of the `ppboot` function from the `ppi_py` library for prediction-powered inference and the `classical_bootstrap_ci` function for classical inference. Both functions compute confidence intervals for the target estimand. **Note**: The full code that calls these functions is not shown in the source text; the snippet below illustrates the imports and setup, but the actual invocation calls are not visible in the provided source. ```APIDOC ## PPBoot and classical bootstrap confidence intervals ### Description This snippet imports the `ppboot` and `classical_bootstrap_ci` functions from the `ppi_py` library. These functions are used to compute prediction-powered and classical bootstrap confidence intervals, respectively. ### Code ```python from ppi_py import ppboot, classical_bootstrap_ci ``` ``` -------------------------------- ### Import ppi_py and dependencies Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/tree_cover_ptd.ipynb Imports required libraries and the ptd module from ppi_py. Adds the parent directory to the Python path to allow importing the package when running from the examples folder. ```python import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd import pickle import matplotlib.pyplot as plt import statsmodels from ppi_py.datasets.datasets import load_dataset from ppi_py import ptd ``` -------------------------------- ### Import libraries and set up environment Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/gene_expression.ipynb Sets up the environment for the notebook: enables autoreload, adds the parent directory to the path, and imports necessary libraries including ppi_py functions for quantile confidence intervals. ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_quantile_ci, classical_quantile_ci from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### Set up environment and imports Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare_ppboot.ipynb Loads the autoreload extension, imports necessary libraries, and sets up the path to include the parent directory for the ppi_py package. ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppboot, classical_bootstrap_ci from sklearn.linear_model import LogisticRegression from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### Compute inverse probability weights for calibration and unlabeled sets Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/tree_cover_ptd.ipynb Computes inverse probability weights for the calibration and unlabeled sets. The probabilities are rescaled to sum to n (the number of calibration samples). A calibration point with sampling probability p gets weight 1/p, while an unlabeled point gets weight 1/(1-p). ```python # to compute sample weights, we need to rescale the probabilities so they sum to n (the number of calibration samples) data['probability'] = n*data['probability'] # sample weights (Inverse Probability Weighting) w = (1/data['probability'])[calibration_indices].to_numpy() w_unlabeled = np.delete(1/(1-data['probability']), calibration_indices, axis=0) # all predicted datapoints except calibration indices ``` -------------------------------- ### Import libraries and set up path Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare.ipynb Imports necessary libraries including ppi_py, sklearn, and utility functions. Adds the parent directory to the system path to import local utilities. ```python import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_logistic_ci, classical_logistic_ci from sklearn.linear_model import LogisticRegression from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### Plot results with make_plots Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/semisupervised.ipynb Generates plots for the semisupervised census income example, saving to a PDF. The output is a matplotlib figure with 2 Axes. ```python make_plots( df, "./plots/semisupervised_census_income.pdf", n_idx=-1, intervals_xlabel="Income", true_theta=true_theta, ) ``` -------------------------------- ### Import libraries and set up environment for PPI power analysis Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/power_analysis.ipynb Sets up the environment for the PPI power analysis notebook: adds the parent directory to the path, imports numpy, matplotlib, the ppi_py dataset loader, the power analysis functions (ppi_power, ppi_mean_power, ppi_ols_power, ppi_logistic_power, ppi_poisson_power), and a utils module. It also enables all user warnings and sets a random seed for reproducibility. ```python import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import matplotlib.pyplot as plt from ppi_py.datasets import load_dataset from ppi_py.ppi_power_analysis import ( ppi_power, ppi_mean_power, ppi_ols_power, ppi_logistic_power, ppi_poisson_power, ) from utils import * import warnings warnings.simplefilter("always", UserWarning) np.random.seed(0) # For reproducibility ``` -------------------------------- ### Import libraries and configure environment Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/galaxies.ipynb Sets up the notebook environment: enables autoreload, adds the parent directory to the Python path, and imports the required libraries (numpy, pandas, ppi_py, tqdm, scipy, and local utils). ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppi_mean_ci, classical_mean_ci from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` -------------------------------- ### Plot intervals and widths with make_plots Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/gene_expression.ipynb Generates the plots for the gene expression example, saving them to ./plots/gene_expression.pdf. The function make_plots is called with the dataframe df, the output path, an x-axis label, and the true quantile value. ```python make_plots( df, "./plots/gene_expression.pdf", intervals_xlabel="Median gene expression", true_theta=true_quantile_q50, ) ``` -------------------------------- ### Find required sample sizes with brentq Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/semisupervised.ipynb Uses brentq to find the smallest n for each method to achieve 80% power. Prints the required sample sizes. The output shows PPI requires 120, semisupervised 208, and classical 201 labeled data points. ```python n_ppi = int(brentq(_to_invert_ppi, 1, 5000, xtol=1)) n_semisupervised = int(brentq(_to_invert_semisupervised, 1, 5000, xtol=1)) n_classical = int(brentq(_to_invert_classical, 1, 5000, xtol=1)) print( f"The PPI test requires n={n_ppi} labeled data points to reject the null." ) print( f"The semisupervised test requires n={n_semisupervised} labeled data points to reject the null." ) print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` -------------------------------- ### Power experiment: find n for 80% power with ppboot and classical bootstrap Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare_ppboot.ipynb Defines two functions that simulate repeated experiments to estimate the rejection rate for a given labeled sample size n, returning the difference from 0.8. The first uses ppboot() to compute a confidence interval for the Pearson correlation, the second uses classical_bootstrap_ci(). Then brentq() finds the smallest n where each method reaches 80% power against the null H0: theta* < 0.15. The printed output shows the required sample sizes: 264 for PPBoot and 314 for the classical test. ```python # Find n such that we reject H0: Corr coeff < 0.15 with probability 80% using a test at level alpha num_experiments = 100 list_rand_idx = [ np.random.permutation(n_total) for i in range(num_experiments) ] def _to_invert_ppi(n): # print(f"PPI: {n}") n = int(n) nulls_rejected = 0 # Data setup for i in range(num_experiments): # print(f"PPI: {n}, {i}") rand_idx = list_rand_idx[i] _X, _X_unlabeled = X_total[rand_idx[:n]], X_total[rand_idx[n:]] _Y, _Y_unlabeled = Y_total[rand_idx[:n]], Y_total[rand_idx[n:]] _Yhat, _Yhat_unlabeled = ( Yhat_total[rand_idx[:n]], Yhat_total[rand_idx[n:]], ) ppi_ci = ppboot( pearson, _Y, _Yhat, _Yhat_unlabeled, _X, _X_unlabeled, alpha=alpha ) if ppi_ci[0] > 0.15: nulls_rejected += 1 return nulls_rejected / num_experiments - 0.8 def _to_invert_classical(n): # print(f"Classical: {n}") n = int(n) nulls_rejected = 0 # Data setup for i in range(num_experiments): rand_idx = list_rand_idx[i] _X, _X_unlabeled = X_total[rand_idx[:n]], X_total[rand_idx[n:]] _Y, _Y_unlabeled = Y_total[rand_idx[:n]], Y_total[rand_idx[n:]] _Yhat, _Yhat_unlabeled = ( Yhat_total[rand_idx[:n]], Yhat_total[rand_idx[n:]], ) classical_ci = classical_bootstrap_ci(pearson, _X, _Y, alpha=alpha) if classical_ci[0] > 0.15: nulls_rejected += 1 return nulls_rejected / num_experiments - 0.8 ``` ```python n_ppi = int(brentq(_to_invert_ppi, 50, 800, xtol=50)) n_classical = int(brentq(_to_invert_classical, 50, 1000, xtol=50)) print( f"The PPBoot test requires n={n_ppi} labeled data points to reject the null." ) print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` ```text Output: The PPBoot test requires n=264 labeled data points to reject the null. The classical test requires n=314 labeled data points to reject the null. ``` -------------------------------- ### Load the census healthcare data set Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare_ppboot.ipynb The `load_dataset` function is used to load the dataset for this example. The dataset contains reported indicators of health insurance (`Y`), predicted indicators of health insurance (`Yhat`), and reported income (`X`). ```APIDOC ## Load the census healthcare data set ### Description This snippet loads the census healthcare dataset using the `load_dataset` function from `ppi_py.datasets`. ### Code ```python dataset_folder = "./data/" data = load_dataset(dataset_folder, "census_healthcare") Y_total = data["Y"] Yhat_total = data["Yhat"] X_total = data["X"][:, 0] # first coordinate is income; second is constant term ``` ``` -------------------------------- ### Set up problem parameters and compute true quantiles Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/gene_expression.ipynb Sets up the problem parameters: error level alpha=0.05, a range of labeled data sizes (ns) from 100 to 2000, and 100 trials. Computes the true quantiles (25%, 50%, 75%) from the full dataset and sets the target quantile q=0.5 for median inference. ```python alpha = 0.05 n_total = Y_total.shape[0] # Total number of labeled examples ns = np.linspace(100, 2000, 5).astype( int ) # Test for different numbers of labeled examples num_trials = 100 # True quantiles (computed on all data) true_quantile_q25 = np.quantile(Y_total, 0.25) true_quantile_q50 = np.quantile(Y_total, 0.5) true_quantile_q75 = np.quantile(Y_total, 0.75) q = 0.5 # Target problem is inference on the median ``` -------------------------------- ### Compute cheapest pair with n_max constraint (achievable effective_n) Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/power_analysis.ipynb Computes the cheapest pair for Poisson regression with n_max=6000 and effective_n=5950, reusing initial data and costs from the previous example. The result shows n=5913, N=87, cost=5973, effective_n=5950. ```python n_max = 6000 effective_n = 5950 coord = 0 achieveable_cheapest_pair = ppi_poisson_power( X_initial, Y_initial, Yhat_initial, cost_X, cost_Y, cost_Yhat, coord, effective_n=effective_n, n_max=n_max, ) achieveable_cheapest_pair ``` ```text Result: {'n': 5913, 'N': 87, 'cost': 5973.0, 'effective_n': 5950, 'ppi_corr': 0.6555119650114937} ``` -------------------------------- ### Sample calibration set and build X, Xhat, Y, Yhat arrays Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/tree_cover_ptd.ipynb Randomly samples 500 calibration points from the 67968 total points and constructs the calibration and unlabeled datasets. Adds a constant column to covariate arrays so the regression coefficients include an intercept term. ```python np.random.seed(seed=100) calibration_indices = np.random.choice(np.arange(0, len(data)), size=500, replace=False) X = statsmodels.tools.add_constant(truth_X[calibration_indices]) Xhat = statsmodels.tools.add_constant(preds_X[calibration_indices]) Xhat_unlabeled = statsmodels.tools.add_constant(np.delete(preds_X, calibration_indices, axis=0)) # all predicted datapoints except calibration indices Y = truth_Y[calibration_indices] Yhat = preds_Y[calibration_indices] Yhat_unlabeled = np.delete(preds_Y, calibration_indices, axis=0) # all predicted datapoints except calibration indices ``` -------------------------------- ### Sample calibration set and prepare data Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/tree_cover_ptd.ipynb Randomly samples 1000 points for the calibration set and constructs the design matrices and response vectors for calibration and unlabeled sets, adding a constant term for the intercept. ```python np.random.seed(seed=100) calibration_indices = np.random.choice(np.arange(0, len(data)), size=1000, replace=False) X = statsmodels.tools.add_constant(truth_X[calibration_indices]) Xhat = statsmodels.tools.add_constant(preds_X[calibration_indices]) Xhat_unlabeled = statsmodels.tools.add_constant(np.delete(preds_X, calibration_indices, axis=0)) # all predicted datapoints except calibration indices Y = truth_Y[calibration_indices] Yhat = preds_Y[calibration_indices] Yhat_unlabeled = np.delete(preds_Y, calibration_indices, axis=0) # all predicted datapoints except calibration indices ``` -------------------------------- ### Set up problem parameters and compute ground truth Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare.ipynb Specifies error level, coordinate, total labeled examples, range of labeled set sizes, number of trials, and optimizer options. Computes the ground-truth logistic regression coefficient using the full dataset. ```python alpha = 0.05 coordinate = 0 # Choose between 0, 1 n_total = Y_total.shape[0] # Total number of labeled examples ns = np.array([500, 1000, 2000]).astype( int ) # Test for different numbers of labeled ballots num_trials = 100 optimizer_options = { "ftol": 1e-5, "gtol": 1e-5, "maxls": 10000, "maxiter": 10000, } # Compute ground truth true_theta = ( LogisticRegression( penalty="none", solver="lbfgs", max_iter=10000, tol=1e-15, fit_intercept=False, ) .fit(X_total, Y_total) .coef_.squeeze()[coordinate] ) ``` -------------------------------- ### Specify parameter setup and compute ground-truth estimand Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare_ppboot.ipynb This snippet sets the error level `alpha`, the range of labeled data sizes `ns`, and the number of trials `num_trials`. It also defines the Pearson correlation coefficient function and computes the ground-truth value of the estimand. ```APIDOC ## Specify parameter setup and compute ground-truth estimand ### Description This snippet defines the parameters for the experiment and computes the ground-truth value of the Pearson correlation coefficient. ### Code ```python alpha = 0.1 n_total = Y_total.shape[0] # Total number of labeled examples ns = np.array([500, 750, 1000, 1500, 2000]).astype(int) # Test for different numbers of labeled ballots num_trials = 100 # define Pearson correlation coefficient def pearson(X, Y): return np.corrcoef(X, Y)[0, 1] # Compute ground truth true_theta = pearson(X_total, Y_total) ``` ``` -------------------------------- ### Compute required sample sizes for power experiment Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/ballots.ipynb Computes the required sample size for the classical method using brentq, and prints the results. The PPI method rejects for any n, so it is hardcoded to 1. Output shows PPI requires n=1 and classical requires n=112. ```python # n_ppi = int(brentq(_to_invert_ppi, 1, 1000, xtol=1)) # This rejects for any value n. Try it! n_classical = int(brentq(_to_invert_classical, 1, 1000, xtol=1)) print(f"The PPI test requires n={1} labeled data points to reject the null.") print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` -------------------------------- ### Set problem parameters and compute true mean Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/forest_badmodel.ipynb Sets the error level (alpha=0.05), the total number of labeled examples, the range of labeled set sizes (ns from 50 to 500), and the number of trials (100). Computes the true mean of the labels as the ground-truth estimand. ```python alpha = 0.05 n_total = Y_total.shape[0] # Total number of labeled examples ns = np.linspace(50, 500, 10).astype( int ) # Test for different numbers of labeled examples num_trials = 100 # True mean (computed on all labels) true_theta = Y_total.mean() ``` -------------------------------- ### Plot PPI intervals and widths for plankton example Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/plankton.ipynb Plots five randomly chosen intervals from the dataframe for PPI and the imputed interval, along with the average interval width and a scatterplot of widths from the random draws. The plot_classical=False argument omits the classical baseline due to label shift. The output is a matplotlib figure of size 648x216 with 3 Axes. ```python make_plots( df, "./plots/plankton.pdf", n_idx=4, intervals_xlabel="Number of plankton", plot_classical=False, true_theta=true_theta, ) ``` ```text Result:
``` -------------------------------- ### Find cheapest pair and simulate power Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/power_analysis.ipynb Uses ppi_poisson_power with an effective_n to find the cheapest pair (n, N) achieving the desired power, then simulates 100 repetitions to verify the empirical power. The output shows an empirical power of 0.94 against a desired power of 0.9. ```python cheapest_pair = ppi_poisson_power( X_initial, Y_initial, Yhat_initial, cost_X, cost_Y, cost_Yhat, coord, effective_n=effective_n, ) n_star = cheapest_pair["n"] N_star = cheapest_pair["N"] rejections = 0 reps = 100 for _ in range(reps): sample = np.random.choice(n_total, n_star + N_star, replace=False) Y_sample = Y_total[sample[:n_star]] Yhat_sample = Yhat_total[sample[:n_star]] X_sample = X_total[sample[:n_star]] XN_sample = X_total[sample[n_star:]] YhatN_sample = Yhat_total[sample[n_star:]] lower, upper = ppi_poisson_ci( X_sample, Y_sample, Yhat_sample, XN_sample, YhatN_sample, alpha=level, coord=coord, ) if lower[coord] > 0 or upper[coord] < 0: rejections += 1 print("Desired power: ", power) print("Empirical power: ", rejections / reps) ``` ```text Output: Desired power: 0.9 Empirical power: 0.94 ``` -------------------------------- ### Compute effective sample size from effect size Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/power_analysis.ipynb Computes the effective sample size needed to achieve a desired power given an effect size and standard error. The function uses scipy's brentq to solve for the noncentrality parameter and returns the rounded effective sample size. The example call with delta=0.1, sigma=1, power=0.9, level=0.05 yields 1051. ```python from scipy.optimize import brentq from scipy.stats import norm, foldnorm def effective_n_from_effect_size(delta, sigma, power, level=0.05): assert 0 < power < 1, "Power must be between 0 and 1" assert 0 < level < 1, "Level must be between 0 and 1" assert 0 != delta, "Effect size must be non-zero" assert power > level, "Power must be greater than level" q = norm.ppf(1 - level / 2) beta = 1 - power def f(mu): return foldnorm.cdf(q, c=mu) - beta a = 0 b = q - norm.ppf(beta) mu_star = brentq(f, a, b) d = delta / sigma effective_n = (mu_star / d) ** 2 return round(effective_n) delta = 0.1 sigma = 1 power = 0.9 level = 0.05 effective_n = effective_n_from_effect_size(delta, sigma, power, level) effective_n ``` ```text Result: 1051 ``` -------------------------------- ### Compute required sample sizes using brentq Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/alphafold.ipynb Uses scipy's brentq to find the smallest n for which the power inversion functions cross zero, yielding the required sample sizes for PPI and classical tests. Prints the results. ```python n_ppi = int(brentq(_to_invert_ppi, 10, 3000, xtol=1)) n_classical = int(brentq(_to_invert_classical, 10, 3000, xtol=1)) print( f"The PPI test requires n={n_ppi} labeled data points to reject the null." ) print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` -------------------------------- ### Import necessary libraries and modules Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare_ppboot.ipynb This code snippet imports the required libraries and the ppi_py functions for the prediction-powered inference (PPBoot) and classical bootstrap confidence interval. ```APIDOC ## Import necessary libraries and modules ### Description This snippet loads the required Python libraries and the `ppi_py` modules for this analysis. ### Code ```python %load_ext autoreload %autoreload 2 import os, sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) import numpy as np import pandas as pd from ppi_py.datasets import load_dataset from ppi_py import ppboot, classical_bootstrap_ci from sklearn.linear_model import LogisticRegression from tqdm import tqdm from scipy.optimize import brentq from utils import * ``` ``` -------------------------------- ### Find required sample size for 80% power using brentq and ppi_ols_ci Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_income_covshift.ipynb Defines two functions that simulate repeated experiments to estimate the rejection rate for a one-sided test on the OLS coefficient (null: coefficient < 800) at a given sample size n, returning the difference from the target 0.8 power. The PPI version uses ppi_ols_ci with importance weights, while the classical version sets lam=0. Then brentq finds the smallest n for each method, and the printed output shows the required sample sizes: 225 for PPI and 318 for classical. ```python # Find n such that we reject H0: OLS coeff < 800 with probability 80% using a test at level alpha num_experiments = 100 list_rand_idx = [np.random.permutation(n_inD) for i in range(num_experiments)] def _to_invert_ppi(n): n = int(n) nulls_rejected = 0 # Data setup for i in range(num_experiments): rand_idx = list_rand_idx[i] _X = X_inD[rand_idx[:n]] _Y = Y_inD[rand_idx[:n]] _Z = Z_inD[rand_idx[:n]] _Yhat = Yhat_inD[rand_idx[:n]] importance_weights = np.array( [ ( weights[0] / inD_weights[0] if z == 0 else weights[1] / inD_weights[1] ) for z in _Z ] ) ppi_ci = ppi_ols_ci( _X, _Y, _Yhat, X_ooD, Yhat_ooD, w=importance_weights, alpha=alpha ) if ppi_ci[0][coordinate] > 800: nulls_rejected += 1 return nulls_rejected / num_experiments - 0.8 def _to_invert_classical(n): n = int(n) nulls_rejected = 0 # Data setup for i in range(num_experiments): rand_idx = list_rand_idx[i] _X = X_inD[rand_idx[:n]] _Y = Y_inD[rand_idx[:n]] _Z = Z_inD[rand_idx[:n]] _Yhat = Yhat_inD[rand_idx[:n]] importance_weights = np.array( [ ( weights[0] / inD_weights[0] if z == 0 else weights[1] / inD_weights[1] ) for z in _Z ] ) classical_ci = ppi_ols_ci( _X, _Y, _Yhat, X_ooD, Yhat_ooD, w=importance_weights, alpha=alpha, lam=0, ) if classical_ci[0][coordinate] > 800: nulls_rejected += 1 return nulls_rejected / num_experiments - 0.8 ``` ```python n_ppi = int(brentq(_to_invert_ppi, 100, 2000, xtol=1)) n_classical = int(brentq(_to_invert_classical, 100, 10000, xtol=1)) print( f"The PPI test requires n={n_ppi} labeled data points to reject the null." ) print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` ```text Output: The PPI test requires n=225 labeled data points to reject the null. The classical test requires n=318 labeled data points to reject the null. ``` -------------------------------- ### Construct confidence intervals for all methods Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/semisupervised.ipynb Runs the main experiment: for each labeled data size and trial, computes confidence intervals using PPI, the semisupervised baseline, and classical inference, then aggregates results into a dataframe with a 'width' column. ```python # Run prediction-powered inference and classical inference for many values of n results = [] for i in tqdm(range(ns.shape[0])): for j in range(num_trials): # Prediction-Powered Inference n = ns[i] rand_idx = np.random.permutation(n_total) _Yhat = Yhat_total[rand_idx[:n]] _Y = Y_total[rand_idx[:n]] _X = X_total[rand_idx[:n]] _X_unlabeled = X_total[rand_idx[n:]] _Yhat_unlabeled = Yhat_total[rand_idx[n:]] # PPI interval ppi_ci = ppi_mean_ci(_Y, _Yhat, _Yhat_unlabeled, alpha=alpha) # Semisupervised interval semisupervised_ci = semisupervised_mean_ci( _X, _Y, _X_unlabeled, K, alpha=alpha ) # Classical interval classical_ci = classical_mean_ci(_Y, alpha=alpha) # Append results results += [ pd.DataFrame( [ { "method": "PPI", "n": n, "lower": ppi_ci[0], "upper": ppi_ci[1], "trial": j, } ] ) ] results += [ pd.DataFrame( [ { "method": "Baseline", # Semisupervised baseline "n": n, "lower": semisupervised_ci[0], "upper": semisupervised_ci[1], "trial": j, } ] ) ] results += [ pd.DataFrame( [ { "method": "Classical", "n": n, "lower": classical_ci[0], "upper": classical_ci[1], "trial": j, } ] ) ] df = pd.concat(results, axis=0, ignore_index=True) df["width"] = df["upper"] - df["lower"] ``` -------------------------------- ### Find n using brentq for PPI and classical tests Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_healthcare.ipynb Uses scipy.optimize.brentq to find the smallest integer n for which the rejection rate reaches 80% by inverting the power functions. Prints the resulting n values. ```python n_ppi = int(brentq(_to_invert_ppi, 150, 15000, xtol=100)) n_classical = int(brentq(_to_invert_classical, 150, 20000, xtol=100)) print( f"The PPI test requires n={n_ppi} labeled data points to reject the null." ) print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` -------------------------------- ### Sample calibration set and prepare data for weighted linear regression Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/tree_cover_ptd.ipynb Samples a calibration set of size n=1000 from the data using the given probabilities, then constructs the design matrices and response vectors for the calibration and unlabeled sets. The unlabeled sets exclude the calibration indices. ```python # sample calibration set using the given probabilities n = 1000 calibration_indices = np.random.choice(np.arange(0, len(data)), size=n, replace=False, p=data['probability']) X = statsmodels.tools.add_constant(truth_X[calibration_indices]) Xhat = statsmodels.tools.add_constant(preds_X[calibration_indices]) Xhat_unlabeled = statsmodels.tools.add_constant(np.delete(preds_X, calibration_indices, axis=0)) # all predicted datapoints except calibration indices Y = truth_Y[calibration_indices] Yhat = preds_Y[calibration_indices] Yhat_unlabeled = np.delete(preds_Y, calibration_indices, axis=0) # all predicted datapoints except calibration indices ``` -------------------------------- ### Find required sample sizes using brentq Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/baselines/forest_badmodel.ipynb Uses scipy's brentq to find the smallest n for which the power functions cross zero, then prints the required labeled data points for PPI and classical tests. ```python n_ppi = int(brentq(_to_invert_ppi, 5, 500, xtol=1)) n_classical = int(brentq(_to_invert_classical, 10, 500, xtol=1)) print( f"The PPI test requires n={n_ppi} labeled data points to reject the null." ) print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` -------------------------------- ### Power Experiment: Find Minimum n Using Brent's Method Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/galaxies.ipynb Uses scipy's brentq to find the smallest n for which each method achieves 80% power against the null hypothesis. Prints the required labeled data sizes for PPI and classical tests. ```python n_ppi = int(brentq(_to_invert_ppi, 1, 1000, xtol=1)) n_classical = int(brentq(_to_invert_classical, 1, 1000, xtol=1)) print( f"The PPI test requires n={n_ppi} labeled data points to reject the null." ) print( f"The classical test requires n={n_classical} labeled data points to reject the null." ) ``` ```text Output: The PPI test requires n=131 labeled data points to reject the null. The classical test requires n=359 labeled data points to reject the null. ``` -------------------------------- ### Create inD and ooD datasets Source: https://github.com/aangelopoulos/ppi_py/blob/main/examples/census_income_covshift.ipynb Creates the in-distribution and out-of-distribution datasets by indexing the total arrays with the previously generated indices. ```python # Generate inD and ooD datasets X_inD, X_ooD = X_total[inD_idxs], X_total[shifted_idxs] Y_inD, Y_ooD = Y_total[inD_idxs], Y_total[shifted_idxs] Yhat_inD, Yhat_ooD = Yhat_total[inD_idxs], Yhat_total[shifted_idxs] Z_inD, Z_ooD = Z_total[inD_idxs], Z_total[shifted_idxs] ```