### Install crepes using pip

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Install the crepes library from PyPI using pip.

```bash
pip install crepes
```

--------------------------------

### Install crepes

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/Tutorial 2026-06-02.ipynb

Install crepes using pip or conda. This is the first step before using the library.

```bash
pip install crepes
```

```bash
conda install conda-forge::crepes
```

--------------------------------

### Install crepes using conda

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Install the crepes library from conda-forge using conda.

```bash
conda install conda-forge::crepes
```

--------------------------------

### Example evaluation results

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/getting_started.md

Example output of the evaluate method, showing various performance metrics like error rate, average coverage, etc.

```python
{'error': 0.007575757575757569,
 'avg_c': 1.6325757575757576,
 'one_c': 0.36742424242424243,
 'empty': 0.0,
 'ks_test': 0.0033578466103315894,
 'time_fit': 1.9073486328125e-06,
 'time_evaluate': 0.04798746109008789}
```

--------------------------------

### Online Calibration CPD Size Plot (Normalized, No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Plots the calibration set size during online prediction for normalized systems when warm_start is disabled. Highlights that the calibration set size starts at zero.

```python
cpds = cps_norm.predict_cpds(X_test, y_test, online=True, warm_start=False)
plt.plot(np.arange(len(cpds)), [len(cpds[i]) for i in range(len(cpds))])
plt.xlabel("#test objects")
plt.ylabel("calibration set size")
plt.ylim(0)
plt.xlim(0)
plt.show()
```

--------------------------------

### Numpy Array Example

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example of a NumPy array output, often seen in machine learning contexts.

```numpy
array([[8.13837566e-05, 8.86436603e-01],
       [6.60518590e-02, 4.02350293e-01],
       [4.28646783e-01, 4.29930890e-02],
       ...,
       [7.05118942e-01, 9.45056960e-03],
       [7.27003479e-01, 1.27347189e-02],
       [1.76403756e-01, 1.21434924e-01]])
```

--------------------------------

### Online prediction set generation with custom confidence and warm start

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Generates prediction sets online, allowing for a custom confidence level and disabling warm start. Setting `warm_start=False` ensures that the original calibration set is ignored, starting fresh with new data.

```python
cc_mond.predict_set_online(alphas_test, rf.classes_, y_test, bins_test, 
                           confidence=0.99, warm_start=False)
```

--------------------------------

### Apply Step Betting Function with Drift Dynamically

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/martingales_nb.ipynb

Demonstrates creating a StepBettingFunction instance and then applying it with different start indices ('s') directly in the apply method. This allows for flexible drift parameterization.

```python
bf = StepBettingFunction(Gs = Gs, A = a)

rewards_s_100 = bf.apply(p_values, s=100)
rewards_s_200 = bf.apply(p_values, s=200)
rewards_s_300 = bf.apply(p_values, s=300)
```

--------------------------------

### Evaluate WrapRegressor with Warm Start Disabled

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Evaluates a WrapRegressor with online calibration and warm start disabled. This forces the model to retrain on the full calibration set each time. Note the different metrics and timing compared to the default.

```python
cps_mond_norm.evaluate(X_test, y_test, confidence=0.99, y_min=0, y_max=1, 
                       online=True, warm_start=False)
```

--------------------------------

### Evaluate Normalized CPS Online (No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Evaluates a normalized CPS online with warm start disabled, considering test set variances. This demonstrates performance when the original calibration set is ignored.

```python
cps_norm.evaluate(y_hat_test, y_test, 
                  sigmas=sigmas_test_var, 
                  confidence=0.99, y_min=0, y_max=1, 
                  online=True, warm_start=False)
```

--------------------------------

### Evaluate Standard CPS Online (No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Evaluates a standard CPS online, explicitly disabling the warm start option. This affects how the calibration set is handled.

```python
cps_std.evaluate(y_hat_test, y_test, 
                 confidence=0.99, y_min=0, y_max=1, 
                 online=True, warm_start=False)
```

--------------------------------

### Example NumPy Array Output

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/getting_started.md

This snippet shows an example of a NumPy array output, likely representing data or results from a previous operation.

```numpy
array([[0.01179923, 0.91376964],
       [0.01179923, 0.91376964],
       [0.06997053, 0.39823081],
       [0.43432425, 0.0446698 ],
       ...,
       [0.69912443, 0.01199548],
       [0.70007216, 0.01196775],
       [0.17997245, 0.11987124]])
```

--------------------------------

### Evaluate Standard Conformal Regressor (Online, Warm Start False)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Evaluates a standard conformal regressor with online calibration, disabling warm start to exclude the initial calibration set. This affects the evaluation results.

```python
cr_std.evaluate(y_hat_test, y_test, confidence=0.99, y_min=0, y_max=1, 
                online=True, warm_start=False)
```

--------------------------------

### Online Prediction Intervals (Normalized, No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Generates prediction intervals using online calibration for normalized systems, without warm-starting. Useful for understanding interval behavior when the calibration set is not reused.

```python
cps_norm.predict_int(X_test, y_test, online=True, warm_start=False)
```

--------------------------------

### Evaluate Mondrian Normalized CPS Online (No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Evaluates a Mondrian normalized CPS online with warm start disabled, incorporating variances and bins. This provides performance metrics when the calibration set is reset.

```python
cps_mond_norm.evaluate(y_hat_test, y_test,
                       sigmas=sigmas_test_var,
                       bins=bins_test, 
                       confidence=0.99, y_min=0, y_max=1, 
                       online=True, warm_start=False)
```

--------------------------------

### Online Prediction Intervals (Standard, No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Generates prediction intervals using online calibration without warm-starting, meaning the original calibration set is ignored. Initial intervals may be very wide.

```python
cps_std.predict_int(X_test, y_test, online=True, warm_start=False)
```

--------------------------------

### Predict prediction intervals with Standard Conformal Regressor (Online, Warm Start False)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Calculates prediction intervals with online calibration, explicitly disabling warm start to ignore the original calibration set. This can lead to wider initial intervals.

```python
cr_std.predict_int_online(y_hat_test, y_test, y_min=0, y_max=1, confidence=0.9,
                      warm_start=False)
```

--------------------------------

### Online Prediction Intervals (Mondrian Normalized, No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Computes prediction intervals for Mondrian normalized systems with online calibration and warm_start=False. Shows how intervals are formed without leveraging previous calibration data.

```python
cps_mond_norm.predict_int(X_test, y_test, online=True, warm_start=False)
```

--------------------------------

### Import CREPES and Dependencies

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/martingales_nb.ipynb

Imports necessary libraries like numpy, pandas, matplotlib, scikit-learn, rdkit, and specific modules from crepes for martingales and wrappers. Prints the installed crepes version.

```python
import numpy as np
import pandas as pd

np.set_printoptions(legacy='1.25')

import matplotlib.pyplot as plt
from matplotlib.colors import TABLEAU_COLORS as colors
colors = list(colors.keys())

from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.datasets import fetch_openml

import rdkit # Used in the Lipophilicity example; install with "%pip install rdkit"
from rdkit.Chem import AllChem

from crepes import WrapRegressor, __version__

from crepes.martingales import (
    EpsilonBettingFunction,
    StepBettingFunction,
    semi_online_p_values,
    SimpleJumper, 
    SleeperStayer, 
    SleeperDrifter, 
    CompositeMartingale
)

print(f"crepes v. {__version__}")
```

--------------------------------

### Online Prediction Intervals (Mondrian Standard, No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Calculates prediction intervals for Mondrian standard systems with online calibration and warm_start=False. Demonstrates interval generation when the calibration set is reset for each prediction.

```python
cps_mond_std.predict_int(X_test, y_test, online=True, warm_start=False)
```

--------------------------------

### Setting Difficulty Beta Parameter

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/Tutorial 2026-06-02.ipynb

Sets the beta parameter, which influences the difficulty estimation. This is a common setup step for difficulty estimators.

```python
beta = 0.01
```

--------------------------------

### Predict Intervals with Warm Start Disabled

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Generates prediction intervals with online calibration and `warm_start=False`, disabling the use of the original calibration set. This can lead to wider initial intervals.

```python
rf.predict_int(X_test, y_test, confidence=0.99, y_min=0, y_max=1, 
               online=True, warm_start=False)
```

--------------------------------

### Import Libraries and Print Version

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/Tutorial 2026-06-02.ipynb

Imports necessary libraries like numpy, pandas, matplotlib, and scikit-learn. It also prints the installed version of crepes.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sklearn # may be installed by: pip install scikit-learn

from crepes import __version__

print(f"crepes v. {__version__}")

np.random.seed(602211023)
np.set_printoptions(legacy='1.25')
```

--------------------------------

### Initialize SleeperDrifter with Custom Settings

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Initialize SleeperDrifter with a specific investment rate (J), wake-up frequency (M), and a custom betting function (EpsilonBettingFunction). Note: The example in the source uses SleeperStayer constructor name.

```python
m = SleeperStayer(J = 0.1, M = 50, bf = EpsilonBettingFunction())
```

--------------------------------

### Online Percentile Prediction (Standard, No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Requests specific percentiles for predictions using online calibration and warm_start=False. Allows for customized interval bounds by specifying lower and higher percentiles.

```python
cps_std.predict_percentiles(X_test, y_test, 
                            lower_percentiles = [2.5, 25],
                            higher_percentiles = [50, 75, 97.5],
                            y_min=0, y_max=1, 
                            online=True, warm_start=False)
```

--------------------------------

### Evaluate rf_norm_knn_dist with Warm Start Disabled

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Evaluates the `rf_norm_knn_dist` conformal regressor with online calibration and `warm_start=False`. This snippet shows the evaluation metrics when the initial calibration set is ignored.

```python
rf_norm_knn_dist.evaluate(X_test, y_test, confidence=0.99, y_min=0, y_max=1, 
                          online=True, warm_start=False)
```

--------------------------------

### Load and Prepare House Sales Dataset

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/Tutorial 2026-06-02.ipynb

Loads the house sales dataset from OpenML and prepares it for training, calibration, and testing.

```python
from crepes.martingales import SleeperDrifter

martingales_sleeper_drifter = {
    "Sleeper/Drifter (standard) " : SleeperDrifter().apply(p_values),
    "Sleeper/Drifter (epsilon)" : SleeperDrifter(bf = EpsilonBettingFunction()).apply(p_values),
}

plot_martingales(martingales_sleeper_drifter, no_alphas_1)
```

```python
from sklearn.ensemble import RandomForestRegressor
from crepes import WrapRegressor
from sklearn.datasets import fetch_openml

dataset = fetch_openml(name="house_sales", version=3, parser="auto")

sorted_index = dataset.data.sort_values(["date_year", "date_month", "date_day"]).index

X = dataset.data.loc[sorted_index].drop(["date_year", "date_month", "date_day"], axis="columns").values.astype(float)

y = dataset.target.loc[sorted_index].values.astype(float)

cal_start = 14000
test_start = 15000

X_train = X[:cal_start]
y_train = y[:cal_start]

X_cal = X[cal_start:test_start]
y_cal = y[cal_start:test_start]

X_test = X[test_start:]
y_test = y[test_start:]
```

--------------------------------

### Evaluate Mondrian Standard CPS Online (No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Evaluates a Mondrian standard CPS online, disabling warm start. This shows the system's performance when starting with an empty calibration set for each category.

```python
cps_mond_std.evaluate(y_hat_test, y_test, 
                      bins=bins_test, 
                      confidence=0.99, y_min=0, y_max=1, 
                      online=True, warm_start=False)
```

--------------------------------

### Online Calibration CPD Size Plot (Mondrian Normalized, No Warm Start)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Visualizes the calibration set size for Mondrian normalized systems during online prediction with warm_start disabled. Shows the initial zero size of the calibration set.

```python
cpds = cps_mond_norm.predict_cpds(X_test, y_test, online=True, warm_start=False)
plt.plot(np.arange(len(cpds)), [len(cpds[i]) for i in range(len(cpds))])
plt.xlabel("#test objects")
plt.ylabel("calibration set size")
plt.ylim(0)
plt.xlim(0)
plt.show()
```

--------------------------------

### Initialize SimpleJumper with default settings

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Create a SimpleJumper instance with the default jumping rate (J=0.01) and the default EpsilonBettingFunction.

```python
m = SimpleJumper()
```

--------------------------------

### Initialize SleeperDrifter with Default Settings

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Obtain an instance of SleeperDrifter with its default investment rate, wake-up frequency, and betting function.

```python
m = SleeperDrifter()
```

--------------------------------

### Initialize StepBettingFunction with default values

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Instantiate a StepBettingFunction using its default parameters for grid size and thresholds.

```python
bf = StepBettingFunction()
```

--------------------------------

### Step Betting Function with Drift Visualization

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/martingales_nb.ipynb

Compares the rewards of a standard StepBettingFunction with versions incorporating drift (using different start indices 's'). This highlights how drift affects rewards over a sequence of p-values.

```python
p = 0.05

n = 500

a = 0.1
b = 0.8

Gs = [b]

p_values = np.full(n, p)

rewards_s_100 = StepBettingFunction(Gs = Gs, A = a, s=100).apply(p_values)
rewards_s_200 = StepBettingFunction(Gs = Gs, A = a, s=200).apply(p_values)
rewards_s_300 = StepBettingFunction(Gs = Gs, A = a, s=300).apply(p_values)

rewards_c = StepBettingFunction(Gs = Gs, A = a).apply(p_values)


plt.plot(np.arange(n), rewards_c, label="no drift")

plt.plot(np.arange(n), rewards_s_100, label="s = 100")
plt.plot(np.arange(n), rewards_s_200, label="s = 200")
plt.plot(np.arange(n), rewards_s_300, label="s = 300")

plt.ylim(0,8.5)
plt.legend()
plt.show()
```

--------------------------------

### Initialize and Fit DifficultyEstimator for KNN Distances (OOB)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initialize a DifficultyEstimator for k-nearest neighbors using distances only, with OOB enabled for difficulty estimation. Fit it to the training data.

```python
de_knn_dist_oob = DifficultyEstimator()

de_knn_dist_oob.fit(X=X_train, scaler=True, oob=True)

display(de_knn_dist_oob)
```

--------------------------------

### Get Prediction Thresholds and p-values with predict_cps

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Use `predict_cps` to get p-values and threshold values for specified lower and higher percentiles. This provides a more detailed prediction interval.

```python
results = cps_mond_norm.predict_cps(X_test, y=y_test,
                                    lower_percentiles=[2.5, 5],
                                    higher_percentiles=[95, 97.5])

display(results)
```

--------------------------------

### Initialize StepBettingFunction with custom thresholds

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Create a StepBettingFunction specifying custom threshold values (A). If A is not provided, it defaults to values derived from G.

```python
bf = EpsilonBettingFunction(A = [0.2, 0.4, 0.6])
```

--------------------------------

### Initialize and Fit DifficultyEstimator for KNN Labels (OOB)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initialize a DifficultyEstimator for k-nearest neighbors using labels, with OOB enabled. Fit it to the training data including labels.

```python
de_knn_std_oob = DifficultyEstimator()

de_knn_std_oob.fit(X=X_train, y=y_train, scaler=True, oob=True)

display(de_knn_std_oob)
```

--------------------------------

### Initialize and Fit DifficultyEstimator for KNN Residuals (OOB)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initialize a DifficultyEstimator for k-nearest neighbors using OOB residuals, with OOB enabled. Fit it to the training data using the calculated residuals.

```python
de_knn_res_oob = DifficultyEstimator()

de_knn_res_oob.fit(X=X_train, residuals=residuals_oob, scaler=True, oob=True)

display(de_knn_res_oob)
```

--------------------------------

### Initialize EpsilonBettingFunction with default values

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Obtain an EpsilonBettingFunction instance using its default epsilon values.

```python
bf = EpsilonBettingFunction()
```

--------------------------------

### Predict prediction intervals with Normalized Variance Conformal Regressor (Online, Warm Start False)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Generates prediction intervals using online calibration, with warm start disabled to exclude the initial calibration set. This affects the initial interval width.

```python
cr_norm_var.predict_int_online(y_hat_test, y_test, sigmas=sigmas_test_var, 
                               y_min=0, y_max=1, confidence=0.9, warm_start=False)
```

--------------------------------

### Loading and Preprocessing House Sales Dataset

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/martingales_nb.ipynb

Loads the house sales dataset from OpenML, sorts it by date, removes date-related features, and splits it into training and testing sets. This prepares the data for subsequent martingale analysis.

```python
np.random.seed(2025)

dataset = fetch_openml(name="house_sales", version=3, parser="auto")

sorted_index = dataset.data.sort_values(["date_year", 
                                         "date_month", 
                                         "date_day"]).index

# Uncomment the following if you want to investigate what happens when 
# the dataset is randomly shuffled

#sorted_index = np.random.choice(np.arange(len(dataset.data)), 
#                                len(dataset.data), 
#                                replace=False)

X = dataset.data.loc[sorted_index].drop(["date_year", 
                                         "date_month", 
                                         "date_day"], 
                                        axis="columns").values.astype(float)

y = dataset.target.loc[sorted_index].values.astype(float)

test_start = 10000

X_train = X[:test_start]
y_train = y[:test_start]
X_test = X[test_start:]
y_test = y[test_start:]

print(f"X_train: {X_train.shape}")
print(f"y_train: {y_train.shape}")
print(f"X_test: {X_test.shape}")
print(f"y_test: {y_test.shape}")
```

--------------------------------

### Predict prediction intervals with Mondrian Conformal Regressor (Online, Warm Start False)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Obtains prediction intervals from a Mondrian conformal regressor with online calibration, disabling warm start to ignore the original calibration set. Initial intervals may be wider.

```python
cr_mond.predict_int_online(y_hat_test, y_test, bins=bins_test, y_min=0, y_max=1, 
                           confidence=0.9, warm_start=False)
```

--------------------------------

### Import and Split Dataset

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/Tutorial 2026-06-02.ipynb

Fetches the 'qsar-biodeg' dataset from OpenML and splits it into training, testing, and calibration sets. This prepares the data for model training and evaluation.

```python
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

dataset = fetch_openml(name="qsar-biodeg", version=1, parser="auto")

X = dataset.data.values.astype(float)

target = dataset.target.cat.rename_categories({"1":"RB", "2":"NRB"})
y = target.values

X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=1/4)

X_prop_train, X_cal, y_prop_train, y_cal = \
    train_test_split(X_train, y_train, test_size=1/3)
```

--------------------------------

### NumPy Array of Predicted Percentiles

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output format for predicted percentiles.

```numpy
array([[3120432.14791764, 3403976.16608241, 3952384.13595105],
       [ 930191.36994287,  979804.59585495, 1075762.49571536],
       [ 236278.82580469,  253387.66592079,  329933.49293406],
       ...,
       [1336110.21956702, 1477739.04927264, 1751666.10820498],
       [ 298621.13482031,  317029.35735016,  399388.68783836],
       [ 564574.75363948,  615226.06727944,  762212.9912238 ]])
```

--------------------------------

### Create and Fit WrapRegressor

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initializes a WrapRegressor with a RandomForestRegressor and fits it to the proper training data. The wrapped learner can be fitted before or after wrapping.

```python
rf = WrapRegressor(RandomForestRegressor(n_jobs=-1, n_estimators=500, oob_score=True))

rf.fit(X_prop_train, y_prop_train)

display(rf)
```

--------------------------------

### Class-conditional evaluation metrics

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output of evaluation metrics for a class-conditional conformal classifier.

```python
{'error': 0.0018939393939394478,
 'avg_c': 1.740530303030303,
 'one_c': 0.25946969696969696,
 'empty': 0.0,
 'ks_test': 0.11458837583733483,
 'time_fit': 7.152557373046875e-07,
 'time_evaluate': 0.06147575378417969}
```

--------------------------------

### NumPy Array of Predicted Probabilities

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output format for predicted probabilities with online calibration.

```numpy
array([0.09369225, 0.52548032, 0.49992477, ..., 0.72979714, 0.87495964,
       0.58352253])
```

--------------------------------

### Load and split dataset

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Load a dataset from OpenML and split it into training, testing, and calibration sets using scikit-learn.

```python
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

dataset = fetch_openml(name="qsar-biodeg", parser="auto")

X = dataset.data.values.astype(float)
y = dataset.target.values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(X_train, y_train,
                                                            test_size=0.25)
```

--------------------------------

### Classifier Evaluation Results

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output from evaluating a conformal classifier, showing various metrics.

```python
{'error': 0.007575757575757569,
 'avg_c': 1.6117424242424243,
 'one_c': 0.38825757575757575,
 'empty': 0.0,
 'ks_test': 0.14097384777782784,
 'time_fit': 1.9073486328125e-06,
 'time_evaluate': 0.05298352241516113}
```

--------------------------------

### Evaluate WrapRegressor with Default Settings

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Evaluates a WrapRegressor with default online calibration settings. Shows the resulting error metrics and timing.

```python
cps_mond_norm.evaluate(X_test, y_test, confidence=0.99, y_min=0, y_max=1, 
                       online=True)
```

--------------------------------

### SimpleJumper Strategies Comparison

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/martingales_nb.ipynb

Compares SimpleJumper with default and modified jumping rates (J) and betting functions (epsilon, step).

```python
martingales_simple_jumper = {
    "SimpleJumper (epsilon)" : SimpleJumper().apply(p_values),
    "SimpleJumper (step)" : SimpleJumper(bf = StepBettingFunction()).apply(p_values),
    "SimpleJumper (epsilon, J = 0.0001)" : SimpleJumper(J = 0.0001).apply(p_values),
    "SimpleJumper (step, J = 0.0001)" : SimpleJumper(J = 0.0001, bf = StepBettingFunction()).apply(p_values),
}

plot_martingales(martingales_simple_jumper)
```

--------------------------------

### Mondrian prediction sets output

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output of prediction sets from a Mondrian conformal classifier.

```numpy
[['2'],
 ['1', '2'],
 ['1'],
 ...,
 ['1'],
 ['1'],
 ['1', '2']]
```

--------------------------------

### Import and Split House Sales Dataset

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/Tutorial 2026-06-02.ipynb

Loads the house sales dataset from OpenML and splits it into training and testing sets. Features related to date are dropped before splitting.

```python
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

dataset = fetch_openml(name="house_sales", version=3, parser="auto")

df = dataset.data.drop(["date_year", "date_month", "date_day"], axis="columns")

X = dataset.data.values.astype(float)
y = dataset.target.values.astype(float)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(X_train, y_train,
                                                            test_size=0.25)
```

--------------------------------

### Predicted p-values output

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output of predicted p-values, showing probabilities for each class for multiple test instances.

```numpy
array([[0.00427104, 0.74842304],
       [0.07874355, 0.2950549 ],
       [0.50529983, 0.01557963],
       ...,
       [0.8413356 , 0.00201167],
       [0.84402215, 0.00654927],
       [0.29601955, 0.07766093]])
```

--------------------------------

### Initialize EpsilonBettingFunction with custom epsilons

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Create an EpsilonBettingFunction with a specific list of epsilon values. Ensure all values are within the (-2, 2) range to avoid runtime errors.

```python
bf = EpsilonBettingFunction(E = [-1, 0, 1])
```

--------------------------------

### Get Multiple Custom Percentiles

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Request multiple lower and higher percentiles simultaneously. Ensure `cps=True` during calibration.

```python
percentiles = cps.predict_percentiles(X_test,
> : lower_percentiles=[2.5,5],
>   higher_percentiles=[95,97.5])
```

--------------------------------

### Get Prediction Intervals

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Obtain prediction intervals at a specified confidence level. Ensure the WrapRegressor object has been calibrated.

```python
intervals = w.predict_int(X_test, confidence=0.99)
```

--------------------------------

### Numpy Array of Clamped Prediction Intervals

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example NumPy array output for prediction intervals that have been clamped to a minimum value.

```numpy
array([[2302049.84, 2783188.76],
       [ 588518.88, 1069657.8 ],
       [      0.  ,  441017.21],
       ...,
       [ 806926.11, 1288065.03],
       [  19499.3 ,  500638.22],
       [ 209553.85,  690692.77]])
```

--------------------------------

### Prediction sets output

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output of prediction sets, represented as lists of predicted class labels for each test instance.

```numpy
[['2'],
 ['1', '2'],
 ['1', '2'],
 ...,
 ['1'],
 ['1'],
 ['1', '2']]
```

--------------------------------

### Initialize SleeperStayer with Default Settings

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Obtain an instance of SleeperStayer with its default investment rate and betting function.

```python
m = SleeperStayer()
```

--------------------------------

### NumPy Array of Prediction Intervals with Online CPS

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output format for prediction intervals from an online-calibrated conformal predictive system.

```numpy
array([[1719676.80439219, 3707173.76806116],
       [ 684289.27240227, 1032856.71531186],
       [ 127189.61835749,  274385.59426486],
       ...,
       [ 630347.70469164, 1594876.58130005],
       [ 167399.51044545,  337513.60197203],
       [ 232815.51352497,  641580.14787679]])
```

--------------------------------

### Initialize SimpleJumper with custom parameters

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Initialize SimpleJumper with a specified jumping rate (J) and a custom betting function (bf). The betting function must implement an 'apply' method.

```python
m = SimpleJumper(J = 0.1, bf = StepBettingFunction())
```

--------------------------------

### Get Conformal Predictive Distributions

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Obtain conformal predictive distributions from a conformal predictive system. This can be done with or without online calibration.

```python
cpds = cps.predict_cpds(X_test, online=True)
```

--------------------------------

### Initialize Conformal Regressors Dictionary

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Sets up a dictionary mapping names to conformal regressor objects and their corresponding difficulty estimates for test instances. This is used for evaluating multiple regressors.

```python
all_cr = {
    "Std CR": (cr_std, []),
    "Norm CR knn dist": (cr_norm_knn_dist, sigmas_test_knn_dist),
    "Norm CR knn std": (cr_norm_knn_std, sigmas_test_knn_std),
    "Norm CR knn res": (cr_norm_knn_res, sigmas_test_knn_res),
    "Norm CR var" : (cr_norm_var, sigmas_test_var),
    "Mond CR": (cr_mond, sigmas_test_var),
}
```

--------------------------------

### Obtain Prediction Intervals with Online Calibration

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Generate prediction intervals using online calibration. This method allows for extending the calibration set with `warm_start=True`. Requires predicted values and true targets.

```python
intervals = cps_std.predict_int_online(y_hat_test, y=y_test)
```

--------------------------------

### Train RandomForestClassifier on Full Dataset

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Trains a RandomForestClassifier on the entire training dataset. Use this when starting with an empty calibration set for online calibration.

```python
rf_full = RandomForestClassifier(n_jobs=-1, n_estimators=500)

rf_full.fit(X_train, y_train)
```

--------------------------------

### Initialize CompositeMartingale

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Create a CompositeMartingale by providing a list of conformal test martingales. Each martingale must have an 'apply' method.

```python
cm = CompositeMartingale([SimpleJumper(), SleeperStayer()])
```

--------------------------------

### Standard Prediction Sets

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Example of obtaining prediction sets at the default 95% confidence level using a fitted standard conformal classifier.

```APIDOC
## Standard Prediction Sets

### Description
Obtain prediction sets at the default 95% confidence level using a fitted standard conformal classifier.

### Method
```python
prediction_sets = cc_std.predict_set(alphas_test)
```

### Parameters
- **alphas_test** (*array-like*): Non-conformity scores for the test set.
- **cc_std** (*object*): A fitted standard conformal classifier.
```

--------------------------------

### Initialize and Fit RandomForestRegressor for OOB

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initialize a RandomForestRegressor with OOB scoring enabled and fit it to the training data. This prepares the regressor for OOB-based calibration.

```python
learner_full = RandomForestRegressor(n_jobs=-1, n_estimators=500, oob_score=True)

rf = WrapRegressor(learner_full)

rf.fit(X_train, y_train)
```

--------------------------------

### Numpy Array of Prediction Intervals

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example NumPy array output representing prediction intervals (lower and upper bounds) for test instances.

```numpy
array([[1938866.06, 3146372.54],
       [ 225335.1 , 1432841.58],
       [-403305.49,  804200.99],
       ...,
       [ 443742.33, 1651248.81],
       [-343684.48,  863822.  ],
       [-153629.93, 1053876.55]])
```

--------------------------------

### Binary prediction sets output

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output of prediction sets as binary vectors, indicating the presence or absence of each class for each test instance.

```numpy
array([[0, 1],
       [1, 1],
       [1, 1],
       ...,
       [1, 0],
       [1, 0],
       [1, 1]])
```

--------------------------------

### Load and Split Dataset for Regression

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Loads the 'house_sales' dataset and splits it into training, testing, and calibration sets for regression tasks.

```python
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

dataset = fetch_openml(name="house_sales", version=3, parser="auto")

X = dataset.data.values.astype(float)
y = dataset.target.values.astype(float)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(X_train, y_train,
                                                            test_size=0.25)
```

--------------------------------

### Get Prediction Sets with OOB Calibration

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Generate prediction sets for the test set using a conformal classifier calibrated with OOB predictions.

```python
prediction_sets_oob = rf.predict_set(X_test)

display(prediction_sets_oob[:40])
```

--------------------------------

### Get p-values for true targets

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Obtain p-values for the true targets using the predict_cps method with y_test. This requires the WrapRegressor to be calibrated with cps=True.

```python
p_values = w.predict_cps(X_test, y=y_test)
```

--------------------------------

### Create Standard Conformal Predictive System

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initializes and calibrates a standard Conformal Predictive System (CPS) using WrapRegressor. Requires WrapRegressor.

```python
cps_std = WrapRegressor(learner_prop)

cps_std.calibrate(X_cal, y_cal, cps=True)

display(cps_std)
```

--------------------------------

### Initialize SleeperStayer with Custom Settings

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.martingales.md

Initialize SleeperStayer with a specific investment rate (J) and a custom betting function (EpsilonBettingFunction).

```python
m = SleeperStayer(J = 0.1, bf = EpsilonBettingFunction())
```

--------------------------------

### Get Percentiles with Custom Lower and Higher Percentiles

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Obtain custom percentiles from a calibrated conformal predictive system. Requires `cps=True` during calibration.

```python
percentiles = cps.predict_percentiles(X_test, lower_percentiles=2.5,
                                    higher_percentiles=97.5)
```

--------------------------------

### Import Core crepes and Utility Packages

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Imports necessary libraries including NumPy, pandas, matplotlib, scikit-learn, scipy, and crepes components. It also prints the crepes version and sets random seed for reproducibility.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_openml

from scipy.stats import kstest

from crepes import (
                    ConformalClassifier,
                    ConformalRegressor,
                    ConformalPredictiveSystem,
                    __version__)

from crepes.extras import hinge, margin, binning, DifficultyEstimator

print(f"crepes v. {__version__}")

np.random.seed(602211023)
np.set_printoptions(legacy='1.25')
```

--------------------------------

### Load and Prepare Classification Dataset

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Fetches the 'gas-drift' dataset from OpenML and prepares the features (X) and target (y) for use. Ensures data is in float format.

```python
dataset = fetch_openml(name="gas-drift", parser="auto")

X = dataset.data.values.astype(float)
y = dataset.target.values
```

--------------------------------

### Evaluation metrics output

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example output of evaluation metrics for the conformal classifier, including error rate, average set size, and computation times.

```python
{'error': 0.007575757575757569,
 'avg_c': 1.6136363636363635,
 'one_c': 0.38636363636363635,
 'empty': 0.0,
 'ks_test': 0.0030978768439862566,
 'time_fit': 9.5367431640625e-07,
 'time_evaluate': 0.024116039276123047}
```

--------------------------------

### Fitting Difficulty Estimator (k-NN Distances)

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Initializes and fits a DifficultyEstimator using k-nearest neighbor distances. Normalizes difficulty estimates using min-max scaling.

```python
de_knn = DifficultyEstimator()

de_knn.fit(X=X_prop_train, scaler=True)

display(de_knn)

sigmas_cal_knn_dist = de_knn.apply(X_cal)

cr_norm_knn_dist = ConformalRegressor()

cr_norm_knn_dist.fit(residuals_cal, sigmas=sigmas_cal_knn_dist)

display(cr_norm_knn_dist)
```

--------------------------------

### Get and Visualize p-values

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Calculates p-values using `predict_p_online` and visualizes their empirical cumulative distribution function (ECDF). Also prints the KS-test p-value.

```python
p_values = cr_full.predict_p_online(y_hat_test, y_test, bins=bins_test)

display(p_values)

plt.ecdf(p_values)
plt.xlim(0,1)
plt.xlabel("p-value")
plt.ylabel("ECDF")
plt.show()

print(f"KS-test: {kstest(p_values, "uniform").pvalue}")
```

--------------------------------

### Create Normalized OOB Conformal Predictive System

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initializes and calibrates a normalized Out-of-Bag (OOB) Conformal Predictive System (CPS) using WrapRegressor with OOB predictions and a specified difficulty estimator. Requires WrapRegressor.

```python
cps_norm_oob = WrapRegressor(learner_full).calibrate(X_train, y_train, 
                                                     de=de_var_oob, oob=True, cps=True)
display(cps_norm_oob)
```

--------------------------------

### Get Percentiles with Online Calibration

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Obtain percentiles using online calibration. Requires the correct labels `y_test` for the test objects and `higher_percentiles` specified.

```python
intervals = cps.predict_percentiles(X_test, y_test,
                                    higher_percentiles=[90,95,99],
                                    online=True)
```

--------------------------------

### KNN Difficulty Estimator with OOB Labels

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Initializes and fits a k-nearest neighbors difficulty estimator using out-of-bag (OOB) labels. This method is suitable when direct label information is available for OOB estimation.

```python
de_knn_std_oob = DifficultyEstimator()

de_knn_std_oob.fit(X=X_train, y=y_train, scaler=True, oob=True)

display(de_knn_std_oob)

sigmas_knn_std_oob = de_knn_std_oob.apply()

cr_norm_knn_std_oob = ConformalRegressor()

cr_norm_knn_std_oob.fit(residuals=residuals_oob, sigmas=sigmas_knn_std_oob)

sigmas_test_knn_std_oob = de_knn_std_oob.apply(X_test)

intervals_norm_knn_std_oob = cr_norm_knn_std_oob.predict_int(
    y_hat_full, sigmas=sigmas_test_knn_std_oob, y_min=0, y_max=1)
```

--------------------------------

### Get Prediction Intervals with Online Calibration

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Obtain prediction intervals using online calibration. Requires the correct labels `y_test` for the test objects.

```python
intervals = w.predict_int(X_test, y_test, online=True)
```

--------------------------------

### Get Prediction Sets with Labels

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb

Retrieve prediction sets as lists of labels by providing class names. This offers a more human-readable output for the predicted labels.

```python
label_sets = cc_std.predict_set(alphas_test, labels=rf.classes_, confidence=0.95)
label_sets[:20]
```

--------------------------------

### Create OOB Conformal Predictive System

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Initializes and calibrates an Out-of-Bag (OOB) Conformal Predictive System (CPS) using WrapRegressor with OOB predictions enabled. Requires WrapRegressor.

```python
cps_std_oob = WrapRegressor(learner_full).calibrate(X_train, y_train, oob=True,
                                                    cps=True)
display(cps_std_oob)
```

--------------------------------

### Creating a Composite Martingale

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/martingales_nb.ipynb

Demonstrates the creation of a composite martingale by averaging the output of multiple individual martingales. This approach can be used to combine the strengths of different martingale strategies.

```python
martingales = [SimpleJumper(), SleeperStayer(), SleeperDrifter()]

martingales_composite = {
    "SimpleJumper" : martingales[0].apply(p_values),
    "SleeperStayer" : martingales[1].apply(p_values),
    "SleeperDrifter" : martingales[2].apply(p_values),
    "Composite" : CompositeMartingale(martingales).apply(p_values),
}

plot_martingales(martingales_composite, change_point=t)
```

--------------------------------

### Mondrian Prediction Sets

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.md

Example of obtaining prediction sets at a 90% confidence level using a fitted Mondrian conformal classifier, specifying bins for categories.

```APIDOC
## Mondrian Prediction Sets

### Description
Obtain prediction sets at a specified confidence level (e.g., 90%) using a fitted Mondrian conformal classifier, providing bin labels for categories.

### Method
```python
p_values = cc_mond.predict_set(alphas_test, bins=bins_test, confidence=0.9)
```

### Parameters
- **alphas_test** (*array-like*): Non-conformity scores for the test set.
- **bins_test** (*array-like*): Mondrian categories (bin labels) for the test set.
- **cc_mond** (*object*): A fitted Mondrian conformal classifier.
- **confidence** (*float*): The desired confidence level (e.g., 0.9 for 90%).
```

--------------------------------

### Numpy Array of Normalized Prediction Intervals

Source: https://github.com/henrikbostrom/crepes/blob/main/README.md

Example NumPy array output for prediction intervals generated with a difficulty estimator, showing normalized interval widths.

```numpy
array([[1769594.36212355, 3315644.23787645],
       [ 693827.99796647,  964348.68203353],
       [ 124886.97469338,  276008.52530662],
       ...,
       [ 661373.45043166, 1433617.68956833],
       [ 178769.2939384 ,  341368.2260616 ],
       [ 222837.12801117,  677409.49198883]])
```

--------------------------------

### Get Prediction Thresholds with predict_cps

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb

Obtain prediction thresholds for test data using the `predict_cps` method. This is useful for determining the range of plausible target values.

```python
thresholds = cps_mond_norm.predict_cps(X_test, higher_percentiles=50)

display(thresholds)
```

--------------------------------

### Fitting KNN Difficulty Estimator

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/Tutorial 2026-06-02.ipynb

Initializes and fits a KNN-based difficulty estimator. Requires training data (X, y) and specifies parameters like k, scaler, and beta.

```python
from crepes.extras import DifficultyEstimator

de_knn = DifficultyEstimator()
de_knn.fit(X=X_prop_train, 
           k=5,
           scaler=True,
           y=y_prop_train,
           beta=beta)
```

--------------------------------

### Create Bins from Difficulty Estimates

Source: https://github.com/henrikbostrom/crepes/blob/main/docs/crepes.extras.md

Use the binning function to create bins and their thresholds from a vector of difficulty estimates. The bins are assigned labels from 0 to 19.

```python
from crepes.extras import binning

bins, bin_thresholds = binning(sigmas, bins=20)
```