Probatus

https://ing-bank.github.io/probatus/
Admin
Probatus is a Python library for analyzing binary classification models and their training data,...
Tokens:78,462
Snippets:460
Trust Score:9.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Probatus from Source

Source: https://ing-bank.github.io/probatus/index.html

Install Probatus by cloning the repository and installing locally. This method requires Git and Python 3.9 or higher.

```bash
git clone https://gitlab.com/ing_rpaa/probatus.git
cd probatus
pip install .

```

--------------------------------

### Install Probatus

Source: https://ing-bank.github.io/probatus/howto/grouped_data.html

Installs the Probatus library. This is a prerequisite for using the library's functionalities.

```bash
%%capture
!pip install probatus

```

--------------------------------

### Install Probatus and CatBoost

Source: https://ing-bank.github.io/probatus/tutorials/nb_automatic_best_num_features.html

Install the necessary libraries, Probatus and CatBoost, using pip. This is a prerequisite for using the feature selection functionalities.

```bash
!pip install probatus
!pip install catboost
```

--------------------------------

### Install Probatus via Pip

Source: https://ing-bank.github.io/probatus/index.html

Install the Probatus library using pip. Ensure you have Python 3.9 or higher.

```bash
pip install probatus

```

--------------------------------

### Basic ShapRFECV Example

Source: https://ing-bank.github.io/probatus/tutorials/nb_automatic_best_num_features.html

A simple example demonstrating ShapRFECV. It generates classification data, initializes a CatBoostClassifier, and then uses ShapRFECV to fit and compute the feature elimination report.

```python
# Simple ShapRFECV example
X, y = make_classification(n_samples=500, n_informative=20, n_features=50)
model = CatBoostClassifier(n_estimators=100, verbose=0)
shap_elimination = ShapRFECV(model, step=0.2, min_features_to_select=5, cv=5, scoring="f1")
report = shap_elimination.fit_compute(X, y)
```

--------------------------------

### Get Fit Parameters for CatBoost

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves fit parameters for a CatBoost model. Ensure CatBoost is installed.

```python
from catboost import CatBoost

if isinstance(model, CatBoost):
    return self._get_fit_params_CatBoost(
        X_train=X_train,
        y_train=y_train,
        X_val=X_val,
        y_val=y_val,
        sample_weight=sample_weight,
        train_index=train_index,
        val_index=val_index,
    )
```

--------------------------------

### Get Fit Parameters for LightGBM

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves fit parameters for a LightGBM model. Ensure LightGBM is installed.

```python
from lightgbm import LGBMModel

if isinstance(model, LGBMModel):
    return self._get_fit_params_lightGBM(
        X_train=X_train,
        y_train=y_train,
        X_val=X_val,
        y_val=y_val,
        sample_weight=sample_weight,
        train_index=train_index,
        val_index=val_index,
    )
```

--------------------------------

### Setup Data and Model for Resemblance Analysis

Source: https://ing-bank.github.io/probatus/tutorials/nb_custom_scoring.html

Prepares sample data (X1, X2) and a RandomForestClassifier model for use with Probatus's resemblance analysis. Imports necessary libraries.

```python
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import make_scorer

from probatus.sample_similarity import SHAPImportanceResemblance
from probatus.utils import Scorer

# Prepare two samples
feature_names = ["f1", "f2", "f3", "f4"]
X1 = pd.DataFrame(make_classification(n_samples=1000, n_features=4, random_state=0)[0], columns=feature_names)
X2 = pd.DataFrame(
    make_classification(n_samples=1000, n_features=4, shift=0.5, random_state=0)[0], columns=feature_names
)

# Prepare model
model = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0)

```

--------------------------------

### Initialize ShapRFECV with RandomForestClassifier

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Instantiate ShapRFECV with a RandomForestClassifier and specify parameters for feature elimination. This example demonstrates setting up the recursive feature elimination process.

```python
import numpy as np
import pandas as pd
from probatus.feature_elimination import ShapRFECV
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV

feature_names = [
    'f1', 'f2', 'f3', 'f4', 'f5', 'f6', 'f7',
    'f8', 'f9', 'f10', 'f11', 'f12', 'f13', 'f14',
    'f15', 'f16', 'f17', 'f18', 'f19', 'f20'
]

X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, n_redundant=5, n_repeated=0, n_classes=2, random_state=42)
X = pd.DataFrame(X, columns=feature_names)
y = pd.Series(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Define the model and its hyperparameters for tuning
model = RandomForestClassifier(random_state=42)
param_distributions = {
    'n_estimators': [50, 100, 150],
    'max_depth': [5, 10, 15, None],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4]
}

# Create a RandomizedSearchCV object
search = RandomizedSearchCV(
    estimator=model,
    param_distributions=param_distributions,
    n_iter=10,
    cv=3,
    scoring='accuracy',
    random_state=42,
    n_jobs=-1
)

# Initialize ShapRFECV with the search object
shap_rfe_cv = ShapRFECV(
    model=search,
    step=1,
    cv=3,
    scoring='accuracy',
    verbose=1,
    n_jobs=-1
)

# Fit ShapRFECV to the training data
shap_rfe_cv.fit(X_train, y_train)

# Get the final feature set
final_features = shap_rfe_cv.get_final_features()
print(f"Final features: {final_features}")

# Plot the performance
shap_rfe_cv.plot(X_test, y_test)

```

--------------------------------

### Get LightGBM Fit Parameters

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Prepares the necessary parameters for fitting a LightGBM model during cross-validation. Supports sample weights and custom train/validation indices.

```python
def _get_fit_params_lightGBM(
    self, X_train, y_train, X_val, y_val, sample_weight=None, train_index=None, val_index=None
):
    """Get the fit parameters for for a LightGBM Model.

    Args:

        X_train (pd.DataFrame):
            Train Dataset used in CV.

        y_train (pd.Series):
            Train labels for X.

        X_val (pd.DataFrame):
            Validation Dataset used in CV.

        y_val (pd.Series):
            Validation labels for X.

        sample_weight (pd.Series, np.ndarray, list, optional): 
            array-like of shape (n_samples,) - only use if the model you're using supports

```

--------------------------------

### Get CatBoost Fit Parameters

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Prepares fit parameters for CatBoost models, including handling categorical features and optional sample weighting. Uses the `Pool` object for data input.

```python
from catboost import Pool

cat_features = [col for col in X_train.select_dtypes(include=["category"]).columns]
fit_params = {
    "X": Pool(X_train, y_train, cat_features=cat_features),
    "eval_set": Pool(X_val, y_val, cat_features=cat_features),
    # Evaluation metric should be passed during initialization
}
if sample_weight is not None:
    fit_params["X"].set_weight(sample_weight.iloc[train_index])
    fit_params["eval_set"].set_weight(sample_weight.iloc[val_index])

return fit_params
```

--------------------------------

### Get Fit Parameters for LightGBM

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves the fit parameters specifically for a LightGBM model, including data, evaluation sets, and callbacks for early stopping and logging.

```APIDOC
## _get_fit_params_LightGBM

### Description
Get the fit parameters for for a LightGBM Model.

### Args
- **X_train** (pd.DataFrame): Train Dataset used in CV.
- **y_train** (pd.Series): Train labels for X.
- **X_val** (pd.DataFrame): Validation Dataset used in CV.
- **y_val** (pd.Series): Validation labels for X.
- **sample_weight** (pd.Series, np.ndarray, list, optional): array-like of shape (n_samples,) - only use if the model you're using supports sample weighting (check the corresponding scikit-learn documentation). Array of weights that are assigned to individual samples. Note that they're only used for fitting of the model, not during evaluation of metrics. If not provided, then each sample is given unit weight.
- **train_index** (np.array): Positions of train folds samples.
- **val_index** (np.array): Positions of validation fold samples.

### Raises
- ValueError: if the model is not supported.

### Returns
- dict: fit parameters
```

--------------------------------

### Get LightGBM Fit Parameters

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Constructs fit parameters for LightGBM models, including early stopping and evaluation metrics. Handles optional sample weighting for training and validation sets.

```python
from lightgbm import early_stopping, log_evaluation

fit_params = {
    "X": X_train,
    "y": y_train,
    "eval_set": [(X_val, y_val)],
    "eval_metric": self.eval_metric,
    "callbacks": [
        early_stopping(self.early_stopping_rounds, first_metric_only=True),
        log_evaluation(1 if self.verbose >= 2 else 0),
    ],
}

if sample_weight is not None:
    fit_params["sample_weight"] = sample_weight.iloc[train_index]
    fit_params["eval_sample_weight"] = [sample_weight.iloc[val_index]]

return fit_params
```

--------------------------------

### Get Fit Parameters for XGBoost

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves fit parameters for an XGBoost model. Ensure XGBoost is installed.

```python
from xgboost.sklearn import XGBModel

if isinstance(model, XGBModel):
    return self._get_fit_params_XGBoost(
        X_train=X_train,
        y_train=y_train,
        X_val=X_val,
        y_val=y_val,
        sample_weight=sample_weight,
        train_index=train_index,
        val_index=val_index,
    )
```

--------------------------------

### Initialize and Run ShapRFECV

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Demonstrates initializing ShapRFECV with a RandomizedSearchCV model, specifying step, cross-validation, scoring, and number of jobs. It then fits and computes the feature elimination report on sample data.

```python
import numpy as np
import pandas as pd
from probatus.feature_elimination import ShapRFECV
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV

feature_names = [
    'f1', 'f2', 'f3', 'f4', 'f5', 'f6', 'f7',
    'f8', 'f9', 'f10', 'f11', 'f12', 'f13',
    'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f20']

# Prepare two samples
X, y = make_classification(n_samples=200, class_sep=0.05, n_informative=6, n_features=20,
                           random_state=0, n_redundant=10, n_clusters_per_class=1)
X = pd.DataFrame(X, columns=feature_names)


# Prepare model and parameter search space
model = RandomForestClassifier(max_depth=5, class_weight='balanced')

param_grid = {
    'n_estimators': [5, 7, 10],
    'min_samples_leaf': [3, 5, 7, 10],
}
search = RandomizedSearchCV(model, param_grid)


# Run feature elimination
shap_elimination = ShapRFECV(
    model=search, step=0.2, cv=10, scoring='roc_auc', n_jobs=3)
report = shap_elimination.fit_compute(X, y)

# Make plots
performance_plot = shap_elimination.plot()
```

--------------------------------

### Initialize and Use ShapModelInterpreter

Source: https://ing-bank.github.io/probatus/api/model_interpret.html

Demonstrates how to initialize ShapModelInterpreter, fit it with a model and data, and generate various interpretation plots. Requires scikit-learn, pandas, and numpy.

```python
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from probatus.interpret import ShapModelInterpreter
import numpy as np
import pandas as pd

feature_names = ['f1', 'f2', 'f3', 'f4']

# Prepare two samples
X, y = make_classification(n_samples=5000, n_features=4, random_state=0)
X = pd.DataFrame(X, columns=feature_names)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Prepare and fit model. Remember about class_weight="balanced" or an equivalent.
model = RandomForestClassifier(class_weight='balanced', n_estimators = 100, max_depth=2, random_state=0)
model.fit(X_train, y_train)

# Train ShapModelInterpreter
shap_interpreter = ShapModelInterpreter(model)
feature_importance = shap_interpreter.fit_compute(X_train, X_test, y_train, y_test)

# Make plots
ax1 = shap_interpreter.plot('importance')
ax2 = shap_interpreter.plot('summary')
ax3 = shap_interpreter.plot('dependence', target_columns=['f1', 'f2'])
ax4 = shap_interpreter.plot('sample', samples_index=[X_test.index.tolist()[0]])
```

--------------------------------

### Initialize Output Variables

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Sets up placeholder variables for storing results like training/testing data and scores, which will be populated during the `fit()` method execution.

```python
def _init_output_variables(self):
    """
    Initializes variables that will be filled in during fit() method, and are used as output.
    """
    self.X_train = None
    self.X_test = None
    self.y_train = None
    self.y_test = None
    self.train_score = None
    self.test_score = None
    self.report = None
```

--------------------------------

### get_shap_values

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Gets the SHAP values generated on the test set.

```APIDOC
## get_shap_values

### Description
Gets the SHAP values generated on the test set.

### Returns
- (np.array) - SHAP values.
```

--------------------------------

### Prepare Data and Model for Resemblance Analysis

Source: https://ing-bank.github.io/probatus/tutorials/nb_sample_similarity.html

Prepares two distinct datasets (X1, X2) using `make_classification` and initializes a `RandomForestClassifier` for resemblance analysis. Ensure datasets have the same features.

```python
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

# Prepare two samples
feature_names = ["f1", "f2", "f3", "f4"]
X1 = pd.DataFrame(make_classification(n_samples=1000, n_features=4, random_state=0)[0], columns=feature_names)
X2 = pd.DataFrame(
    make_classification(n_samples=1000, n_features=4, shift=0.5, random_state=0)[0], columns=feature_names
)

# Prepare model
model = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0)
```

--------------------------------

### Get Reduced Feature Set

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieve the final set of features after elimination. Specify the desired number of features to retain.

```python
final_features_set = shap_elimination.get_reduced_features_set(num_features=3)
```

--------------------------------

### Get SHAP Values

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Retrieves the computed SHAP values from the test set. This method is useful for further analysis or custom visualizations.

```python
def get_shap_values(self):
        """
        Gets the SHAP values generated on the test set.

        Returns:
             (np.array):
        """
```

--------------------------------

### Import Libraries and Load Data

Source: https://ing-bank.github.io/probatus/tutorials/nb_shap_variance_penalty_and_results_comparison.html

Imports required libraries including ShapRFECV, data generation tools, a CatBoost classifier, and numerical/data manipulation libraries. It then generates synthetic data for demonstration.

```python
from probatus.feature_elimination import ShapRFECV
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from catboost import CatBoostClassifier
import numpy as np
import pandas as pd
```

```python
X, y = make_classification(n_samples=500, n_informative=20, n_features=100)
model = CatBoostClassifier(n_estimators=100, verbose=0)
shap_elimination = ShapRFECV(model=model, step=0.2, min_features_to_select=5, cv=5, scoring="f1", n_jobs=5, verbose=1)
report_with_penalty = shap_elimination.fit_compute(X, y, shap_variance_penalty_factor=1.0)
report_without_penalty = shap_elimination.fit_compute(X, y, shap_variance_penalty_factor=0)
```

--------------------------------

### Get Fit Parameters for CatBoost

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves the fit parameters for a CatBoost model, handling categorical features and optional sample weights.

```APIDOC
## _get_fit_params_CatBoost

### Description
Get the fit parameters for for a CatBoost Model.

### Args
- **X_train** (pd.DataFrame): Train Dataset used in CV.
- **y_train** (pd.Series): Train labels for X.
- **X_val** (pd.DataFrame): Validation Dataset used in CV.
- **y_val** (pd.Series): Validation labels for X.
- **sample_weight** (pd.Series, np.ndarray, list, optional): array-like of shape (n_samples,) - only use if the model you're using supports sample weighting (check the corresponding scikit-learn documentation). Array of weights that are assigned to individual samples. Note that they're only used for fitting of the model, not during evaluation of metrics. If not provided, then each sample is given unit weight.
- **train_index** (np.array): Positions of train folds samples.
- **val_index** (np.array): Positions of validation fold samples.

### Raises
- ValueError: if the model is not supported.

### Returns
- dict: fit parameters
```

--------------------------------

### __init__

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Initializes the ResemblanceModel class with a given model and configuration parameters for similarity analysis.

```APIDOC
## __init__

### Description
Initializes the class.

### Parameters
#### Path Parameters
- **model** (model object) - Required - Regression or classification model or pipeline.
- **scoring** (string or Scorer) - Optional - Metric for which the model performance is calculated. It can be either a metric name aligned with predefined classification scorers names in sklearn. Another option is using probatus.utils.Scorer to define a custom metric. The recommended option for this class is 'roc_auc'. Default: 'roc_auc'
- **test_prc** (float) - Optional - Percentage of data used to test the model. By default 0.25 is set. Default: 0.25
- **n_jobs** (int) - Optional - Number of parallel executions. If -1 use all available cores. By default 1. Default: 1
- **verbose** (int) - Optional - Controls verbosity of the output: 0 - neither prints nor warnings are shown, 1 - only most important warnings, 2 - shows all prints and all warnings. Default: 0
- **random_state** (int) - Optional - Random state set at each round of feature elimination. If it is None, the results will not be reproducible and in random search at each iteration a different hyperparameters might be tested. For reproducible results set it to an integer. Default: None
```

--------------------------------

### DependencePlotter Initialization and Usage

Source: https://ing-bank.github.io/probatus/api/model_interpret.html

Demonstrates how to initialize the DependencePlotter with a model and use its methods for fitting and plotting.

```APIDOC
## DependencePlotter

### Description
Plotter used to plot SHAP dependence plot together with the target rates. Currently it supports tree-based and linear models.

### Parameters
- **model** (object) - Required - The classifier for which interpretation is done.
- **verbose** (int, optional) - Controls verbosity of the output. Defaults to 0.
- **random_state** (int, optional) - Random state for reproducibility. Defaults to None.

### Methods
#### `__init__(self, model, verbose=0, random_state=None)`
Initializes the DependencePlotter.

#### `__repr__(self)`
Returns a string representation of the plotter.

#### `fit(self, X, y, column_names=None, class_names=None, precalc_shap=None, **shap_kwargs)`
Fits the plotter to the model and data by computing the shap values. If `precalc_shap` is provided, it is used directly.

- **X** (pd.DataFrame) - Input variables.
- **y** (pd.Series) - Target variable.
- **column_names** (list of str, optional) - List of feature names. Defaults to column names from X.
- **class_names** (list of str, optional) - List of class names. Defaults to ['Negative Class', 'Positive Class'].
- **precalc_shap** (np.array, optional) - Pre-calculated SHAP values.
- **`**shap_kwargs`** - Keyword arguments passed to `shap.Explainer`.

### Example
```python
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from probatus.interpret import DependencePlotter

X, y = make_classification(n_samples=15, n_features=3, n_informative=3, n_redundant=0, random_state=42)
model = RandomForestClassifier().fit(X, y)
bdp = DependencePlotter(model)
shap_values = bdp.fit_compute(X, y)

bdp.plot(feature=2)
```
```

--------------------------------

### Get Fit Parameters

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves the fit parameters for a given classifier or regressor model. This method supports LightGBM, XGBoost, and CatBoost models.

```APIDOC
## Get Fit Parameters

### Description
Retrieves the fit parameters for the specified classifier or regressor.

### Method Signature
`get_fit_params(model, X_train, y_train, X_val, y_val, sample_weight=None, train_index=None, val_index=None)`

### Parameters
- **model** (classifier or regressor): The model to be fitted on the train folds.
- **X_train** (pd.DataFrame): The training dataset used in CV.
- **y_train** (pd.Series): The training labels for X.
- **X_val** (pd.DataFrame): The validation dataset used in CV.
- **y_val** (pd.Series): The validation labels for X.
- **sample_weight** (pd.Series, np.ndarray, list, optional): Array-like of shape (n_samples,) assigned to individual samples for fitting. Only used if the model supports sample weighting.
- **train_index** (np.array): Positions of train folds samples.
- **val_index** (np.array): Positions of validation fold samples.

### Returns
- **dict**: A dictionary containing the fit parameters.

### Raises
- **ValueError**: If the provided model type is not supported.
```

--------------------------------

### Import Necessary Libraries

Source: https://ing-bank.github.io/probatus/tutorials/nb_automatic_best_num_features.html

Import ShapRFECV from probatus.feature_elimination, make_classification for generating sample data, and CatBoostClassifier for the model.

```python
from probatus.feature_elimination import ShapRFECV
from sklearn.datasets import make_classification
from catboost import CatBoostClassifier
```

--------------------------------

### Get Fit Parameters for XGBoost

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves the fit parameters for an XGBoost model, including training and validation data, and optional sample weights.

```APIDOC
## _get_fit_params_XGBoost

### Description
Get the fit parameters for for a XGBoost Model.

### Args
- **X_train** (pd.DataFrame): Train Dataset used in CV.
- **y_train** (pd.Series): Train labels for X.
- **X_val** (pd.DataFrame): Validation Dataset used in CV.
- **y_val** (pd.Series): Validation labels for X.
- **sample_weight** (pd.Series, np.ndarray, list, optional): array-like of shape (n_samples,) - only use if the model you're using supports sample weighting (check the corresponding scikit-learn documentation). Array of weights that are assigned to individual samples. Note that they're only used for fitting of the model, not during evaluation of metrics. If not provided, then each sample is given unit weight.
- **train_index** (np.array): Positions of train folds samples.
- **val_index** (np.array): Positions of validation fold samples.

### Raises
- ValueError: if the model is not supported.

### Returns
- dict: fit parameters
```

--------------------------------

### Get XGBoost Fit Parameters

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Generates fit parameters for XGBoost models. Supports optional sample weighting for training and validation data.

```python
fit_params = {
    "X": X_train,
    "y": y_train,
    "eval_set": [(X_val, y_val)],
}
if sample_weight is not None:
    fit_params["sample_weight"] = sample_weight.iloc[train_index]
    fit_params["eval_sample_weight"] = [sample_weight.iloc[val_index]]

return fit_params
```

--------------------------------

### Prepare Data and Train Model

Source: https://ing-bank.github.io/probatus/tutorials/nb_shap_model_interpreter.html

Sets up a synthetic dataset and trains a RandomForestClassifier model. Ensure to use 'class_weight="balanced"' or an equivalent for imbalanced datasets.

```python
import warnings

import pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from probatus.interpret import ShapModelInterpreter

warnings.filterwarnings("ignore")

feature_names = ["f1", "f2", "f3", "f4"]

# Prepare two samples
X, y = make_classification(n_samples=1000, n_features=4, random_state=0)
X = pd.DataFrame(X, columns=feature_names)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Prepare and fit model. Remember about class_weight="balanced" or an equivalent.
model = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0)
model = model.fit(X_train, y_train)

```

--------------------------------

### Initialize and Compute SHAPImportanceResemblance

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Demonstrates how to initialize SHAPImportanceResemblance with a RandomForestClassifier, compute feature importance for two datasets, and plot the results. Ensure the model is tree-based.

```python
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from probatus.sample_similarity import SHAPImportanceResemblance
X1, _ = make_classification(n_samples=100, n_features=5)
X2, _ = make_classification(n_samples=100, n_features=5, shift=0.5)
model = RandomForestClassifier(max_depth=2)
rm = SHAPImportanceResemblance(model)
feature_importance = rm.fit_compute(X1, X2)
rm.plot()
```

--------------------------------

### Get Best Parsimonious Features

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Selects the fewest number of features within a threshold of the highest score. Useful when model interpretability and parsimony are prioritized.

```python
highest_score = shap_report["val_metric_mean"].max()
within_threshold = shap_report[shap_report["val_metric_mean"] >= highest_score - standard_error_threshold]
fewest_features_index = within_threshold["num_features"].idxmin()
best_num_features = within_threshold.loc[fewest_features_index, "num_features"]
```

--------------------------------

### Initialize ShapDependencePlotter

Source: https://ing-bank.github.io/probatus/api/model_interpret.html

Initializes the ShapDependencePlotter with a model and optional verbosity and random state settings. Use verbose=0 for no output, 1 for important warnings, and 2 for all output. Set random_state for reproducible results.

```python
def __init__(self, model, verbose=0, random_state=None):
    """
    Initializes the class.

    Args:
        model (model object):
            regression or classification model or pipeline.

        verbose (int, optional):
            Controls verbosity of the output:

            - 0 - neither prints nor warnings are shown
            - 1 - only most important warnings
            - 2 - shows all prints and all warnings.

        random_state (int, optional):
            Random state set for the nr of samples. If it is None, the results will not be reproducible. For
            reproducible results set it to an integer.
    """
    self.model = model
    self.verbose = verbose
    self.random_state = random_state
```

--------------------------------

### Get Feature Support Mask

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Generates a boolean mask indicating which features were selected by the RFECV method. This is useful for filtering datasets or understanding feature importance.

```python
support = [True if col in feature_names_selected else False for col in self.column_names]

return support
```

--------------------------------

### Get Single Scorer Utility

Source: https://ing-bank.github.io/probatus/api/utils.html

Retrieves a Scorer object based on the provided input. It accepts either a string metric name or an existing Scorer object.

```python
def get_single_scorer(scoring):
    """
    Returns Scorer, based on provided input in scoring argument.

    Args:
        scoring (string or probatus.utils.Scorer, optional):
            Metric for which the model performance is calculated. It can be either a metric name aligned with
            predefined classification scorers names in sklearn
            ([link](https://scikit-learn.org/stable/modules/model_evaluation.html)).
            Another option is using probatus.utils.Scorer to define a custom metric.

    Returns:
        (probatus.utils.Scorer):
            Scorer that can be used for scoring models
    """
    if isinstance(scoring, str):
        return Scorer(scoring)
    elif isinstance(scoring, Scorer):
        return scoring
    else:
        raise (ValueError("The scoring should contain either strings or probatus.utils.Scorer class"))
```

--------------------------------

### Get Feature Ranking

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Returns a list representing the ranking of features. Selected features (estimated best) are ranked 1, while eliminated features are ranked based on their importance.

```python
flipped_report_df = self.report_df.iloc[::-1]

# Some features are not eliminated. All have importance of zero (highest importance)
features_not_eliminated = flipped_report_df["features_set"].iloc[0]
features_not_eliminated_dict = {v: 0 for v in features_not_eliminated}

# Eliminated features are ranked by shap importance
features_eliminated = np.concatenate(flipped_report_df["eliminated_features"].to_numpy())
features_eliminated_dict = {int(v): k + 1 for (k, v) in enumerate(features_eliminated)}

# Combine dicts with rank info
features_eliminated_dict.update(features_not_eliminated_dict)

# Get ranking per the order of columns
ranking = [features_eliminated_dict[col] for col in self.column_names]

return ranking
```

--------------------------------

### Get Reduced Feature Set

Source: https://ing-bank.github.io/probatus/tutorials/nb_automatic_best_num_features.html

Retrieve the names of the features in the best-performing feature set identified by ShapRFECV. Specify the desired number of features to obtain the corresponding set.

```python
# Once we have identified which iteration was best, we can get the feature names by:
shap_elimination.get_reduced_features_set(num_features=21)
```

--------------------------------

### Prepare SHAP Variables

Source: https://ing-bank.github.io/probatus/api/model_interpret.html

Prepares SHAP values, expected values, and a fitted DependencePlotter for a given dataset. Handles SHAP calculation and extracts necessary components for model interpretation. Supports approximate SHAP calculation and returns the explainer object.

```python
@staticmethod
def _prep_shap_related_variables(
        model,
        X,
        y,
        approximate=False,
        verbose=0,
        random_state=None,
        column_names=None,
        class_names=None,
        **shap_kwargs,
    ):
        """
        The function prepares the variables related to shap that are used to interpret the model.

        Returns:
            (np.array, int, DependencePlotter):
                Shap values, expected value of the explainer, and fitted TreeDependencePlotter for a given dataset.
        """
        shap_values, explainer = shap_calc(
            model,
            X,
            approximate=approximate,
            verbose=verbose,
            random_state=random_state,
            return_explainer=True,
            **shap_kwargs,
        )

        expected_value = explainer.expected_value

        # For sklearn models the expected values consists of two elements (negative_class and positive_class)
        if isinstance(expected_value, list) or isinstance(expected_value, np.ndarray):
            expected_value = expected_value[1]

        # Initialize tree dependence plotter

```

--------------------------------

### Tune Best Parsimonious with Standard Error Threshold

Source: https://ing-bank.github.io/probatus/tutorials/nb_automatic_best_num_features.html

Adjust the behavior of 'best_coherent' and 'best_parsimonious' strategies by setting the `standard_error_threshold` parameter. This example demonstrates tuning 'best_parsimonious' with a threshold of 0.5.

```python
# Note, you can change the behavior of `best_coherent` and `best_parsimonious` by changing the `standard_error_threshold` parameter
# Best coherent (standard_error_threshold=0.5)
best_features = shap_elimination.get_reduced_features_set(
    num_features="best_parsimonious", standard_error_threshold=0.5
)
print(f"The {len(best_features)} best coherent features are: {best_features}")
```

--------------------------------

### Initialize Resemblance Model

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Initializes the Resemblance Model with parameters for training iterations, scoring metric, test data percentage, parallel jobs, verbosity, and random state for reproducibility.

```python
def __init__(
    self,
    model,
    iterations=100,
    scoring="roc_auc",
    test_prc=0.25,
    n_jobs=1,
    verbose=0,
    random_state=None,
):
    """
    Initializes the class.

    Args:
        model (model object):
            Regression or classification model or pipeline.

        iterations (int, optional):
            Number of iterations performed to calculate permutation importance. By default 100 iterations per
            feature are done.

        scoring (string or probatus.utils.Scorer, optional):
            Metric for which the model performance is calculated. It can be either a metric name aligned with
            predefined
            [classification scorers names in sklearn](https://scikit-learn.org/stable/modules/model_evaluation.html).
            Another option is using probatus.utils.Scorer to define a custom metric. Recommended option for this
            class is 'roc_auc'.

        test_prc (float, optional):
            Percentage of data used to test the model. By default 0.25 is set.

        n_jobs (int, optional):
            Number of parallel executions. If -1 use all available cores. By default 1.

        verbose (int, optional):
            Controls verbosity of the output:

            - 0 - neither prints nor warnings are shown
            - 1 - only most important warnings
            - 2 - shows all prints and all warnings.

        random_state (int, optional):
            Random state set at each round of feature elimination. If it is None, the results will not be
            reproducible and in random search at each iteration a different hyperparameters might be tested. For
            reproducible results set it to integer.
    """  # noqa
    super().__init__(
        model=model,
        scoring=scoring,
        test_prc=test_prc,
        n_jobs=n_jobs,
        verbose=verbose,
        random_state=random_state,
    )

    self.iterations = iterations

    self.iterations_columns = ["feature", "importance"]
    self.iterations_results = pd.DataFrame(columns=self.iterations_columns)

    self.plot_x_label = "Permutation Feature Importance"
    self.plot_y_label = "Feature Name"
    self.plot_title = "Permutation Feature Importance of Resemblance Model"

```

--------------------------------

### Get Feature Names by Number

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

Retrieves the list of feature names corresponding to a specific number of features. Raises a ValueError if the requested number of features was not achieved during the fitting process.

```python
matching_rows = self.report_df[self.report_df.num_features == num_features]

if matching_rows.empty:
    valid_nums = ", ".join([str(n) for n in sorted(self.report_df.num_features.unique())])
    raise ValueError(
        f"The provided number of features has not been achieved at any stage of the process. "
        f"You can select one of the following: {valid_nums}"
    )

return matching_rows.iloc[0]["features_set"]
```

--------------------------------

### BaseResemblanceModel Initialization

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Initializes the BaseResemblanceModel with a given model, scoring metric, test percentage, and other configuration options.

```APIDOC
## BaseResemblanceModel

### Description
This model checks for the similarity of two samples. A possible use case is analysis of whether the train sample differs from the test sample, due to e.g. non-stationarity.

This is a base class and needs to be extended by a fit() method, which implements how the data is split, how the model is trained and evaluated. Further, inheriting classes need to implement how feature importance should be indicated.

### Method
__init__

### Parameters
#### Arguments
- **model** (model object) - The regression or classification model or pipeline to be used.
- **scoring** (string or probatus.utils.Scorer, optional) - Metric for which the model performance is calculated. Defaults to 'roc_auc'.
- **test_prc** (float, optional) - Percentage of data used to test the model. Defaults to 0.25.
- **n_jobs** (int, optional) - Number of parallel executions. If -1 use all available cores. Defaults to 1.
- **verbose** (int, optional) - Controls verbosity of the output. Defaults to 0.
- **random_state** (int, optional) - Random state set at each round of feature elimination. Defaults to None.
```

--------------------------------

### Get SHAP Values

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

Retrieves the SHAP values that were computed on the test set during the fitting process. These values indicate the contribution of each feature to the model's predictions for the test data.

```python
def get_shap_values(self):
    """
    Gets the SHAP values generated on the test set.

    Returns:
         (np.array):
            SHAP values generated on the test set.
    """
    self._check_if_fitted()
    return self.shap_values_test
```

--------------------------------

### fit(X1, X2, column_names=None, class_names=None)

Source: https://ing-bank.github.io/probatus/api/sample_similarity.html

The `fit` method is the base functionality for comparing two samples. It preprocesses the input data, splits it into training and testing sets, trains a model, and evaluates its performance on both sets.

```APIDOC
## fit(X1, X2, column_names=None, class_names=None)

### Description
Base fit functionality that should be executed before each fit. It preprocesses the input data, splits it into training and testing sets, trains a model, and evaluates its performance on both sets.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **X1** (ndarray or DataFrame) - Required - First sample to be compared. It needs to have the same number of columns as X2.
- **X2** (ndarray or DataFrame) - Required - Second sample to be compared. It needs to have the same number of columns as X1.
- **column_names** (list of str) - Optional - List of feature names of the provided samples. If provided it will be used to overwrite the existing feature names. If not provided the existing feature names are used or default feature names are generated.
- **class_names** (None, or list of str) - Optional - List of class names assigned, in this case provided samples e.g. ['sample1', 'sample2']. If none, the default ['First Sample', 'Second Sample'] are used.

### Returns
- **BaseResemblanceModel** - Fitted object

### Source Code
```python
def fit(self, X1, X2, column_names=None, class_names=None):
    """
    Base fit functionality that should be executed before each fit.

    Args:
        X1 (np.ndarray or pd.DataFrame):
            First sample to be compared. It needs to have the same number of columns as X2.

        X2 (np.ndarray or pd.DataFrame):
            Second sample to be compared. It needs to have the same number of columns as X1.

        column_names (list of str, optional):
            List of feature names of the provided samples. If provided it will be used to overwrite the existing
            feature names. If not provided the existing feature names are used or default feature names are
            generated.

        class_names (None, or list of str, optional):
            List of class names assigned, in this case provided samples e.g. ['sample1', 'sample2']. If none,
            the default ['First Sample', 'Second Sample'] are used.

    Returns:
        (BaseResemblanceModel):
            Fitted object
    """
    # Set class names
    self.class_names = class_names
    if self.class_names is None:
        self.class_names = ["First Sample", "Second Sample"]

    # Ensure inputs are correct
    self.X1, self.column_names = preprocess_data(X1, X_name="X1", column_names=column_names, verbose=self.verbose)
    self.X2, _ = preprocess_data(X2, X_name="X2", column_names=column_names, verbose=self.verbose)

    # Prepare dataset for modelling
    self.X = pd.DataFrame(pd.concat([self.X1, self.X2], axis=0), columns=self.column_names).reset_index(drop=True)

    self.y = pd.Series(
        np.concatenate(
            [
                np.zeros(self.X1.shape[0]),
                np.ones(self.X2.shape[0]),
            ]
        )
    ).reset_index(drop=True)

    # Assure the type and number of classes for the variable
    self.X, _ = preprocess_data(self.X, X_name="X", column_names=self.column_names, verbose=self.verbose)

    self.y = preprocess_labels(self.y, y_name="y", index=self.X.index, verbose=self.verbose)

    # Reinitialize variables in case of multiple times being fit
    self._init_output_variables()

    self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(
        self.X,
        self.y,
        test_size=self.test_prc,
        random_state=self.random_state,
        shuffle=True,
        stratify=self.y,
    )
    self.model.fit(self.X_train, self.y_train)

    self.train_score = np.round(self.scorer.score(self.model, self.X_train, self.y_train), 3)
    self.test_score = np.round(self.scorer.score(self.model, self.X_test, self.y_test), 3)

    self.results_text = (
        f"Train {self.scorer.metric_name}: {np.round(self.train_score, 3)},
"
        f"Test {self.scorer.metric_name}: {np.round(self.test_score, 3)}."
    )
    if self.verbose > 1:
        logger.info(f"Finished model training: \n{self.results_text}")

    if self.verbose > 0:
        if self.train_score > self.test_score:
            warnings.warn(
                f"Train {self.scorer.metric_name} > Test {self.scorer.metric_name}, which might indicate "
                f"an overfit. \n Strong overfit might lead to misleading conclusions when analysing "
                f"feature importance. Consider retraining with more regularization applied to the model."
            )
    self.fitted = True
    return self
```
```

--------------------------------

### Run Feature Elimination with ShapRFECV

Source: https://ing-bank.github.io/probatus/api/feature_elimination.html

This snippet demonstrates how to initialize and run ShapRFECV for feature elimination. It prepares sample data, a model, and a parameter grid, then fits the elimination process and generates a performance plot.

```python
feature_names = [
        'f0', 'f1', 'f2', 'f3', 'f4', 'f5', 'f6', 'f7', 
        'f8', 'f9', 'f10', 'f11', 'f12', 'f13',
        'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f20']

    # Prepare two samples
    X, y = make_classification(n_samples=200, class_sep=0.05, n_informative=6, n_features=20,
                               random_state=0, n_redundant=10, n_clusters_per_class=1)
    X = pd.DataFrame(X, columns=feature_names)


    # Prepare model and parameter search space
    model = RandomForestClassifier(max_depth=5, class_weight='balanced')

    param_grid = {
        'n_estimators': [5, 7, 10],
        'min_samples_leaf': [3, 5, 7, 10],
    }
    search = RandomizedSearchCV(model, param_grid)


    # Run feature elimination
    shap_elimination = ShapRFECV(
        model=search, step=0.2, cv=10, scoring='roc_auc', n_jobs=3)
    report = shap_elimination.fit_compute(X, y)

    # Make plots
    performance_plot = shap_elimination.plot()

    # Get final feature set
    final_features_set = shap_elimination.get_reduced_features_set(num_features=3)
```

--------------------------------

### Prepare Data with Groups

Source: https://ing-bank.github.io/probatus/howto/grouped_data.html

Generates a classification dataset and creates group identifiers for each sample. Ensure all samples from a group are kept together.

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100, n_features=10, random_state=42)
groups = [i % 5 for i in range(100)]
groups[:10]
```
Tokens:
Raw