### Setup Development Environment

Source: https://github.com/seldonio/alibi/blob/master/CONTRIBUTING.md

Commands to clone the repository, create a virtual environment, and install all necessary dependencies for development, documentation, and testing.

```bash
git clone git@github.com:SeldonIO/alibi.git
cd alibi
pip install -e .[all]
pip install -r requirements/dev.txt -r requirements/docs.txt
```

--------------------------------

### Install and Verify Package

Source: https://github.com/seldonio/alibi/wiki/Release-process

Installs the package from Test PyPI into a clean environment to verify installation integrity.

```bash
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple 'alibi[all]' --no-cache-dir
```

--------------------------------

### Counterfactual with Reinforcement Learning - Setup

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/cfrl_adult.ipynb

This section covers the initial setup for generating counterfactuals, including defining constants, dataset attributes, and constraints.

```APIDOC
## Counterfactual with Reinforcement Learning - Setup

### Description
This section covers the initial setup for generating counterfactuals, including defining constants, dataset attributes, and constraints.

### Constants

```python
# Define constants
COEFF_SPARSITY = 0.5               # sparisty coefficient
COEFF_CONSISTENCY = 0.5            # consisteny coefficient
TRAIN_STEPS = 10000                # number of training steps -> consider increasing the number of steps
BATCH_SIZE = 100                   # batch size
```

### Dataset Specific Attributes and Constraints

A desirable property of a method for generating counterfactuals is to allow feature conditioning. Real-world datasets usually include immutable features such as `Sex` or `Race`, which should remain unchanged throughout the counterfactual search procedure. Similarly, a numerical feature such as `Age` should only increase for a counterfactual to be actionable.

#### Define immutable features.

```python
immutable_features = ['Marital Status', 'Relationship', 'Race', 'Sex']
```

#### Define ranges for features.

This means that the `Age` feature can not decrease.

```python
ranges = {'Age': [0.0, 1.0]}
```
```

--------------------------------

### Install XGBoost

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/interventional_tree_shap_adult_xgb.ipynb

Installs the XGBoost library, a popular gradient boosting framework. This is a prerequisite for training and explaining XGBoost models.

```bash
!pip install -q xgboost
```

--------------------------------

### Initialize Kernel SHAP explainer

Source: https://github.com/seldonio/alibi/blob/master/doc/source/methods/KernelSHAP.ipynb

Example showing how to import the KernelShap class and initialize it with a prediction function for a classifier.

```python
from alibi.explainers import KernelShap

predict_fn = lambda x: clf.predict_proba(x)
```

--------------------------------

### Install Alibi with Ray support

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/distributed_kernel_shap_adult_lr.md

Installs the Alibi library with Ray support, which is required for distributed KernelSHAP explanations. Note that Ray support is in beta on Windows.

```bash
pip install alibi[ray]
```

--------------------------------

### Interventional TreeShap Setup

Source: https://github.com/seldonio/alibi/blob/master/doc/source/methods/TreeSHAP.ipynb

Initializes the TreeShap explainer for interventional perturbation, requiring a reference dataset for the fit method.

```python
explainer = TreeShap(model, model_output='raw')
explainer.fit(X_reference)
explanation = explainer.explain(X)
```

--------------------------------

### Alibi - Distributed Explainer Configuration

Source: https://github.com/seldonio/alibi/blob/master/doc/source/overview/getting_started.md

Illustrates how to configure an explainer for distributed execution, using KernelShap with Ray as an example.

```APIDOC
## Distributed Explainer Configuration

### Description
This section shows how to initialize an Alibi explainer for parallel processing using distributed backends like Ray. This is useful for accelerating explanations on large datasets or complex models.

### Method
Python

### Endpoint
N/A (Python Library Usage)

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
```python
# Ensure alibi[ray] is installed: pip install alibi[ray]
from alibi.explainers import KernelShap

# Assume predict_fn is defined

# Initialize KernelShap explainer for distributed execution
distributed_ks = KernelShap(predict_fn, distributed_opts={'n_cpus': 10})
```

### Response
#### Success Response (Initialized Explainer)
- **distributed_ks** (KernelShap object) - An instance of the KernelShap explainer configured for distributed computation.

#### Response Example
```python
# The object itself is the result, no direct JSON output for initialization.
# Example of how it might be represented internally or during debugging:
KernelShap(meta={
    'name': 'KernelShap',
    'type': ['blackbox'],
    'explanations': ['local'],
    'params': {
        'seed': None,
        'distributed_opts': {'n_cpus': 10}
    }
})
```

### Error Handling
- Requires `alibi[ray]` to be installed for distributed options.
- Ensure Ray is properly configured in the environment.
```

--------------------------------

### Install XGBoost and Matplotlib

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/path_dependent_tree_shap_adult_xgb.ipynb

Installs specific versions of XGBoost and Matplotlib required for the example. XGBoost is used for the model, and Matplotlib for plotting.

```bash
!pip install matplotlib==3.5.3
!pip install "xgboost<2.0.0"
```

--------------------------------

### Setup and Imports for Alibi Counterfactuals

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/cfproto_housing.md

Initializes TensorFlow and Alibi environment, imports necessary libraries, and configures TensorFlow logging and behavior for compatibility with Alibi's TF1 constructs. This setup is crucial for running the subsequent counterfactual explanation examples.

```python
import matplotlib.pyplot as plt

import tensorflow as tf
tf.get_logger().setLevel(40) # suppress deprecation messages
tf.compat.v1.disable_v2_behavior() # disable TF2 behaviour as alibi code still relies on TF1 constructs 
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.utils import to_categorical

import os
import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from alibi.explainers import CounterfactualProto

print('TF version: ', tf.__version__)
print('Eager execution enabled: ', tf.executing_eagerly())
```

--------------------------------

### Initialize and Run Explainer with Callbacks

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/cfrl_mnist.md

Demonstrates how to initialize the wandb project, instantiate the CounterfactualRL explainer with custom callbacks, and execute the fitting process.

```python
import wandb

wandb.init(project="MNIST Counterfactual with Reinforcement Learning")

explainer = CounterfactualRL(..., callbacks=[RewardCallback(), ImagesCallback()])
explainer.fit(X=X_train)

wandb.finish()
```

--------------------------------

### Initialize and Run Explainer with Logging Callbacks

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/cfrl_adult.md

This snippet shows how to initialize a Weights & Biases project, instantiate the CounterfactualRLTabular explainer with custom callbacks, and execute the fitting process.

```python
import wandb
wandb_project = "Adult Census Counterfactual with Reinforcement Learning"
wandb.init(project=wandb_project)
explainer = CounterfactualRLTabular(..., callbacks=[LossCallback(), RewardCallback(), TablesCallback()])
explainer = explainer.fit(X=X_train)
wandb.finish()
```

--------------------------------

### Initialize and Execute Sequential KernelSHAP

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/distributed_kernel_shap_adult_lr.md

Demonstrates how to instantiate a KernelSHAP explainer, fit it to background data, and generate explanations sequentially.

```python
seq_lr_explainer = KernelShap(classifier.predict_proba, link='logit', feature_names=perm_feat_names)
seq_lr_explainer.fit(X_train_proc_d[background_data, :], group_names=group_names, groups=groups)
explanation = seq_lr_explainer.explain(sparse2ndarray(X_explain_proc))
```

--------------------------------

### Generate Counterfactual Instances using Reinforcement Learning

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/cfrl_adult.ipynb

This snippet demonstrates the process of generating counterfactual instances. It involves selecting positive examples from the test set, defining target labels and constraints, and then using the fitted explainer to generate counterfactuals based on these inputs. The constraints guide the search for realistic and actionable counterfactuals.

```python
# Select some positive examples.
X_positive = X_test[np.argmax(predictor(X_test), axis=1) == 1]

X = X_positive[:1000]
Y_t = np.array([0])
C = [{"Age": [0, 20], "Workclass": ["State-gov", "?", "Local-gov"]}]
```

```python
# Generate counterfactual instances.
explanation = explainer.explain(X, Y_t, C)
```

--------------------------------

### Initialize Environment and Imports

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/trustscore_mnist.ipynb

Sets up the necessary environment variables, imports deep learning and data processing libraries, and initializes the Alibi TrustScore module.

```python
import os
os.environ["TF_USE_LEGACY_KERAS"] = "1"
import tensorflow as tf
from tensorflow.keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPooling2D, Input, UpSampling2D
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.utils import to_categorical
import matplotlib
%matplotlib inline
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit
from alibi.confidence import TrustScore
```

--------------------------------

### Install Alibi via Mamba

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/overview/getting_started.md

Commands to install Alibi using mamba and conda-forge. This includes setting up the environment and installing specific feature sets.

```bash
conda install mamba -n base -c conda-forge
mamba install -c conda-forge alibi
mamba install -c conda-forge alibi shap
mamba install -c conda-forge alibi ray
```

--------------------------------

### KernelSHAP Initialization and Sequential Explanation

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/distributed_kernel_shap_adult_lr.md

Demonstrates how to initialize the KernelSHAP explainer, fit it with training data, and perform sequential explanations.

```APIDOC
## KernelSHAP Initialization and Sequential Explanation

### Description
This section covers the initialization of the KernelSHAP explainer, including setting up feature names, groups, and then fitting the explainer with processed data. It also shows how to run explanations sequentially and time the execution.

### Method
```python
# Assuming necessary imports and data loading are done
# pred_fcn: prediction function of the model
# X_train_proc_d: processed training data (sparse matrix)
# background_data: slice for background data
# perm_feat_names: list of feature names
# num_feats_names: list of numerical feature names
# cat_feats_names: list of categorical feature names
# feat_enc_dim: list of encoding dimensions for categorical features
# X_explain_proc: processed data to explain (sparse matrix)

def make_groups(num_feats_names: List[str], cat_feats_names: List[str], feat_enc_dim: List[int]) -> Tuple[List[str], List[List[int]]]:
    """
    Given a list with numerical feat. names, categorical feat. names
    and a list specifying the lengths of the encoding for each cat.
    varible, the function outputs a list of group names, and a list
    of the same len where each entry represents the column indices that
    the corresponding categorical feature
    """
    group_names = num_feats_names + cat_feats_names
    groups = []
    cat_var_idx = 0

    for name in group_names:
        if name in num_feats_names:
            groups.append(list(range(len(groups), len(groups) + 1)))
        else:
            start_idx = groups[-1][-1] + 1 if groups else 0
            groups.append(list(range(start_idx, start_idx + feat_enc_dim[cat_var_idx])))
            cat_var_idx += 1
    return group_names, groups

def sparse2ndarray(mat, examples=None):
    """
    Converts a scipy.sparse.csr.csr_matrix to a numpy.ndarray.
    If specified, examples is slice object specifying which selects a
    number of rows from mat and converts only the respective slice.
    """
    if examples:
        return mat[examples, :].toarray()
    return mat.toarray()

# Initialize explainer
pred_fcn = classifier.predict_probase
seq_lr_explainer = KernelShap(pred_fcn, link='logit', feature_names=perm_feat_names)

# Prepare groups and background data
group_names, groups = make_groups(num_feats_names, cat_feats_names, feat_enc_dim)
X_train_proc_d = sparse2ndarray(X_train_proc, examples=background_data)

# Fit the explainer
seq_lr_explainer.fit(X_train_proc_d, group_names=group_names, groups=groups)

# Perform sequential explanations
n_runs = 3
s_explanations, s_times = [], []
for run in range(n_runs):
    t_start = timer()
    explanation = seq_lr_explainer.explain(sparse2ndarray(X_explain_proc))
    t_elapsed = timer() - t_start
    s_times.append(t_elapsed)
    s_explanations.append(explanation.shap_values)

print(f"Sequential average time for {n_runs} runs:")
print(np.round(np.mean(s_times), 3), "s")
```

### Parameters
N/A for this setup, parameters are passed during initialization and fitting.

### Request Example
N/A

### Response
#### Success Response (200)
Returns SHAP values and timing information.

#### Response Example
```json
{
  "shap_values": [[...], [...]],
  "time_elapsed": 119.656
}
```
```

--------------------------------

### Preparing Data and Instantiating KernelShap Explainer

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/kernel_shap_adult_lr.ipynb

Demonstrates how to use the utility functions to prepare the training data and group information, then instantiates the KernelShap explainer with the prepared data.

```python
X_train_proc_d = sparse2ndarray(X_train_proc, examples=background_data)
group_names, groups = make_groups(num_feats_names, cat_feats_names, feat_enc_dim)
```

```python
X_explain_proc_d = sparse2ndarray(X_explain_proc)
grp_lr_explainer = KernelShap(pred_fcn, link='logit', feature_names=perm_feat_names)
grp_lr_explainer.fit(X_train_proc_d, group_names=group_names, groups=groups)
```

--------------------------------

### Install XGBoost

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/linearity_measure_iris.ipynb

Installs the xgboost library using pip. This is a prerequisite for using XGBoost models.

```python
!pip install xgboost
```

--------------------------------

### Alibi Installation with Conda

Source: https://github.com/seldonio/alibi/blob/master/README.md

Instructions for installing Alibi and its optional dependencies using conda and mamba.

```APIDOC
## Installation with Conda

### Description
Install Alibi and its dependencies using conda-forge and mamba for optimal performance.

### Method
Bash commands for installing Alibi.

### Endpoint
N/A

### Parameters
N/A

### Request Example
```bash
# Install mamba to the base conda environment
conda install mamba -n base -c conda-forge

# Standard Alibi install
mamba install -c conda-forge alibi

# For distributed computing support
mamba install -c conda-forge alibi ray

# For SHAP support
mamba install -c conda-forge alibi shap
```

### Response
N/A
```

--------------------------------

### Initialize Environment and Load Data

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/cfproto_cat_adult_ohe.ipynb

Set up the TensorFlow environment and load the adult dataset for counterfactual analysis.

```python
import os
os.environ["TF_USE_LEGACY_KERAS"] = "1"
import tensorflow as tf
tf.compat.v1.disable_v2_behavior()
from alibi.datasets import fetch_adult

adult = fetch_adult()
data, target = adult.data, adult.target
```

--------------------------------

### Install Seaborn and Alibi

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/overview.ipynb

Installs the seaborn library for visualization and the Alibi library with all dependencies using pip.

```bash
pip install -q seaborn
pip install alibi[all]
```

--------------------------------

### Setup and Model Loading for ImageNet Explanations

Source: https://github.com/seldonio/alibi/blob/master/examples/anchor_image_imagenet.ipynb

This snippet sets up the environment by importing necessary libraries and loading the InceptionV3 model pre-trained on ImageNet. It configures TensorFlow to use legacy Keras and prepares for image analysis.

```python
import os
os.environ["TF_USE_LEGACY_KERAS"] = "1"

import numpy as np
import tensorflow as tf
from tensorflow.keras.applications.inception_v3 import InceptionV3, preprocess_input, decode_predictions

import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt

from alibi.datasets import load_cats
from alibi.explainers import AnchorImage

model = InceptionV3(weights='imagenet')
```

--------------------------------

### Install Sentence Transformers Dependency

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/similarity_explanations_20ng.ipynb

Installs the sentence-transformers library required for embedding generation in Alibi explainers.

```bash
!pip install sentence_transformers
```

--------------------------------

### Initialize ProtoSelect Environment

Source: https://github.com/seldonio/alibi/blob/master/examples/protoselect_adult_cifar10.ipynb

Imports necessary libraries for data processing, model training, and prototype selection using the Alibi framework.

```python
import os
os.environ["TF_USE_LEGACY_KERAS"] = "1"
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from typing import List, Dict
import tensorflow as tf
import tensorflow.keras as keras
from alibi.prototypes import ProtoSelect, visualize_image_prototypes
from alibi.utils.kernel import EuclideanDistance
```

--------------------------------

### Install Seaborn Dependency

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/permutation_importance_classification_leave.ipynb

Installs the seaborn visualization library required for plotting the feature importance results.

```python
!pip install -q seaborn
```

--------------------------------

### Install Alibi and Dependencies

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/cem_iris.ipynb

Commands to install the necessary Alibi library with TensorFlow support and the seaborn visualization library.

```bash
pip install alibi[tensorflow]
pip install seaborn
```

--------------------------------

### Instantiate and Summarise with ProtoSelect

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/protoselect_adult_cifar10.ipynb

Instantiates the ProtoSelect summariser with the optimal epsilon value found and the preprocessor. It then fits the summariser to the training data and generates a summary, printing the number of prototypes found.

```python
summariser = ProtoSelect(kernel_distance=EuclideanDistance(), 
                         eps=cv['best_eps'], 
                         preprocess_fn=preprocessor.transform)

summariser = summariser.fit(X=X_train, y=y_train)
summary = summariser.summarise(num_prototypes=num_prototypes)
print(f"Found {len(summary.data['prototypes'])} prototypes.")
```

--------------------------------

### Instantiate and Fit ProtoSelect

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/protoselect_adult_cifar10.ipynb

This snippet shows how to instantiate the ProtoSelect class with the optimal epsilon found via cross-validation and then fit it to the training data. The `summarise` method is called to obtain the prototypes.

```python
summariser = ProtoSelect(kernel_distance=EuclideanDistance(), 
                         eps=cv['best_eps'], 
                         preprocess_fn=preprocess_fn)

summariser = summariser.fit(X=X_train, y=y_train)
summary = summariser.summarise(num_prototypes=num_prototypes)
```

--------------------------------

### Install Dependencies

Source: https://github.com/seldonio/alibi/blob/master/examples/interventional_tree_shap_adult_xgb.ipynb

Installs necessary libraries including matplotlib and xgboost. It's recommended to use specific versions for reproducibility.

```bash
!pip install matplotlib==3.5.3
!pip install -q xgboost
```

--------------------------------

### Configure and Initialize AnchorImage Explainer

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/anchor_image_imagenet.ipynb

Sets up the segmentation function and initializes the AnchorImage explainer with specific superpixel parameters.

```python
segmentation_fn = 'slic'
kwargs = {'n_segments': 15, 'compactness': 20, 'sigma': .5, 'start_label': 0}
explainer = AnchorImage(predict_fn, image_shape, segmentation_fn=segmentation_fn, segmentation_kwargs=kwargs, images_background=None)
```

--------------------------------

### Initialize and use AnchorImage explainer

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/anchor_image_fashion_mnist.ipynb

Demonstrates how to wrap a prediction function, initialize the AnchorImage explainer with a segmentation function, and generate an explanation mask for a specific image prediction.

```python
predict_fn = lambda x: cnn.predict(x)
image_shape = x_train[idx].shape
explainer = AnchorImage(predict_fn, image_shape, segmentation_fn=superpixel)
explanation = explainer.explain(image, threshold=.95, p_sample=.8, seed=0)
```

--------------------------------

### Install Alibi SHAP dependencies

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/kernel_shap_adult_categorical_preproc.ipynb

Command to install the necessary dependencies for using SHAP explainers within the Alibi library.

```bash
pip install alibi[shap]
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/seldonio/alibi/blob/master/CONTRIBUTING.md

Installs Git pre-commit hooks to automatically run linting and type checking tools before every commit.

```bash
pre-commit install
```

--------------------------------

### POST /explainer/cem/initialize

Source: https://github.com/seldonio/alibi/blob/master/doc/source/methods/CEM.ipynb

Initializes the Contrastive Explanations Method (CEM) explainer using a black-box prediction function and numerical gradient configuration.

```APIDOC
## POST /explainer/cem/initialize

### Description
Initializes the CEM explainer instance. This is used when the model architecture is not directly accessible, requiring the use of a provided prediction function and numerical gradient estimation.

### Method
POST

### Endpoint
/explainer/cem/initialize

### Parameters
#### Request Body
- **predict_fn** (function) - Required - A function that accepts input data and returns class probabilities.
- **mode** (string) - Required - The explanation mode (e.g., 'PP' for Pertinent Positive).
- **shape** (tuple) - Required - The shape of the input data.
- **kappa** (float) - Optional - Confidence threshold parameter.
- **beta** (float) - Optional - L1 regularization parameter.
- **feature_range** (tuple) - Optional - The min and max values for input features.
- **eps** (tuple) - Optional - Perturbation sizes for numerical gradient calculation (eps[0] for loss/pred, eps[1] for pred/input).
- **update_num_grad** (int) - Optional - Batch size for evaluating numerical gradients to optimize performance.

### Request Example
{
  "predict_fn": "lambda x: model.predict(x)",
  "mode": "PP",
  "shape": [1, 4],
  "kappa": 0.0,
  "beta": 0.1,
  "eps": [[0.01, 0.01, 0.01], [0.01, 0.01, 0.01, 0.01]],
  "update_num_grad": 100
}

### Response
#### Success Response (200)
- **status** (string) - Initialization status.
- **explainer_id** (string) - Unique identifier for the initialized CEM instance.

#### Response Example
{
  "status": "success",
  "explainer_id": "cem_001"
}
```

--------------------------------

### Set Number of Covered Examples with Alibi

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/api/alibi/explainers/anchors/anchor_tabular.md

Sets the number of examples to be saved or considered, likely for tracking or limiting the scope of explanations or analyses.

```python
def set_n_covered(n_covered: int) -> None:
    """Sets the number of examples to be saved.

    Args:
        n_covered: The number of examples to save.
    """
    pass
```

--------------------------------

### Initialize and prepare environment

Source: https://github.com/seldonio/alibi/blob/master/examples/cfproto_housing.ipynb

Imports necessary libraries and configures TensorFlow for compatibility with Alibi's requirements.

```python
import os
os.environ["TF_USE_LEGACY_KERAS"] = "1"
import tensorflow as tf
tf.compat.v1.disable_v2_behavior()
from alibi.explainers import CounterfactualProto
```

--------------------------------

### Install Specific Matplotlib Version

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/kernel_shap_adult_lr.ipynb

Installs a specific version of Matplotlib (3.5.3) to work around a known compatibility issue with shap.summary_plot in newer versions (>=3.6.0).

```bash
# shap.summary_plot currently doesn't work with matplotlib>=3.6.0,
# see bug report: https://github.com/slundberg/shap/issues/2687
!pip install matplotlib==3.5.3
```

--------------------------------

### Initialize Environment and Imports

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/integrated_gradients_imdb.md

Imports required libraries including TensorFlow, Keras layers, and Alibi explainers to prepare for model training and explanation.

```python
import tensorflow as tf
import numpy as np
import os
import pandas as pd
from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, Embedding, Conv1D, GlobalMaxPooling1D, Dropout 
from tensorflow.keras.utils import to_categorical
from alibi.explainers import IntegratedGradients
import matplotlib.pyplot as plt
print('TF version: ', tf.__version__)
print('Eager execution enabled: ', tf.executing_eagerly())
```

--------------------------------

### Alibi Explainer Initialization and Usage

Source: https://github.com/seldonio/alibi/blob/master/doc/source/overview/getting_started.md

Demonstrates the basic workflow of initializing, fitting, and explaining with an Alibi explainer, using AnchorTabular as an example.

```APIDOC
## Alibi Explainer Basic Usage

### Description
This section illustrates the fundamental steps involved in using Alibi explainers, which follow a pattern similar to scikit-learn: initialize, fit, and explain.

### Method
Python

### Endpoint
N/A (Python Library Usage)

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
```python
from alibi.explainers import AnchorTabular

# Assume predict_fn and feature_names are defined
# Assume X_train and x are defined

# Initialize the explainer
explainer = AnchorTabular(predict_fn, feature_names)

# Fit the explainer (if required by the method)
explainer.fit(X_train)

# Explain an instance
explanation = explainer.explain(x)
```

### Response
#### Success Response (Explanation Object)
- **meta** (dict) - Metadata about the explanation, including explainer parameters and type.
- **data** (dict) - The actual explanation results, such as anchors, precision, and coverage.

#### Response Example
```json
{
  "meta": {
    "name": "AnchorTabular",
    "type": ["blackbox"],
    "explanations": ["local"],
    "params": {
      "seed": null,
      "disc_perc": [25, 50, 75],
      "threshold": 0.95,
      "delta": "..."
    }
  },
  "data": {
    "anchor": [
      "petal width (cm) > 1.80",
      "sepal width (cm) <= 2.80"
    ],
    "precision": 0.9839228295819936,
    "coverage": 0.31724137931034485,
    "raw": {
      "feature": [3, 1],
      "mean": [0.6453362255965293, 0.9839228295819936],
      "precision": [0.6453362255965293, 0.9839228295819936],
      "coverage": [0.20689655172413793, 0.31724137931034485],
      "examples": "..."
    }
  }
}
```

### Error Handling
N/A (Library Usage)
```

--------------------------------

### Install Alibi with Deep Learning Dependencies

Source: https://github.com/seldonio/alibi/blob/master/doc/source/methods/CFRL.ipynb

Commands to install the Alibi library with support for either TensorFlow or PyTorch, which are required for Reinforcement Learning-based counterfactual generation.

```bash
pip install alibi[tensorflow]
```

```bash
pip install alibi[torch]
```

--------------------------------

### AnchorText Explainer Initialization with Sampling Strategies

Source: https://github.com/seldonio/alibi/blob/master/doc/source/methods/Anchors.ipynb

Demonstrates how to initialize the AnchorText explainer with different sampling strategies and their associated parameters.

```APIDOC
## AnchorText Explainer Initialization with Sampling Strategies

### Description
This section shows how to initialize the `AnchorText` explainer with various `sampling_strategy` options, including 'unknown', 'similarity', and 'language_model'. Each strategy has specific parameters that can be passed via `kwargs` to customize its behavior.

### Sampling Strategies

#### 1. `sampling_strategy='unknown'`
This strategy replaces words with an 'UNK' token.

##### Parameters
* `sampling_strategy` (string) - Set to 'unknown'.
* `nlp` (object) - A spaCy language model object.
* `sample_proba` (float) - The probability of a word being replaced by the UNK token.

##### Request Example
```python
from alibi.explainers import AnchorText

# Assuming predict_fn and nlp are defined
explainer = AnchorText(
    predictor=predict_fn, 
    sampling_strategy='unknown',
    nlp=nlp,
    sample_proba=0.5
)
```

#### 2. `sampling_strategy='similarity'`
This strategy replaces words with similar words based on word embeddings.

##### Parameters
* `sampling_strategy` (string) - Set to 'similarity'.
* `nlp` (object) - A spaCy language model object.
* `sample_proba` (float) - The probability of a word being replaced by a similar word.
* `use_proba` (bool) - If True, samples words according to their similarity distribution; otherwise, samples uniformly.
* `top_n` (int) - Considers only the top N most similar words for replacement.
* `temperature` (float) - Controls the randomness of sampling. Higher values increase randomness.

##### Request Example
```python
from alibi.explainers import AnchorText

# Assuming predict_fn and nlp are defined
explainer = AnchorText(
    predictor=predict_fn, 
    sampling_strategy='similarity',
    nlp=nlp,
    sample_proba=0.5,
    use_proba=True,
    top_n=20,
    temperature=0.2
)
```

#### 3. `sampling_strategy='language_model'`
This strategy uses a language model to predict masked words.

##### Parameters
* `sampling_strategy` (string) - Set to 'language_model'.
* `language_model` (object) - The language model to be used (e.g., a Hugging Face transformer model).
* `filling` (string) - Method for filling masked words. Options: 'parallel' (single pass, independent sampling) or 'autoregressive' (multiple passes, conditioned sampling). 'autoregressive' is computationally expensive.
* `sample_proba` (float) - The probability of masking and replacing a word according to the LM.
* `frac_mask_templates` (float) - The fraction of masking templates to use.
* `use_proba` (bool) - If True, uses word distribution from the LM for sampling; otherwise, samples uniformly.
* `top_n` (int) - Considers the top N most likely words predicted by the LM.
* `temperature` (float) - Controls the randomness of sampling. Higher values increase randomness.
* `stopwords` (list of strings) - Words that will not be masked or disturbed.
* `punctuation` (string) - A string containing punctuation tokens that will not be masked or disturbed.
* `sample_punctuation` (bool) - If False, tokens included in `punctuation` will not be sampled.
* `batch_size_lm` (int) - Batch size used for the language model inference.

##### Request Example
```python
import string
from alibi.explainers import AnchorText

# Assuming predict_fn, language_model are defined
explainer = AnchorText(
    predictor=predict_fn,
    sampling_strategy="language_model",
    language_model=language_model,
    filling="parallel",
    sample_proba=0.5,
    frac_mask_templates=0.1,
    use_proba=True,
    top_n=50,
    temperature=0.2,
    stopwords=['and', 'a', 'but'],
    punctuation=string.punctuation,
    sample_punctuation=False,
    batch_size_lm=32
)
```

### General Sampling Parameters

Words outside of the candidate anchor can be replaced using the strategies above. The probability of replacement is controlled by `sample_proba`. The selection of replacement candidates can be further refined using `top_n` and `temperature` parameters. `use_proba` allows sampling based on probability distributions rather than uniform sampling.

### Language Model Filling Methods

For `sampling_strategy='language_model'`, two filling methods are available:

*   **`filling='parallel'`**: Performs a single forward pass through the transformer. Masked words are sampled independently.
*   **`filling='autoregressive'`**: Performs multiple forward passes, generating words one at a time, conditioned on previous words. This method is computationally expensive.

```

--------------------------------

### Install SHAP and compatible matplotlib

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/kernel_shap_wine_lr.ipynb

Installs the alibi[shap] package for SHAP support and a specific version of matplotlib (3.5.3) to avoid compatibility issues with shap.summary_plot.

```bash
pip install alibi[shap]
pip install matplotlib==3.5.3
```

--------------------------------

### Install Alibi Dependencies

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/distributed_kernel_shap_adult_lr.ipynb

Commands to install optional dependencies for SHAP and Ray support. Note that Ray-based parallel execution for KernelSHAP is currently limited on Windows.

```bash
pip install alibi[shap]
pip install alibi[ray]
```

--------------------------------

### Initialize and Run Counterfactual Explanation

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/cfproto_housing.md

Prepares an instance from the test set for counterfactual analysis and initializes the CounterfactualProto explainer from Alibi. This step involves loading the trained neural network model and setting up the explainer with the model and relevant parameters to find counterfactual explanations.

```python
X = X_test[1].reshape((1,) + X_test[1].shape)
shape = X.shape

# define model
nn = load_model('nn_california.h5')

# Initialize explainer
explainer = CounterfactualProto(nn, 
                              shape=shape,
                              feature_names=feature_names,
                              target_names=['0', '1'])

# Generate counterfactuals
explanation = explainer.explain(X)
```

--------------------------------

### Initialize and Use TreeShap

Source: https://github.com/seldonio/alibi/blob/master/doc/source/overview/high_level.md

Demonstrates how to instantiate the TreeShap explainer, fit it to reference data, and compute explanations for a specific instance.

```APIDOC
## POST /explainers/tree-shap

### Description
Initializes and executes the Interventional Tree SHAP explainer to calculate feature contributions for tree-based models.

### Method
POST

### Endpoint
/explainers/tree-shap

### Parameters
#### Request Body
- **model** (object) - Required - The trained tree-based model (e.g., XGBoost, LightGBM, CatBoost).
- **background_data** (array) - Required - Reference dataset used to compute interventional expectations.
- **instance** (array) - Required - The specific data point to explain.

### Request Example
{
  "model": "rfc_model_object",
  "background_data": [[0.1, 0.5], [0.2, 0.4]],
  "instance": [0.15, 0.45]
}

### Response
#### Success Response (200)
- **shap_values** (array) - The calculated Shapley values for each feature.

#### Response Example
{
  "shap_values": [0.05, -0.02]
}
```

--------------------------------

### Set Number of Covered Examples

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/api/alibi/explainers/anchors/anchor_tabular_distributed.md

Updates the internal state to track the number of examples covered by the current anchor or partial anchor. This function does not return a value.

```python
def set_n_covered(n_covered: int) -> None:
```

--------------------------------

### POST /explainer/initialize

Source: https://github.com/seldonio/alibi/blob/master/examples/anchor_text_movie.ipynb

Initializes the AnchorText explainer with specific sampling strategies and language model configurations.

```APIDOC
## POST /explainer/initialize

### Description
Initializes the AnchorText explainer object to interpret text-based model predictions using language model-based sampling.

### Method
POST

### Endpoint
/explainer/initialize

### Parameters
#### Request Body
- **predictor** (callable) - Required - The model prediction function.
- **sampling_strategy** (string) - Required - Strategy to use (e.g., 'language_model').
- **language_model** (object) - Required - The language model instance.
- **filling** (string) - Optional - Filling method ('parallel' or 'autoregressive').
- **sample_proba** (float) - Optional - Probability of masking a word.
- **top_n** (int) - Optional - Number of most likely words to consider.
- **temperature** (float) - Optional - Sampling randomness level.

### Request Example
{
  "sampling_strategy": "language_model",
  "filling": "parallel",
  "sample_proba": 0.5,
  "top_n": 20
}

### Response
#### Success Response (200)
- **status** (string) - Confirmation of initialization.
```

--------------------------------

### Install Alibi and Dependencies

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/overview.md

Commands to install the Alibi library and necessary visualization tools like seaborn. These are required to ensure all explainers and plotting functions work correctly.

```bash
pip install alibi[all]
```

```python
!pip install -q seaborn
```

--------------------------------

### Initialize CounterfactualProto Explainer

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/cfproto_cat_adult_ohe.md

Sets up the CounterfactualProto explainer for generating counterfactuals. It requires the model, instance shape, perturbation parameters (beta, c_init, c_steps, max_iterations), feature ranges, and categorical variable information. Dependencies include NumPy and TensorFlow.

```python
X = X_test[0].reshape((1,) + X_test[0].shape)
shape = X.shape
beta = .01
c_init = 1.
c_steps = 5
max_iterations = 500
rng = (-1., 1.)  # scale features between -1 and 1
rng_shape = (1,) + data.shape[1:]
feature_range = ((np.ones(rng_shape) * rng[0]).astype(np.float32),
                 (np.ones(rng_shape) * rng[1]).astype(np.float32))

def set_seed(s=0):
    np.random.seed(s)
    tf.random.set_seed(s)

set_seed()
cf = CounterfactualProto(nn,
                         shape,
                         beta=beta,
                         cat_vars=cat_vars_ohe,
                         ohe=True,  # OHE flag
                         max_iterations=max_iterations,
                         feature_range=feature_range,
                         c_init=c_init,
                         c_steps=c_steps
                        )
```

--------------------------------

### Install and Use UMAP for Prototype Visualization

Source: https://github.com/seldonio/alibi/blob/master/doc/source/examples/protoselect_adult_cifar10.ipynb

This code block first installs the `umap-learn` library. Then, it defines and fits a UMAP reducer to preprocess the training data for 2D visualization of prototypes.

```python
!pip install umap-learn
```

```python
import umap

# define 2D reducer
reducer = umap.UMAP(random_state=26)
reducer = reducer.fit(preprocess_fn(X_train))
```

--------------------------------

### Install Seaborn for Visualization

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/examples/cem_iris.md

Installs the seaborn package using pip, which is used for data visualization purposes within the notebook. Seaborn enhances the plotting capabilities, making it easier to interpret the results.

```python
!pip install seaborn
```

--------------------------------

### Initialize IntegratedGradients Explainer

Source: https://github.com/seldonio/alibi/blob/master/doc/source/methods/IntegratedGradients.ipynb

Demonstrates how to instantiate the IntegratedGradients class with a TensorFlow/Keras model and configure integration parameters.

```python
import tensorflow as tf
from alibi.explainers import IntegratedGradients

model = tf.keras.models.load_model("path_to_your_model")

ig  = IntegratedGradients(model,
                          layer=None,
                          taget_fn=None,
                          method="gausslegendre",
                          n_steps=50,
                          internal_batch_size=100)
```

--------------------------------

### Initialize and Configure AnchorTabular Explainer

Source: https://github.com/seldonio/alibi/blob/master/docs-gb/source/methods/anchors.md

Demonstrates how to define categorical mappings, initialize the AnchorTabular explainer with a prediction function, and optionally enable one-hot encoding support.

```python
from alibi.utils import gen_category_map
from alibi.explainers import AnchorTabular

# Generate categorical map from dataframe
category_map = gen_category_map(df)

# Define prediction function
predict_fn = lambda x: clf.predict(preprocessor.transform(x))

# Initialize explainer
explainer = AnchorTabular(predict_fn, feature_names, categorical_names=category_map)

# Initialize with one-hot encoding support
explainer_ohe = AnchorTabular(predict_fn, feature_names, categorical_names=category_map, ohe=True)
```

--------------------------------

### Install Mamba and Alibi via Conda

Source: https://github.com/seldonio/alibi/blob/master/README.md

Commands to install the mamba package manager and the Alibi library with optional dependencies like Ray for distributed computing or SHAP for specific explanation methods.

```bash
conda install mamba -n base -c conda-forge
mamba install -c conda-forge alibi
mamba install -c conda-forge alibi ray
mamba install -c conda-forge alibi shap
```