### Install Eurybia with pip

Source: https://eurybia.readthedocs.io/en/latest/_sources/overview.rst.txt

Install the Eurybia library using pip.

```ipython
pip install eurybia
```

--------------------------------

### Install Eurybia with pip

Source: https://eurybia.readthedocs.io/en/latest/overview.html

Install the Eurybia package using pip.

```bash
pip install eurybia

```

--------------------------------

### Install plotly and jupyterlab-plotly extension

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install plotly and the jupyterlab-plotly extension in the JupyterLab environment.

```bash
conda install -c plotly plotly
jupyter labextension install jupyterlab-plotly
```

--------------------------------

### Usage example: calling SmartPlotter methods from SmartDrift

Source: https://eurybia.readthedocs.io/en/latest/autodocs/eurybia.core.smartplotter.html

Shows the standard syntax for using any SmartPlotter method from a SmartDrift object. The example creates a SmartDrift instance, compiles it, and then calls a plotting method with a parameter.

```python
>>> SD = Smartdrift()
>>> SD.compile()
>>> SD.plot.my_plot_method(param=value)

```

--------------------------------

### Install ipywidgets and plotly for JupyterLab

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install ipywidgets and plotly in the JupyterLab environment. The first command installs jupyterlab_widgets in the base environment and ipywidgets in the py36 environment. The second command installs plotly and its JupyterLab extension. These are prerequisites for using Eurybia's interactive widgets and plots.

```bash
conda install -n base -c conda-forge jupyterlab_widgets
conda install -n py36 -c conda-forge ipywidgets
```

```bash
conda install -c plotly plotly
jupyter labextension install jupyterlab-plotly
```

--------------------------------

### Error output for missing nodejs

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Example error output when nodejs is not installed or is too old. The error indicates nodejs >=10.0.0 is required.

```bash
jupyter labextension install @jupyter-widgets/jupyterlab-manager
An error occured.
ValueError: Please install nodejs >=10.0.0 before continuing. nodejs may be installed using conda or directly from the nodejs website.
See the log file for details:  /tmp/jupyterlab-debug-y_g3xxpq.log
```

--------------------------------

### Import required libraries for Eurybia data drift detection

Source: https://eurybia.readthedocs.io/en/latest/tutorials/model_drift/tutorial02-modeldrift-high-datadrift.html

Imports the required libraries for the tutorial, including Eurybia's SmartDrift, pandas, category_encoders, catboost, scikit-learn, and numpy. This setup is necessary before using Eurybia to detect data drift.

```python
from datetime import date
import pandas as pd
from category_encoders import OrdinalEncoder
import catboost
from eurybia import SmartDrift
from sklearn.model_selection import train_test_split
from sklearn import metrics
import numpy as np
```

--------------------------------

### Install ipywidgets with pip

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install ipywidgets with pip to enable interactive widgets in Jupyter Notebook and JupyterLab 3.0.

```bash
pip install ipywidgets
```

--------------------------------

### Install nodejs with conda gcc7 label

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install nodejs with conda using the gcc7 label, as suggested after the nodejs version error.

```bash
conda install -c conda-forge/label/gcc7 nodejs
```

--------------------------------

### Install ipywidgets with conda

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install ipywidgets with conda from the conda-forge channel.

```bash
conda install -c conda-forge ipywidgets
```

--------------------------------

### Install nodejs with conda

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install nodejs with conda, required for installing the JupyterLab extension in JupyterLab 1 or 2.

```bash
conda install -c conda-forge nodejs
```

--------------------------------

### Install JupyterLab manager extension

Source: https://eurybia.readthedocs.io/en/latest/installation-instructions/index.html

Installs the @jupyter-widgets/jupyterlab-manager extension for JupyterLab 1 or 2. Defaults to latest version; may need older version depending on JupyterLab version. Requires nodejs.

```bash
jupyter labextension install @jupyter-widgets/jupyterlab-manager

```

--------------------------------

### Install JupyterLab extension for ipywidgets

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install the JupyterLab extension for ipywidgets in JupyterLab 1 or 2. Defaults to the latest version; older versions may be needed depending on JupyterLab version.

```bash
jupyter labextension install @jupyter-widgets/jupyterlab-manager
```

--------------------------------

### Install widgetsnbextension and ipywidgets in separate conda environments

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Install widgetsnbextension in the Jupyter Notebook environment and ipywidgets in the kernel environment when they are separate conda environments.

```bash
conda install -n base -c conda-forge widgetsnbextension
conda install -n py36 -c conda-forge ipywidgets
```

--------------------------------

### Nodejs version error and fix

Source: https://eurybia.readthedocs.io/en/latest/installation-instructions/index.html

Shows the error when nodejs >=10.0.0 is not installed, and the recommended conda command to install nodejs from the gcc7 label.

```bash
jupyter labextension install @jupyter-widgets/jupyterlab-manager
An error occured.
ValueError: Please install nodejs >=10.0.0 before continuing. nodejs may be installed using conda or directly from the nodejs website.
See the log file for details:  /tmp/jupyterlab-debug-y_g3xxpq.log

```

```bash
conda install -c conda-forge/label/gcc7 nodejs

```

--------------------------------

### Import required libraries

Source: https://eurybia.readthedocs.io/en/latest/tutorials/tutorial01-Eurybia-overview.html

Imports OrdinalEncoder, LGBMRegressor, SmartDrift, and train_test_split. A warning may appear about installing jupyter_bokeh for interactive Panel in VSCode notebooks.

```python
from category_encoders import OrdinalEncoder
from lightgbm import LGBMRegressor
from eurybia import SmartDrift
from sklearn.model_selection import train_test_split

```

--------------------------------

### Get dataset shape

Source: https://eurybia.readthedocs.io/en/latest/tutorials/model_drift/tutorial02-modeldrift-high-datadrift.html

Returns the shape of the dataset, showing 50,000 rows and 13 columns.

```python
df_car_accident.shape
```

--------------------------------

### LightGBM training output for LGBMRegressor fit

Source: https://eurybia.readthedocs.io/en/latest/tutorials/tutorial01-Eurybia-overview.html

Observed console output when fitting the LGBMRegressor. Shows auto-chosen row-wise multi-threading, total bins, training data size (235 points, 53 features), starting score, and repeated warnings about no further splits with positive gain.

```text
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.001133 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1020
[LightGBM] [Info] Number of data points in the train set: 235, number of used features: 53
[LightGBM] [Info] Start training from score 181743.472340
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf

```

--------------------------------

### Create performance tracking DataFrame

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/model_drift/tutorial01-modeldrift.ipynb.txt

Creates a pandas DataFrame to track model performance over the years, starting with the year 2006 and the computed test performance. The DataFrame has columns 'annee', 'mois', and 'performance'.

```python
#Create Dataframe to track performance over the years
df_performance = pd.DataFrame({'annee': [2006], 'mois':[1], 'performance': [performance_test]})
```

--------------------------------

### Error installing jupyterlab-plotly: nodejs required

Source: https://eurybia.readthedocs.io/en/latest/_sources/installation-instructions/index.rst.txt

Example error message when installing jupyterlab-plotly without Node.js. The error indicates that Node.js >=10.0.0 is required and can be installed via conda or from the Node.js website. The log file path is shown for debugging.

```bash
jupyter labextension install jupyterlab-plotly
An error occured.
ValueError: Please install nodejs >=10.0.0 before continuing. nodejs may be installed using conda or directly from the nodejs website.
See the log file for details:  /tmp/jupyterlab-debug-y_g3xxpq.log
```

--------------------------------

### Import required libraries

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial02-datadrift-high-datadrift.html

Import pandas, category_encoders, catboost, eurybia, and sklearn modules needed for the tutorial.

```python
import pandas as pd
from category_encoders import OrdinalEncoder
import catboost
from eurybia import SmartDrift
from sklearn.model_selection import train_test_split
from sklearn import metrics
import numpy as np
```

--------------------------------

### Import required libraries for Eurybia custom colors tutorial

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common01-colors.html

Imports the required libraries for the tutorial: pandas for data handling, category_encoders for ordinal encoding, LightGBM for the regression model, and scikit-learn for splitting the data.

```python
import pandas as pd
from category_encoders import OrdinalEncoder
from lightgbm import LGBMRegressor
from sklearn.model_selection import train_test_split

```

--------------------------------

### Import SmartDrift and instantiate with data drift configuration

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/tutorial01-Eurybia-overview.ipynb.txt

Import SmartDrift from the eurybia package, then fill missing MasVnrType values with 'NR' in both datasets and instantiate SmartDrift with the current and baseline dataframes, the deployed model, encoding, and optional dataset names.

```python
from eurybia import SmartDrift
```

```python
X_df_2007["MasVnrType"] = X_df_2007["MasVnrType"].fillna("NR")
X_df_learning["MasVnrType"] = X_df_learning["MasVnrType"].fillna("NR")

SD = SmartDrift(df_current=X_df_2007,
                df_baseline=X_df_learning,
                deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
                encoding=encoder, # Optional: if deployed_model and encoder to use this model
                dataset_names=("2007 dataset","Learning dataset") # Optional: Names for outputs
               )
```

--------------------------------

### Create selectable tables with pn.widgets.Select and jscallback

Source: https://eurybia.readthedocs.io/en/latest/_modules/eurybia/report/generation.html

Creates a list of blocks containing a pn.widgets.Select with a jscallback and a DataFrame pane for each label. The first table is visible, subsequent ones are hidden via CSS classes.

```python
def _get_select_plots(labels: list, key: str, tab: str, figures: list) -> list:
    blocks = []
    select = pn.widgets.Select(value=labels[0], options=labels)
    select.jscallback(args={"key": f".{key}", "tab": tab}, value=select_callback)
    blocks += [select]
    for i in range(len(labels)):
        f_class = labels[i].replace(" ", "-").lower()
        css_classes = [f_class, key]
        if i > 0:
            css_classes.append("hidden")
        node = pn.pane.DataFrame(tables[i], css_classes=css_classes)
        blocks += [node]
    return blocks
```

--------------------------------

### Nodejs error for jupyterlab-plotly

Source: https://eurybia.readthedocs.io/en/latest/installation-instructions/index.html

Shows the error when nodejs >=10.0.0 is not installed during jupyterlab-plotly installation.

```bash
jupyter labextension install jupyterlab-plotly
An error occured.
ValueError: Please install nodejs >=10.0.0 before continuing. nodejs may be installed using conda or directly from the nodejs website.
See the log file for details:  /tmp/jupyterlab-debug-y_g3xxpq.log

```

--------------------------------

### Import required libraries for Eurybia drift detection

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/tutorial01-Eurybia-overview.ipynb.txt

Imports the required libraries for the tutorial: category_encoders for OrdinalEncoder, lightgbm for LGBMRegressor, eurybia for SmartDrift, and scikit-learn for train_test_split.

```python
from category_encoders import OrdinalEncoder
from lightgbm import LGBMRegressor
from eurybia import SmartDrift
from sklearn.model_selection import train_test_split
```

--------------------------------

### Install jupyterlab_widgets and ipywidgets in separate conda environments

Source: https://eurybia.readthedocs.io/en/latest/installation-instructions/index.html

Two-step installation for JupyterLab 3 when JupyterLab and IPython kernel are in different conda environments: install jupyterlab_widgets in the JupyterLab environment (base) and ipywidgets in the kernel environment (py36).

```bash
conda install -n base -c conda-forge jupyterlab_widgets
conda install -n py36 -c conda-forge ipywidgets

```

--------------------------------

### Preview dataset head

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial02-datadrift-high-datadrift.html

Display the first rows of the dataset to verify loading.

```python
df_car_accident.head()
```

```python
df_car_accident.shape
```

--------------------------------

### Instantiate SmartDrift with production and baseline data

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common01-colors.html

Creates a SmartDrift instance with the production features as current and the learning features as baseline.

```python
SD = SmartDrift(df_current=X_df_production, df_baseline=X_df_learning)

```

--------------------------------

### Initialize SmartDrift with datasets and model

Source: https://eurybia.readthedocs.io/en/latest/tutorials/tutorial01-Eurybia-overview.html

Creates a SmartDrift instance with current and baseline datasets, an optional deployed model, an encoder, and dataset names. The deployed model and encoder are optional but used to put results in perspective with model importance.

```python
X_df_2007["MasVnrType"] = X_df_2007["MasVnrType"].fillna("NR")
X_df_learning["MasVnrType"] = X_df_learning["MasVnrType"].fillna("NR")

SD = SmartDrift(df_current=X_df_2007,
                df_baseline=X_df_learning,
                deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
                encoding=encoder, # Optional: if deployed_model and encoder to use this model
                dataset_names=("2007 dataset","Learning dataset") # Optional: Names for outputs
               )
```

--------------------------------

### Import libraries for model drift detection

Source: https://eurybia.readthedocs.io/en/latest/tutorials/model_drift/tutorial01-modeldrift.html

Import required libraries for the model drift tutorial, including pandas, numpy, category_encoders, lightgbm, eurybia, and scikit-learn modules.

```python
import pandas as pd
import numpy as np
from datetime import date
from category_encoders import OrdinalEncoder
from lightgbm import LGBMRegressor
from eurybia import SmartDrift
from eurybia.data.data_loader import data_loading
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_log_error
```

--------------------------------

### execute_report(smartdrift, explainer, output_file, project_info_file=None, config_report=None, modalities_analysis=False) -> None

Source: https://eurybia.readthedocs.io/en/latest/autodocs/eurybia.report.generation.html

Creates the report and writes it to an HTML file using the compiled SmartDrift and SmartExplainer objects.

```APIDOC
## execute_report(smartdrift, explainer, output_file, project_info_file=None, config_report=None, modalities_analysis=False) -> None

### Description
Creates the report.

### Function Signature
`execute_report(smartdrift : SmartDrift_, explainer : SmartExplainer_, output_file : str_, project_info_file : str | None = None, config_report : dict | None = None, modalities_analysis : bool = False) -> None`

### Parameters
- **smartdrift** (eurybia.core.smartdrift.SmartDrift object) - Required - Compiled SmartDrift class.
- **explainer** (shapash.explainer.smart_explainer.SmartExplainer object) - Required - Compiled shapash explainer.
- **output_file** (str) - Required - Path to the HTML file to write.
- **project_info_file** (str | None = None) - Optional - Path to the file used to display some information about the project in the report.
- **config_report** (dict | None = None) - Optional - Report configuration options.
- **modalities_analysis** (bool = False) - Optional - Boolean indicating whether modalities analysis is enabled.

### Returns
- **None** - This function does not return a value.
```

--------------------------------

### Import required libraries

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Imports the required libraries for the tutorial: OrdinalEncoder from category_encoders, catboost, and train_test_split from scikit-learn.

```python
from category_encoders import OrdinalEncoder
import catboost
from sklearn.model_selection import train_test_split

```

--------------------------------

### Import libraries for data drift analysis

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial01-datadrift-over-years.html

Imports pandas, numpy, category_encoders, lightgbm, eurybia, and sklearn modules needed for the tutorial. A warning about jupyter_bokeh may appear when using Panel interactively in VSCode.

```python
import pandas as pd
import numpy as np
from category_encoders import OrdinalEncoder
from lightgbm import LGBMRegressor
from eurybia import SmartDrift
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_log_error
```

--------------------------------

### data_loading(dataset)

Source: https://eurybia.readthedocs.io/en/latest/autodocs/eurybia.data.html

Loads a small, clearly labeled dataset for testing the Eurybia library. Available datasets are Titanic, house_prices, and us_car_accident.

```APIDOC
## data_loading(dataset)

### Description
Loads a small, clearly labeled dataset for testing the Eurybia library. Available datasets are Titanic, house_prices, and us_car_accident.

### Method
Function call

### Endpoint
`eurybia.data.data_loader.data_loading(dataset)`

### Parameters
#### Path Parameters
- **dataset** (String) - Required - Dataset's name to return. Valid values: 'titanic', 'house_prices', 'us_car_accident'.

### Returns
- **data** (pandas.DataFrame) - The requested dataset.
- **dict** (Dictionary, Optional) - If exists, column labels dictionary associated with the dataset.

### Example
```python
>>> from eurybia.data.data_loader import data_loading
>>> house_df, house_dict = data_loading('house_prices')
```
```

--------------------------------

### Initialize CatBoostClassifier

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_validation/tutorial01-data-validation.html

Creates a CatBoostClassifier with specified hyperparameters.

```python
model = catboost.CatBoostClassifier(loss_function= "Logloss", eval_metric="Logloss",
        learning_rate=0.143852,
        iterations=500,
        l2_leaf_reg=15,
        max_depth = 4)
```

--------------------------------

### Get indices of categorical features

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Builds a list of column indices for the categorical features to be used as cat_features in CatBoost.

```python
i=0
indice_cat  = []
for feature in titan_df_encoded:
    if feature in features_to_encode:
        indice_cat.append(i)
    i=i+1

```

--------------------------------

### Stop Shapash WebApp with kill()

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/common/tuto-common02-shapash-webapp.ipynb.txt

Stops the Shapash WebApp that was started with run_app. Call this after you are done using the app.

```python
app.kill()
```

--------------------------------

### Create SmartDrift, compile, and generate HTML report

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_validation/tutorial01-data-validation.html

Create a SmartDrift object with current and baseline data, optionally passing a deployed model and encoder. Compile with full_validation=True to analyze column modality consistency; the output shows timing and Shap explainer type. Generate an HTML report with output file, title, optional subtitle, and optional project info file.

```python
SD = SmartDrift(df_current=df_current,
                df_baseline=df_baseline,
                deployed_model=model, # Optional: put in perspective result with importance on deployed model
                encoding=encoder # Optional: if deployed_model and encoder to use this model
                )
```

```python
%time
SD.compile(full_validation=True # Optional: to save time, leave the default False value. If True, analyze consistency on modalities between columns.
            )
```

```text
CPU times: user 2 μs, sys: 1e+03 ns, total: 3 μs
Wall time: 4.77 μs
INFO: Shap explainer type - <shap.explainers._tree.TreeExplainer object at 0x11f8956a0>
```

```python
SD.generate_report(
    output_file='report_titanic.html',
    title_story="Data validation",
    title_description="""Titanic Data validation""", # Optional: add a subtitle to describe report
    project_info_file="../../eurybia/data/project_info_titanic.yml" # Optional: add information on report
    )
```

--------------------------------

### Prepare 2010 house data for drift analysis

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/model_drift/tutorial01-modeldrift.ipynb.txt

Filters the house dataset for the year 2010, separates the target variable SalePrice into y_df_2010, and creates X_df_2010 with all columns except SalePrice and YrSold. This prepares the data for drift analysis on the 2010 period.

```python
house_df_2010 = house_df.loc[house_df['YrSold'] == 2010]

y_df_2010=house_df_2010['SalePrice'].to_frame()
X_df_2010=house_df_2010[house_df_2010.columns.difference(['SalePrice','YrSold'])]
```

--------------------------------

### Plot data drift feature importance with SD.xpl.plot.features_importance

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_validation/tutorial01-data-validation.html

Displays a bar plot of the most important features in the data drift classification model. In the example, fare, age, and sex are the features that explain most of the drift.

```python
SD.xpl.plot.features_importance()
```

--------------------------------

### Split house prices data into training and production subsets

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common01-colors.html

Splits the loaded house prices data into training (YearBuilt < 1980) and production (YearBuilt >= 1980) subsets to introduce a bias in construction date for drift analysis.

```python
#For the purpose of this tutorial and to better represent a common use case of Eurybia,
#the house_prices dataset was split in two smaller sets : "training" and "production"
# To see an interesting analysis, let's test for a bias in the date of construction of training and production dataset
house_df_learning = house_df.loc[house_df['YearBuilt'] < 1980]
house_df_production = house_df.loc[house_df['YearBuilt'] >= 1980]

```

--------------------------------

### Prepare 2010 data for drift analysis

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial01-datadrift-over-years.html

Filters the house price dataset for the year 2010, separates the target variable, and prepares the feature set by removing 'SalePrice' and 'YrSold' columns. Also fills missing values in the 'MasVnrType' column with 'NR'.

```python
house_df_2010 = house_df.loc[house_df['YrSold'] == 2010]

y_df_2010=house_df_2010['SalePrice'].to_frame()
X_df_2010=house_df_2010[house_df_2010.columns.difference(['SalePrice','YrSold'])]
X_df_2010["MasVnrType"] = X_df_2010["MasVnrType"].fillna("NR")

```

--------------------------------

### Prepare learning and production dataframes for SmartDrift

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_validation/tutorial02-data-validation-iteration.html

Prepares the learning and production datasets by separating the target variable 'SalePrice' and dropping the same set of columns from both dataframes. The dropped columns include categorical and numeric features that are not used in the validation.

```python
y_df_learning=house_df_learning['SalePrice'].to_frame()
X_df_learning=house_df_learning[house_df_learning.columns.difference(['SalePrice','YearBuilt','BsmtQual',
                                                'Neighborhood','Foundation','GarageYrBlt','YearRemodAdd',
                                                'GarageFinish','OverallCond','MSZoning','BsmtFinType1','MSSubClass',
                                                'ExterQual','KitchenQual','Exterior2nd','Exterior1st','OverallQual',
                                                'HeatingQC','FullBath','OpenPorchSF','GarageType','GrLivArea','GarageArea'])]

y_df_production=house_df_production['SalePrice'].to_frame()
X_df_production=house_df_production[house_df_production.columns.difference(['SalePrice','YearBuilt','BsmtQual',
                                                'Neighborhood','Foundation','GarageYrBlt','YearRemodAdd',
                                                'GarageFinish','OverallCond','MSZoning','BsmtFinType1','MSSubClass',
                                                'ExterQual','KitchenQual','Exterior2nd','Exterior1st','OverallQual',
                                                'HeatingQC','FullBath','OpenPorchSF','GarageType','GrLivArea','GarageArea'])]
```

--------------------------------

### Generate Report

Source: https://eurybia.readthedocs.io/en/latest/_sources/overview.rst.txt

Generate an HTML report. Optional parameters include a subtitle and a project info file. Providing deployed_model and encoding enriches the report.

```ipython
sd.generate_report(
  output_file='output/my_report_name.html',
  title_story="my_report_title",
  title_description="my_report_subtitle", # Optional: add a subtitle to describe report
  project_info_file='project_info.yml' # Optional: add information on report
  )
```

--------------------------------

### Create SmartDrift instance with data and model

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/common/tuto-common02-shapash-webapp.ipynb.txt

Instantiates a SmartDrift object with the current and baseline dataframes, the deployed model, and the encoder.

```python
sd = SmartDrift(df_current=df_current, df_baseline=df_baseline, deployed_model=model, encoding=encoder)
```

--------------------------------

### Create multi-indicator performance DataFrame

Source: https://eurybia.readthedocs.io/en/latest/tutorials/model_drift/tutorial01-modeldrift.html

Create a dummy performance table with multiple indicators (rmse and mse) by exponentiating the performance values and concatenating with the original DataFrame. The result has an 'indicator' column to distinguish metrics.

```python
df_performance_mse = df_performance.copy()
df_performance_mse['performance']= np.exp(df_performance_mse['performance'])
df_performance2 = pd.concat([df_performance, df_performance_mse], keys=["rmse", "mse"]).reset_index().rename(columns={"level_0": "indicator"})
df_performance2
```

--------------------------------

### Create and compile SmartDrift for data drift detection

Source: https://eurybia.readthedocs.io/en/latest/tutorials/model_drift/tutorial02-modeldrift-high-datadrift.html

Import SmartDrift, instantiate it with current and baseline dataframes, optionally a deployed model and encoder, then compile with full validation and optional date and datadrift file. The compile method displays consistency warnings and creates the datadrift CSV if missing.

```python
from eurybia import SmartDrift
```

```python
SD = SmartDrift(df_current=X_df_2017,
                df_baseline=X_df_learning,
                deployed_model=model, # Optional: put in perspective result with importance on deployed model
                encoding=encoder # Optional: if deployed_model and encoder to use this model
               )
```

```python
%time
SD.compile(full_validation=True, # Optional: to save time, leave the default False value. If True, analyze consistency on modalities between columns.
           date_compile_auc = date(2017,1,1), # Optional: useful when computing the drift for a time that is not now
           datadrift_file = "car_accident_auc.csv" # Optional: name of the csv file that contains the performance history of data drift
          )
```

--------------------------------

### Prepare 2009 data subset for drift analysis

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial01-datadrift-over-years.html

Prepares the 2009 data subset by filtering the house_df on 'YrSold' equal to 2009, separating the target 'SalePrice' into y_df_2009, and building X_df_2009 with all columns except 'SalePrice' and 'YrSold'. Also fills missing 'MasVnrType' values with the string "NR".

```python
house_df_2009 = house_df.loc[house_df['YrSold'] == 2009]

y_df_2009=house_df_2009['SalePrice'].to_frame()
X_df_2009=house_df_2009[house_df_2009.columns.difference(['SalePrice','YrSold'])]
X_df_2009["MasVnrType"] = X_df_2009["MasVnrType"].fillna("NR")

```

--------------------------------

### Initialize CatBoostClassifier

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Initializes a CatBoostClassifier with Logloss loss and evaluation metric, learning rate 0.143852, 500 iterations, L2 leaf regularization 15, and max depth 4.

```python
model = catboost.CatBoostClassifier(loss_function= "Logloss", eval_metric="Logloss",
        learning_rate=0.143852,
        iterations=500,
        l2_leaf_reg=15,
        max_depth = 4)

```

--------------------------------

### Evaluate model on 2008 data and append performance

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/model_drift/tutorial01-modeldrift.ipynb.txt

Prepares the 2008 test data by separating the target variable and features, then encodes the features, predicts with the regressor, and computes the mean squared log error rounded to 6 decimals. Finally, appends the 2008 performance to the performance DataFrame and displays it.

```python
y_df_2008=house_df_2008['SalePrice'].to_frame()
X_df_2008=house_df_2008[house_df_2008.columns.difference(['SalePrice','YrSold'])]
```

```python
df_2008_encode=encoder.transform(X_df_2008)
y_pred_2008 = regressor.predict(df_2008_encode)
performance_2008 = mean_squared_log_error(y_df_2008, y_pred_2008).round(6)
performance_2008
```

```python
df_performance = pd.concat([df_performance, pd.DataFrame([{'annee': 2008, 'mois': 1, 'performance': performance_2008}])], ignore_index=True)
df_performance
```

--------------------------------

### Compile SmartDrift with %time

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/tutorial01-Eurybia-overview.ipynb.txt

Compile the SmartDrift object to compute drift metrics. The %time magic measures execution time; output shows the SHAP explainer type and timing (user 13.6 s, sys 3.54 s, total 17.2 s, wall 3.68 s).

```python
%time SD.compile()
```

--------------------------------

### Load house_prices dataset with eurybia.data.data_loader.data_loading

Source: https://eurybia.readthedocs.io/en/latest/autodocs/eurybia.data.html

Loads the house_prices dataset using the data_loading function from the eurybia data loader module. The function returns a pandas DataFrame and an optional dictionary of column labels.

```python
>>> from eurybia.data.data_loader import data_loading
>>> house_df, house_dict = data_loading('house_prices')

```

--------------------------------

### Create CatBoost Pools

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Creates CatBoost Pool objects for training and testing, specifying the categorical feature indices.

```python
train_pool_cat = catboost.Pool(data=X_train, label= y_train, cat_features = indice_cat)
test_pool_cat = catboost.Pool(data=X_test, label=y_test, cat_features = indice_cat)

```

--------------------------------

### Generate data drift report with SD.generate_report

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/tutorial01-Eurybia-overview.ipynb.txt

Generate an HTML drift report for the 2007 house price dataset. The output file is 'report_house_price_datadrift_2007.html', with a title story 'Data drift', an optional subtitle, and an optional project info file.

```python
SD.generate_report(    
    output_file='report_house_price_datadrift_2007.html',    
    title_story="Data drift",
    title_description="""House price Data drift 2007""", # Optional: add a subtitle to describe report
    project_info_file="../eurybia/data/project_info_house_price.yml" # Optional: add information on report
    )
```

--------------------------------

### Show head of current dataset

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_validation/tutorial01-data-validation.html

Displays the first five rows of the current (production) dataset.

```python
df_current.head()
```

--------------------------------

### Fit LGBMRegressor with 200 estimators

Source: https://eurybia.readthedocs.io/en/latest/tutorials/tutorial01-Eurybia-overview.html

Trains a LGBMRegressor on the training data. The displayed output shows LightGBM's training logs, including auto-choosing row-wise multi-threading and a warning about no further splits with positive gain. Optional memory-related settings include force_row_wise and force_col_wise.

```python
regressor = LGBMRegressor(n_estimators=200).fit(Xtrain,ytrain)

```

--------------------------------

### Create and compile SmartDrift with full validation

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common01-colors.html

Creates a SmartDrift object with production and learning dataframes, then compiles it with full validation. The compile output lists mismatching unique values for many variables and reports the SHAP explainer type and CPU/wall time.

```python
SD3 = SmartDrift(df_current=X_df_production, df_baseline=X_df_learning)
```

```python
%time SD3.compile(full_validation=True)
```

--------------------------------

### Split data by year for training and production

Source: https://eurybia.readthedocs.io/en/latest/tutorials/tutorial01-Eurybia-overview.html

Uses the 'YrSold' column as reference date. Selects 2006 data for training and 2007 data for production to detect data drift.

```python
# Let us consider that the column "YrSold" corresponds to the reference date.
#In 2006, a model was trained using data. And in 2007, we want to detect data drift on new data in production to predict
#house price
house_df_learning = house_df.loc[house_df['YrSold'] == 2006]
house_df_2007 = house_df.loc[house_df['YrSold'] == 2007]

```

--------------------------------

### Create SmartDrift, compile, and generate report

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Use Eurybia to create a SmartDrift object, compile it with full validation, and generate a drift report. The compile step outputs a Shap explainer type and timing information. The report is saved to 'report_titanic.html'.

```python
from eurybia import SmartDrift
```

```python
sd = SmartDrift(df_current=df_current, df_baseline=df_baseline, deployed_model=model, encoding=encoder)
```

```python
%time sd.compile(full_validation=True)
```

```python
sd.generate_report(
    output_file='report_titanic.html',
    title_story="Data validation",
    title_description="""Titanic Data validation"""
    )
```

--------------------------------

### Import SmartDrift from eurybia

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/common/tuto-common02-shapash-webapp.ipynb.txt

Imports the SmartDrift class from the eurybia library for drift detection and data validation.

```python
from eurybia import SmartDrift
```

--------------------------------

### Compile SmartDrift analysis with full validation

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common01-colors.html

Compiles the SmartDrift analysis with full validation enabled, timing the execution with the %time magic command.

```python
%time SD.compile(full_validation=True)

```

--------------------------------

### Instantiate SmartDrift for 2010 data

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial01-datadrift-over-years.html

Creates a SmartDrift object to compare the current 2010 data against the baseline learning data. The deployed model and encoder are optional but provide additional context for the drift analysis.

```python
SD = SmartDrift(df_current=X_df_2010,
                df_baseline=X_df_learning,
                deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
                encoding=encoder # Optional: if deployed_model and encoder to use this model
               )

```

--------------------------------

### Create SmartDrift with custom colors_dict and compile

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common01-colors.html

Creates a SmartDrift instance with custom colors for the feature importance bar, univariate categorical bar, and univariate continuous bar plots. The colors_dict parameter replaces the default palette. Then compiles the explainer with full validation, timing the execution.

```python
SD2 = SmartDrift(df_current=X_df_production, df_baseline=X_df_learning,
    colors_dict=dict(
        featureimp_bar={
            "1": "rgba(244, 192, 0, 1.0)",
            "2": "rgba(52, 192, 54, 0.7)",
            "3": "rgba(192, 55, 54, 0.7)",
            "4": "rgba(52, 55, 54, 0.7)"
            },
            univariate_cat_bar={
            "1": "rgba(244, 192, 0, 1.0)",
            "2": "rgba(52, 55, 54, 0.7)"
            },
            univariate_cont_bar={
            "1": "rgba(244, 192, 0, 1.0)",
            "2": "rgba(52, 55, 54, 0.7)"
            })
                )
```

```python
%time SD2.compile(full_validation=True)
```

--------------------------------

### Prepare target and feature dataframes per year

Source: https://eurybia.readthedocs.io/en/latest/tutorials/model_drift/tutorial02-modeldrift-high-datadrift.html

Separates target and feature data for baseline and each production year, excluding non-feature columns. This prepares the datasets for model training and drift detection.

```python
y_df_learning=df_accident_baseline['target'].to_frame()
X_df_learning=df_accident_baseline[df_accident_baseline.columns.difference(["target", "target_multi", "year_acc", "Description"])]

y_df_2017=df_accident_2017['target'].to_frame()
X_df_2017=df_accident_2017[df_accident_2017.columns.difference(["target", "target_multi", "year_acc", "Description"])]

y_df_2018=df_accident_2018['target'].to_frame()
X_df_2018=df_accident_2018[df_accident_2018.columns.difference(["target", "target_multi", "year_acc", "Description"])]

y_df_2019=df_accident_2019['target'].to_frame()
X_df_2019=df_accident_2019[df_accident_2019.columns.difference(["target", "target_multi", "year_acc", "Description"])]

y_df_2020=df_accident_2020['target'].to_frame()
X_df_2020=df_accident_2020[df_accident_2020.columns.difference(["target", "target_multi", "year_acc", "Description"])]

y_df_2021=df_accident_2021['target'].to_frame()
X_df_2021=df_accident_2021[df_accident_2021.columns.difference(["target", "target_multi", "year_acc", "Description"])]
```

--------------------------------

### Train LightGBM regressor with 200 estimators

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_validation/tutorial02-data-validation-iteration.html

Trains a LightGBM regressor with 200 estimators on the training data. The training output includes warnings about no further splits with positive gain, indicating the model may have reached its capacity.

```python
regressor = LGBMRegressor(n_estimators=200).fit(Xtrain,ytrain)
```

--------------------------------

### Create fake production dataset

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/common/tuto-common02-shapash-webapp.ipynb.txt

Creates a fake production dataset by copying the original Titanic data and randomly altering the Age, Fare, and Sex columns. The random module is imported, then the copy is made, and the three columns are modified using lambda functions with random ranges and choices.

```python
import random
```

```python
df_production = titan_df.copy()
```

```python
df_production['Age'] = df_production['Age'].apply(lambda x: random.randrange(10, 76)).astype(float)
df_production['Fare'] = df_production['Fare'].apply(lambda x: random.randrange(1, 100)).astype(float)
list_sex= ["male", "female"]
df_production['Sex'] = df_production['Sex'].apply(lambda x: random.choice(list_sex))
```

--------------------------------

### Fit model and predict

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_validation/tutorial01-data-validation.html

Trains the CatBoost model and generates predictions on the test set.

```python
model.fit(train_pool_cat, eval_set=test_pool_cat, silent=True)
y_pred = model.predict(X_test)
```

--------------------------------

### Create SmartDrift for 2010 with filled MasVnrType

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/model_drift/tutorial01-modeldrift.ipynb.txt

Fills missing values in the MasVnrType column with 'NR' for both 2010 and learning datasets, then creates a SmartDrift object with the current 2010 data, baseline learning data, the deployed regressor model, and the encoder. The output shows mismatching unique values for Condition1 and Electrical variables.

```python
X_df_2010["MasVnrType"] = X_df_2010["MasVnrType"].fillna("NR")
X_df_learning["MasVnrType"] = X_df_learning["MasVnrType"].fillna("NR")
SD = SmartDrift(df_current=X_df_2010,
                df_baseline=X_df_learning,
                deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
                encoding=encoder # Optional: if deployed_model and encoder to use this model
               )
```

--------------------------------

### Load data

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial02-datadrift-high-datadrift.html

Import the data_loading function from Eurybia and load the us_car_accident dataset.

```python
from eurybia.data.data_loader import data_loading
```

```python
df_car_accident = data_loading("us_car_accident")
```

--------------------------------

### Launch and stop Shapash WebApp from SmartDrift

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Launch a Shapash WebApp from the SmartDrift object after compilation. The app runs on a local HTTP server and prints its URL. Use the kill() method to stop the app after use.

```python
app = sd.xpl.run_app(title_story='Eurybia datadrift classifier')
```

```python
app.kill()
```

--------------------------------

### Create SmartDrift object for 2008

Source: https://eurybia.readthedocs.io/en/latest/tutorials/model_drift/tutorial01-modeldrift.html

Fills missing values in the 'MasVnrType' column for both current and baseline data, then creates a SmartDrift object with the 2008 data as current, the learning data as baseline, and optionally the deployed model and encoder.

```python
X_df_2008["MasVnrType"] = X_df_2008["MasVnrType"].fillna("NR")
X_df_learning["MasVnrType"] = X_df_learning["MasVnrType"].fillna("NR")
SD = SmartDrift(df_current=X_df_2008,
                df_baseline=X_df_learning,
                deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
                encoding=encoder # Optional: if deployed_model and encoder to use this model
               )

```

--------------------------------

### Select features for baseline and current datasets

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Selects the features columns from the baseline and production datasets for comparison.

```python
df_baseline = titan_df[features]
df_current = df_production[features]

```

--------------------------------

### Import data_loading from Eurybia

Source: https://eurybia.readthedocs.io/en/latest/tutorials/common/tuto-common02-shapash-webapp.html

Imports the data_loading function from Eurybia to load the Titanic dataset.

```python
from eurybia.data.data_loader import data_loading

```

--------------------------------

### Generate model drift report

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/model_drift/tutorial01-modeldrift.ipynb.txt

Generates a report for the house price model drift and saves it to the specified HTML file.

```python
SD.generate_report(    
    output_file='report_house_price_modeldrift_2008.html',    
```

--------------------------------

### Compile SmartDrift with full validation

Source: https://eurybia.readthedocs.io/en/latest/_sources/tutorials/common/tuto-common02-shapash-webapp.ipynb.txt

Compiles the SmartDrift object with full validation enabled, timing the execution. Output shows the Shap explainer type and CPU/wall time.

```python
%time sd.compile(full_validation=True)
```

--------------------------------

### Create SmartDrift object for 2009 data

Source: https://eurybia.readthedocs.io/en/latest/tutorials/data_drift/tutorial01-datadrift-over-years.html

Instantiates a SmartDrift object with the 2009 data as current and X_df_learning as baseline. The deployed_model and encoding parameters are optional; they are used to put the drift in perspective with the deployed model's importance.

```python
SD = SmartDrift(df_current=X_df_2009,
                df_baseline=X_df_learning,
                deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
                encoding=encoder # Optional: if deployed_model and encoder to use this model
                )

```