### Install AutoFeat

Source: https://github.com/cod3licious/autofeat/blob/main/docs/index.md

Install the AutoFeat library components using pip. This command installs the library and its dependencies.

```bash
$ pip install autofeat
```

--------------------------------

### Install Autofeat with Pip

Source: https://github.com/cod3licious/autofeat/blob/main/README.md

Install the Autofeat library using pip. Ensure you have Python and pip installed.

```bash
pip install autofeat
```

--------------------------------

### Import Libraries for Autofeat

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Imports necessary libraries for data manipulation, machine learning, and Autofeat. Ensure all dependencies are installed.

```python
import os
import sys
import warnings

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer, load_iris, load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

from autofeat import AutoFeatClassifier

%load_ext autoreload
%autoreload 2
```

--------------------------------

### Define Datasets for Autofeat

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Specifies the names of the datasets to be used in the examples. These correspond to datasets available in scikit-learn.

```python
datasets = ["iris", "wine", "breast_cancer"]
```

--------------------------------

### Import necessary libraries for Autofeat

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Imports standard Python libraries for data science and machine learning, along with the Autofeat library. Ensure these libraries are installed before running.

```python
import os
import sys
import warnings

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing, load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR

from autofeat import AutoFeatRegressor

%load_ext autoreload
%autoreload 2
```

--------------------------------

### Get Autofeat Dataset

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Generates a dataset with specified true features and noise features, then scales and transforms the target variable. Supports regression and classification.

```python
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler

def get_autofeat_dataset(n_feat_noise=100, noise=0.01, ptype="regression", random_seed=None):
    if random_seed is not None:
        np.random.seed(random_seed)
    true_features = ["x1", "(x2 + log(x1))**3", "1/(x2 - 1/x3)"]
    noise_features = list(np.random.permutation(list(set(df.columns).difference(true_features)))[:n_feat_noise])
    X = StandardScaler().fit_transform(df[true_features + noise_features])
    y = target - target.mean()
    y /= y.std()
    if ptype == "regression":
        # add normally distributed noise
        y = (1 - noise) * y + noise * np.random.randn(len(y))
    else:
        # threshold to transform into classification problem
        y = np.array(y > y.mean(), dtype=int)
        # randomly flip some idx
        flip_idx = np.random.permutation(len(y))[: int(np.ceil(len(y) * noise))]
        y[flip_idx] -= 1
        y = np.abs(y)
    return X, y, [0, 1, 2]
```

--------------------------------

### Integrate AutoFeat FeatureSelector with Scikit-learn Pipelines

Source: https://context7.com/cod3licious/autofeat/llms.txt

AutoFeat models follow the scikit-learn API and can be seamlessly integrated into pipelines. This example demonstrates using `FeatureSelector` within a `Pipeline` for regression tasks, including scaling and accessing selected features.

```python
import numpy as np
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from autofeat import FeatureSelector

# Sample data
np.random.seed(42)
X = pd.DataFrame(np.random.rand(500, 10), columns=[f'f{i}' for i in range(10)])
y = 3 * X['f0'] + 2 * X['f1'] ** 2 + np.random.randn(500) * 0.1

# Create pipeline with feature selector
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('featsel', FeatureSelector(problem_type='regression', featsel_runs=3, verbose=0))
])

# Fit pipeline
X_selected = pipeline.fit_transform(X, y)
print(f"Selected {X_selected.shape[1]} features from {X.shape[1]}")

# Access selected features from pipeline
fsel = pipeline.named_steps['featsel']
print(f"Selected columns: {fsel.good_cols_}")
```

--------------------------------

### Initialize and Fit AutoFeatLight

Source: https://context7.com/cod3licious/autofeat/llms.txt

Initializes AutoFeatLight for lightweight feature engineering, including ratios and products. Use this for quick feature generation without extensive computation.

```python
afl = AutoFeatLight(
    compute_ratio=True,        # Compute x/y features
    compute_product=True,      # Compute x*y features
    scale=False,               # Apply StandardScaler
    power_transform=False,     # Apply Yeo-Johnson power transform
    corrthr=0.995,             # Correlation threshold for redundancy
    corrthr_init=0.99999,      # Initial correlation threshold
    verbose=1
)

# Fit and transform
X_transformed = afl.fit_transform(X)
print(f"Original features: {X.shape[1]}")
print(f"Transformed features: {X_transformed.shape[1]}")
print(f"Feature names: {afl.features_[:10]}...")  # First 10 features
```

--------------------------------

### AutoFeatLight with Power Transform

Source: https://context7.com/cod3licious/autofeat/llms.txt

Initializes AutoFeatLight with power transformation enabled for more normally distributed features. This can improve model performance for algorithms sensitive to feature distribution.

```python
afl_powered = AutoFeatLight(
    compute_ratio=True,
    compute_product=True,
    power_transform=True,
    verbose=1
)
X_powered = afl_powered.fit_transform(X)
```

--------------------------------

### Import Libraries and Load Extensions

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_examples.ipynb

Imports necessary libraries like matplotlib, numpy, pandas, and autofeat. Also loads IPython extensions for autoreload.

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from autofeat import AutoFeatRegressor, FeatureSelector

%load_ext autoreload
%autoreload 2
```

--------------------------------

### Load Classification Datasets

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Loads standard scikit-learn classification datasets. Supports 'iris', 'wine', and 'breast_cancer'. Raises an error for unknown dataset names.

```python
def load_classification_dataset(name):
    # load one of the datasets as X and y
    units = {}
    if name == "iris":
        # sklearn iris housing dataset
        X, y = load_iris(return_X_y=True)

    elif name == "wine":
        # sklearn wine dataset
        X, y = load_wine(return_X_y=True)

    elif name == "breast_cancer":
        # sklearn breast_cancer dataset
        X, y = load_breast_cancer(return_X_y=True)

    else:
        raise RuntimeError(f"Unknown dataset {name}")
    return np.array(X, dtype=float), np.array(y, dtype=float), units
```

--------------------------------

### AutoFeat with Varying Feature Engineering Steps

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_examples.ipynb

Demonstrates the effect of the `feateng_steps` parameter in AutoFeatRegressor on model performance (R^2 score) and the number of new features generated. Visualizes predictions against the target.

```python
# autofeat with different number of feature engineering steps
# 3 are perfect
for steps in range(5):
    np.random.seed(55)
    print("### AutoFeat with %i feateng_steps" % steps)
    afreg = AutoFeatRegressor(verbose=1, feateng_steps=steps)
    df = afreg.fit_transform(df_org, target)
    r2 = afreg.score(df_org, target)
    print("## Final R^2: %.4f" % r2)
    plt.figure()
    plt.scatter(afreg.predict(df_org), target, s=2)
    plt.title("%i FE steps (R^2: %.4f; %i new features)" % (steps, r2, len(afreg.new_feat_cols_)))
```

--------------------------------

### Test Autofeat with 3 Feature Engineering Steps

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Applies feature engineering with three steps for each dataset, enabling the creation of highly complex features.

```python
for dsname in datasets:
    print("####", dsname)
    test_autofeat(dsname, feateng_steps=3)
```

--------------------------------

### Create Precision and Recall Plots

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Generates various plots to visualize precision and recall metrics against different noise levels and feature fractions. Requires pre-computed precision and recall data.

```python
def create_plots(precision, recall, noise_levels, noise_feat_frac, n_train, n_feat_true):
    colors = get_colors(len(noise_levels))
    plt.figure()
    for i, noise in enumerate(noise_levels):
        plt.plot(noise_feat_frac, precision[i, :], c=colors[i], label="noise: %g" % noise)
    plt.legend(bbox_to_anchor=(1.0, 1.0))
    plt.xlabel("# noise features / # training samples ($n$)")
    plt.title("Precision ($n$: %i; $d$: %i)" % (n_train, n_feat_true))
    plt.figure()
    for i, noise in enumerate(noise_levels):
        plt.plot(noise_feat_frac, recall[i, :], c=colors[i], label="noise: %g" % noise)
    plt.legend(bbox_to_anchor=(1.0, 1.0))
    plt.xlabel("# noise features / # training samples ($n$)")
    plt.title("Recall ($n$: %i; $d$: %i)" % (n_train, n_feat_true))
    colors = get_colors(len(noise_feat_frac))
    plt.figure()
    for i, nfeat in enumerate(noise_feat_frac):
        plt.plot(noise_levels, precision[:, i], c=colors[i], label="noise feat frac: %g" % nfeat)
    plt.legend(bbox_to_anchor=(1.0, 1.0))
    plt.xlabel("noise level")
    plt.title("Precision ($n$: %i; $d$: %i)" % (n_train, n_feat_true))
    plt.figure()
    for i, nfeat in enumerate(noise_feat_frac):
        plt.plot(noise_levels, recall[:, i], c=colors[i], label="noise feat frac: %g" % nfeat)
    plt.legend(bbox_to_anchor=(1.0, 1.0))
    plt.xlabel("noise level")
    plt.title("Recall ($n$: %i; $d$: %i)" % (n_train, n_feat_true))
    plt.figure()
    plt.imshow(precision)
    plt.xticks(list(range(len(noise_feat_frac))), noise_feat_frac)
    plt.xlabel("# noise features / # training samples ($n$)")
    plt.yticks(list(range(len(noise_levels))), noise_levels)
    plt.ylabel("noise level")
    plt.clim(0, 1)
    plt.colorbar()
    plt.title("Precision ($n$: %i; $d$: %i)" % (n_train, n_feat_true))
    plt.figure()
    plt.imshow(recall)
    plt.xticks(list(range(len(noise_feat_frac))), noise_feat_frac)
    plt.xlabel("# noise features / # training samples ($n$)")
    plt.yticks(list(range(len(noise_levels))), noise_levels)
    plt.ylabel("noise level")
    plt.clim(0, 1)
    plt.colorbar()
    plt.title("Recall ($n$: %i; $d$: %i)" % (n_train, n_feat_true))
```

--------------------------------

### Test Autofeat with 2 Feature Engineering Steps

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Applies feature engineering with two steps for each dataset. This allows for a more complex set of generated features.

```python
for dsname in datasets:
    print("####", dsname)
    test_autofeat(dsname, feateng_steps=2)
```

--------------------------------

### Feature Selection with AutoFeat

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_examples.ipynb

Generates toy data and uses FeatureSelector to identify relevant features. The output should indicate the selected columns.

```python
# generate some toy data
np.random.seed(10)
x1 = np.random.rand(1000)
x2 = np.random.randn(1000)
x3 = np.random.rand(1000)
x4 = np.random.randn(1000)
x5 = np.random.rand(1000)
target = 2 + 15 * x1 + 3 / (x2 - 1 / x3) + 5 * (x2 + np.log(x1)) ** 3
X = np.vstack([x1, x2, x3, x4, x5, 1 / (x2 - 1 / x3), (x2 + np.log(x1)) ** 3]).T
fsel = FeatureSelector(verbose=1)
new_X = fsel.fit_transform(pd.DataFrame(X, columns=["x1", "x2", "x3", "x4", "x5", "eng6", "eng7"])), target)
# should contain ["x1", "eng6", "eng7"]
print(new_X.columns)
```

--------------------------------

### Load and Print Dataset Shapes

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Iterates through a list of datasets, loads each one, and prints the shape of the feature matrix X.

```python
for dsname in datasets:
    print("####", dsname)
    X, y, _ = load_regression_dataset(dsname)
    print(X.shape)
```

--------------------------------

### Initialize Random Forest Regressor

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Initializes a RandomForestRegressor with a fixed number of estimators and a random state for reproducibility.

```python
for dsname in datasets:
    print("####", dsname)
    rforest = RandomForestRegressor(n_estimators=100, random_state=13)

```

--------------------------------

### Test Support Vector Regression Across Datasets

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Applies a Support Vector Regression model with specified C parameters to multiple datasets, skipping 'california_housing' due to long processing time.

```python
for dsname in datasets:
    if dsname == "california_housing":
        # takes too long because too many data points
        continue
    print("####", dsname)
    svr = SVR(gamma="scale")
    params = {"C": [1.0, 10.0, 25.0, 50.0, 100.0, 250.0]}
    svr = test_model(dsname, svr, params)
```

--------------------------------

### Iterate and Load Classification Datasets

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Iterates through a list of dataset names, loads each one using `load_classification_dataset`, and prints their shapes and unique target values.

```python
for dsname in datasets:
    print("####", dsname)
    X, y, _ = load_classification_dataset(dsname)
    print(X.shape, np.unique(y))
```

--------------------------------

### Compute Autofeat Dataset

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Generates a dataset with synthetic features and then applies Autofeat for feature engineering. Requires numpy and pandas.

```python
import numpy as np
import pandas as pd
from autofeat import AutoFeatModel

def compute_autofeat_dataset(n_train=100):
    np.random.seed(10)
    x1 = np.random.rand(n_train)
    x2 = np.random.randn(n_train)
    x3 = np.random.rand(n_train)
    y = 2 + 15 * x1 + 3 / (x2 - 1 / x3) + 5 * (x2 + np.log(x1)) ** 3
    X = pd.DataFrame(np.vstack([x1, x2, x3]).T, columns=["x1", "x2", "x3"])
    # generate new features with autofeat
    afreg = AutoFeatModel(verbose=1, feateng_steps=3, featsel_runs=-1, problem_type=None)
    X = afreg.fit_transform(X, y)
    return X, y
```

--------------------------------

### Autofeat Regression with Ridge and Random Forest

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Demonstrates using AutoFeatRegressor for feature engineering and then training Ridge Regression and Random Forest models with hyperparameter tuning.

```python
def test_autofeat(dataset, feateng_steps=2):
    # load data
    X, y, units = load_regression_dataset(dataset)
    # split in training and test parts
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=12)
    # run autofeat
    afreg = AutoFeatRegressor(verbose=1, feateng_steps=feateng_steps, units=units)
    # fit autofeat on less data, otherwise ridge reg model with xval will overfit on new features
    X_train_tr = afreg.fit_transform(X_train, y_train)
    X_test_tr = afreg.transform(X_test)
    print("autofeat new features:", len(afreg.new_feat_cols_))
    print("autofeat MSE on training data:", mean_squared_error(y_train, afreg.predict(X_train_tr)))
    print("autofeat MSE on test data:", mean_squared_error(y_test, afreg.predict(X_test_tr)))
    print("autofeat R^2 on training data:", r2_score(y_train, afreg.predict(X_train_tr)))
    print("autofeat R^2 on test data:", r2_score(y_test, afreg.predict(X_test_tr)))
    # train rreg on transformed train split incl cross-validation for parameter selection
    print("# Ridge Regression")
    rreg = Ridge()
    param_grid = {
        "alpha": [
            0.00001,
            0.0001,
            0.001,
            0.01,
            0.1,
            1.0,
            2.5,
            5.0,
            10.0,
            25.0,
            50.0,
            100.0,
            250.0,
            500.0,
            1000.0,
            2500.0,
            5000.0,
            10000.0,
        ]
    }
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        gsmodel = GridSearchCV(rreg, param_grid, scoring="neg_mean_squared_error", cv=5)
        gsmodel.fit(X_train_tr, y_train)
    print("best params:", gsmodel.best_params_)
    print("best score:", gsmodel.best_score_)
    print("MSE on training data:", mean_squared_error(y_train, gsmodel.predict(X_train_tr)))
    print("MSE on test data:", mean_squared_error(y_test, gsmodel.predict(X_test_tr)))
    print("R^2 on training data:", r2_score(y_train, gsmodel.predict(X_train_tr)))
    print("R^2 on test data:", r2_score(y_test, gsmodel.predict(X_test_tr)))
    print("# Random Forest")
    rforest = RandomForestRegressor(n_estimators=100, random_state=13)
    param_grid = {"min_samples_leaf": [0.0001, 0.001, 0.01, 0.05, 0.1, 0.2]}
    gsmodel = GridSearchCV(rforest, param_grid, scoring="neg_mean_squared_error", cv=5)
    gsmodel.fit(X_train_tr, y_train)
    print("best params:", gsmodel.best_params_)
    print("best score:", gsmodel.best_score_)
    print("MSE on training data:", mean_squared_error(y_train, gsmodel.predict(X_train_tr)))
    print("MSE on test data:", mean_squared_error(y_test, gsmodel.predict(X_test_tr)))
    print("R^2 on training data:", r2_score(y_train, gsmodel.predict(X_train_tr)))
    print("R^2 on test data:", r2_score(y_test, gsmodel.predict(X_test_tr)))

```

--------------------------------

### Train and Evaluate Regression Model with GridSearchCV

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Trains a regression model using GridSearchCV for parameter selection and evaluates its performance on training and test data. Includes optional scaling for SVR.

```python
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=12)
if model.__class__.__name__ == "SVR":
    sscaler = StandardScaler()
    X_train = sscaler.fit_transform(X_train)
    X_test = sscaler.transform(X_test)
# train model on train split incl cross-validation for parameter selection
gsmodel = GridSearchCV(model, param_grid, scoring="neg_mean_squared_error", cv=5)
gsmodel.fit(X_train, y_train)
print("best params:", gsmodel.best_params_)
print("best score:", gsmodel.best_score_)
print("MSE on training data:", mean_squared_error(y_train, gsmodel.predict(X_train)))
print("MSE on test data:", mean_squared_error(y_test, gsmodel.predict(X_test)))
print("R^2 on training data:", r2_score(y_train, gsmodel.predict(X_train)))
print("R^2 on test data:", r2_score(y_test, gsmodel.predict(X_test)))
return gsmodel.best_estimator_
```

--------------------------------

### Import Libraries for Autofeat Classification

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Imports necessary libraries for data manipulation, modeling, and visualization in Python. Includes standard scientific computing and machine learning libraries.

```python
import colorsys
import warnings
from time import time

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn.linear_model as lm
from sklearn import svm
from sklearn.datasets import make_classification
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler

from autofeat import AutoFeatModel

%matplotlib inline
%load_ext autoreload
%autoreload 2
```

--------------------------------

### Load Regression Dataset

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Loads a specified regression dataset. Supports California Housing, Diabetes, Concrete, Forest Fires, Wine Quality, and Airfoil. Returns features (X), target (y), and units.

```python
def load_regression_dataset(name, datapath="../my_datasets/regression/"):
    # load one of the datasets as X and y (and possibly units)
    units = {}
    if name == "california_housing":
        # sklearn california housing dataset
        X, y = fetch_california_housing(return_X_y=True)

    elif name == "diabetes":
        # sklearn diabetes dataset
        X, y = load_diabetes(return_X_y=True)

    elif name == "concrete":
        # https://archive.ics.uci.edu/ml/datasets/Concrete+Compressive+Strength
        # Cement (component 1) -- quantitative -- kg in a m3 mixture -- Input Variable
        # Blast Furnace Slag (component 2) -- quantitative -- kg in a m3 mixture -- Input Variable
        # Fly Ash (component 3) -- quantitative -- kg in a m3 mixture -- Input Variable
        # Water (component 4) -- quantitative -- kg in a m3 mixture -- Input Variable
        # Superplasticizer (component 5) -- quantitative -- kg in a m3 mixture -- Input Variable
        # Coarse Aggregate (component 6) -- quantitative -- kg in a m3 mixture -- Input Variable
        # Fine Aggregate (component 7)    -- quantitative -- kg in a m3 mixture -- Input Variable
        # Age -- quantitative -- Day (1~365) -- Input Variable
        # Concrete compressive strength -- quantitative -- MPa -- Output Variable
        df = pd.read_csv(os.path.join(datapath, "concrete.csv"))
        X = df.iloc[:, :8].to_numpy()
        y = df.iloc[:, 8].to_numpy()

    elif name == "forest_fires":
        # https://archive.ics.uci.edu/ml/datasets/Forest+Fires
        # 1. X - x-axis spatial coordinate within the Montesinho park map: 1 to 9
        # 2. Y - y-axis spatial coordinate within the Montesinho park map: 2 to 9
        # 3. month - month of the year: 'jan' to 'dec'
        # 4. day - day of the week: 'mon' to 'sun'
        # 5. FFMC - FFMC index from the FWI system: 18.7 to 96.20
        # 6. DMC - DMC index from the FWI system: 1.1 to 291.3
        # 7. DC - DC index from the FWI system: 7.9 to 860.6
        # 8. ISI - ISI index from the FWI system: 0.0 to 56.10
        # 9. temp - temperature in Celsius degrees: 2.2 to 33.30
        # 10. RH - relative humidity in %: 15.0 to 100
        # 11. wind - wind speed in km/h: 0.40 to 9.40
        # 12. rain - outside rain in mm/m2 : 0.0 to 6.4
        # 13. area - the burned area of the forest (in ha): 0.00 to 1090.84
        # (this output variable is very skewed towards 0.0, thus it may make sense to model with the logarithm transform).
        # --> first 4 are ignored
        df = pd.read_csv(os.path.join(datapath, "forest_fires.csv"))
        X = df.iloc[:, 4:12].to_numpy()
        y = df.iloc[:, 12].to_numpy()
        # perform transformation as they suggested
        y = np.log(y + 1)

    elif name == "wine_quality":
        # https://archive.ics.uci.edu/ml/datasets/Wine+Quality
        # Input variables (based on physicochemical tests):
        # 1 - fixed acidity
        # 2 - volatile acidity
        # 3 - citric acid
        # 4 - residual sugar
        # 5 - chlorides
        # 6 - free sulfur dioxide
        # 7 - total sulfur dioxide
        # 8 - density
        # 9 - pH
        # 10 - sulphates
        # 11 - alcohol
        # Output variable (based on sensory data):
        # 12 - quality (score between 0 and 10)
        df_red = pd.read_csv(os.path.join(datapath, "winequality-red.csv"), sep=";")
        df_white = pd.read_csv(os.path.join(datapath, "winequality-white.csv"), sep=";")
        # add additional categorical feature for red or white
        X = np.hstack(
            [
                np.vstack([df_red.iloc[:, :-1].to_numpy(), df_white.iloc[:, :-1].to_numpy()]),
                np.array([[1] * len(df_red) + [0] * len(df_white)]).T,
            ]
        )
        y = np.hstack([df_red["quality"].to_numpy(), df_white["quality"].to_numpy()])

    elif name == "airfoil":
        # https://archive.ics.uci.edu/ml/datasets/Airfoil+Self-Noise
        # This problem has the following inputs:
        # 1. Frequency, in Hertz.
        # 2. Angle of attack, in degrees.
        # 3. Chord length, in meters.
        # 4. Free-stream velocity, in meters per second.
        # 5. Suction side displacement thickness, in meters.
        # The only output is:
        # 6. Scaled sound pressure level, in decibels.
        units = {"x001": "Hz", "x003": "m", "x004": "m/sec", "x005": "m"}
        df = pd.read_csv(
            os.path.join(datapath, "airfoil_self_noise.tsv"), header=None, names=["x1", "x2", "x3", "x4", "x5", "y"], sep="\t"
        )
        X = df.iloc[:, :5].to_numpy()
        y = df["y"].to_numpy()

    else:
        raise RuntimeError("Unknown dataset %r" % name)
    return np.array(X, dtype=float), np.array(y, dtype=float), units
```

--------------------------------

### Initialize and Fit FeatureSelector

Source: https://context7.com/cod3licious/autofeat/llms.txt

Initializes the FeatureSelector for regression tasks and fits it to the data to select the best features. Use this to reduce dimensionality and improve model performance.

```python
fsel = FeatureSelector(
    problem_type="regression",  # "regression" or "classification"
    featsel_runs=5,             # Number of selection runs with random subsampling
    keep=None,                  # List of features to always keep
    n_jobs=1,                   # Parallel jobs
    verbose=1
)

# Fit and transform - returns DataFrame with selected features only
X_selected = fsel.fit_transform(X, target)
print(f"Original features: {X.shape[1]}")
print(f"Selected features: {X_selected.shape[1]}")
print(f"Good columns: {fsel.good_cols_}")
```

--------------------------------

### Initialize AutoFeatRegressor with Units for Pi Theorem

Source: https://context7.com/cod3licious/autofeat/llms.txt

Initializes AutoFeatRegressor with physical units defined for dimensional analysis using the Pi theorem. This ensures only dimensionally consistent features are generated.

```python
units = {
    'mass': 'kg',
    'velocity': 'm/s',
    'length': 'm',
    'time': 's'
}

# Initialize with units
afreg = AutoFeatRegressor(
    units=units,
    apply_pi_theorem=True,     # Apply dimensional analysis
    feateng_steps=2,
    verbose=1
)

# Fit - only dimensionally consistent features will be generated
df_transformed = afreg.fit_transform(X, target)
print(f"Features with dimensional analysis: {len(df_transformed.columns)}")
```

--------------------------------

### Define dataset names for Autofeat

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Lists the names of datasets that can be used with the Autofeat library. This list is used to configure data loading.

```python
datasets = ["diabetes", "california_housing", "concrete", "airfoil", "wine_quality"]


# same interface for loading all datasets - adapt the datapath
```

--------------------------------

### Test Machine Learning Models with Cross-Validation

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Tests a given model on a specified dataset, including data splitting, optional scaling for SVC, and GridSearchCV for parameter tuning. Prints evaluation metrics.

```python
def test_model(dataset, model, param_grid):
    # load data
    X, y, _ = load_classification_dataset(dataset)
    # split in training and test parts
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=12)
    if model.__class__.__name__ == "SVC":
        sscaler = StandardScaler()
        X_train = sscaler.fit_transform(X_train)
        X_test = sscaler.transform(X_test)
    # train model on train split incl cross-validation for parameter selection
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        gsmodel = GridSearchCV(model, param_grid, cv=5)
        gsmodel.fit(X_train, y_train)
    print("best params:", gsmodel.best_params_)
    print("best score:", gsmodel.best_score_)
    print("Acc. on training data:", accuracy_score(y_train, gsmodel.predict(X_train)))
    print("Acc. on test data:", accuracy_score(y_test, gsmodel.predict(X_test)))
    return gsmodel.best_estimator_
```

--------------------------------

### Compute Original Autofeat Dataset

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Computes the original autofeat dataset with all features using the compute_autofeat_dataset function. Requires numpy and pandas.

```python
# compute the original autofeat dataset with all features
df, target = compute_autofeat_dataset(n_train=100)
```

--------------------------------

### Test Model with Parameter Tuning

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Tunes a random forest model by iterating through different minimum leaf sample parameters. This snippet is part of a larger testing function.

```python
params = {"min_samples_leaf": [0.0001, 0.001, 0.01, 0.05, 0.1, 0.2]}
    rforest = test_model(dsname, rforest, params)
```

--------------------------------

### Evaluate Autofeat Features Performance

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Evaluates the performance of autofeat features across various noise levels and feature fractions using LassoLarsCV for regression. Requires numpy, time, warnings, and sklearn.linear_model.

```python
import numpy as np
import warnings
from time import time
from sklearn import linear_model as lm

# autofeat features
ptype = "regression"
t0 = time()
n_train = 100
n_feat_true = 3
noise_levels = [0.0, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 0.75, 1.0]
noise_feat_frac = [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 25]
precision = np.zeros((len(noise_levels), len(noise_feat_frac)))
recall = np.zeros((len(noise_levels), len(noise_feat_frac)))
for i, noise in enumerate(noise_levels):
    print("noise level: %g" % noise)
    for j, nfeat in enumerate(noise_feat_frac):
        ps, rs = [], []
        for seed in range(10):
            X, y, true_features = get_autofeat_dataset(int(nfeat * n_train), noise, ptype, random_seed=seed)
            with warnings.catch_warnings():
                warnings.simplefilter("ignore")
                if ptype == "regression":
                    # model = lm.OrthogonalMatchingPursuitCV(cv=5, max_iter=min(X.shape[1], 20)).fit(X, y)
                    model = lm.LassoLarsCV(cv=5, eps=1e-5).fit(X, y)
                    coefs = model.coef_
                else:
                    model = lm.LogisticRegressionCV(cv=5, penalty="l1", solver="saga", class_weight="balanced").fit(X, y)
                    coefs = np.max(np.abs(model.coef_), axis=0)
                selected_features = np.where(coefs > 1e-8)[0]
            p, r = prec_rec_autofeat(selected_features, true_features)
            ps.append(p)
            rs.append(r)
        precision[i, j] = np.mean(ps)
        recall[i, j] = np.mean(rs)
print("took %.1f sec" % (time() - t0))

create_plots(precision, recall, noise_levels, noise_feat_frac, n_train, n_feat_true)
```

--------------------------------

### AutoFeat with Noisy Target Data

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_examples.ipynb

Trains an AutoFeatRegressor with `feateng_steps=3` on noisy target data and evaluates its performance on the original target data. Visualizes predictions.

```python
afreg = AutoFeatRegressor(verbose=1, feateng_steps=3)
# train on noisy data
df = afreg.fit_transform(df_org, target_noisy)
# test on real targets
print("Final R^2: %.4f" % afreg.score(df, target))
plt.figure()
plt.scatter(afreg.predict(df), target, s=2);
```

--------------------------------

### Test Autofeat with 1 Feature Engineering Step

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Iterates through specified datasets and runs the `test_autofeat` function with `feateng_steps` set to 1, applying a single step of automatic feature engineering.

```python
for dsname in datasets:
    print("####", dsname)
    test_autofeat(dsname, feateng_steps=1)
```

--------------------------------

### Generate Synthetic Dataset

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Creates a synthetic dataset for regression or classification tasks. It generates features, defines true features, and adds noise to the target variable.

```python
def get_dataset(n_train=100, n_feat_noise=100, n_feat_true=10, noise=0.01, ptype="regression", random_seed=None):
    if random_seed is not None:
        np.random.seed(random_seed)
    # create a simple problem with 1 target variable,
    # generated as a linear combination of random features
    # and include some additional noise filters
    X = np.random.randn(n_train, n_feat_noise + n_feat_true)
    true_features = np.random.permutation(n_feat_noise + n_feat_true)[:n_feat_true]
    y = X[:, true_features].sum(axis=1)
    if ptype == "regression":
        # add normally distributed noise
        y -= y.mean()
        y /= y.std()
        y = (1 - noise) * y + noise * np.random.randn(len(y))
        # y = y/y.std() + noise*np.random.randn(len(y))
    else:
        # threshold to transform into classification problem
        y = np.array(y > y.mean(), dtype=int)
        # randomly flip some idx
        flip_idx = np.random.permutation(len(y))[: int(np.ceil(len(y) * noise))]
        y[flip_idx] -= 1
        y = np.abs(y)
```

--------------------------------

### Transform New Data with AutoFeatLight

Source: https://context7.com/cod3licious/autofeat/llms.txt

Transforms new data using a fitted AutoFeatLight model. This applies the learned feature engineering steps to new samples.

```python
# Transform new data
X_new = pd.DataFrame({
    'a': np.random.rand(100),
    'b': np.random.randn(100),
    'c': np.random.rand(100) + 1,
    'd': np.random.randn(100)
})
X_new_transformed = afl.transform(X_new)
```

--------------------------------

### Data Generation for AutoFeat

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_examples.ipynb

Generates toy data for use with AutoFeatRegressor, including original features and a target variable.

```python
# generate some toy data
np.random.seed(10)
x1 = np.random.rand(1000)
x2 = np.random.randn(1000)
x3 = np.random.rand(1000)
target = 2 + 15 * x1 + 3 / (x2 - 1 / x3) + 5 * (x2 + np.log(x1)) ** 3
target_noisy = target + 0.01 * target.std() * np.random.randn(1000)
target_very_noisy = target + 0.1 * target.std() * np.random.randn(1000)
X = np.vstack([x1, x2, x3]).T
df_org = pd.DataFrame(X, columns=["x1", "x2", "x3"])
```

--------------------------------

### AutoFeat with Very Noisy Target Data

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_examples.ipynb

Trains an AutoFeatRegressor with `feateng_steps=3` on very noisy target data and evaluates its performance on the original target data. Visualizes predictions.

```python
afreg = AutoFeatRegressor(verbose=1, feateng_steps=3)
# train on noisy data
df = afreg.fit_transform(df_org, target_very_noisy)
# test on real targets
print("Final R^2: %.4f" % afreg.score(df, target))
plt.figure()
plt.scatter(afreg.predict(df), target, s=2);
```

--------------------------------

### Test Support Vector Classifier (SVC) on Datasets

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Iterates through specified datasets and tests an SVC model using the `test_model` function with a parameter grid for the 'C' hyperparameter.

```python
for dsname in datasets:
    print("####", dsname)
    svc = SVC(gamma="scale", class_weight="balanced")
    params = {"C": [1.0, 10.0, 25.0, 50.0, 100.0, 250.0]}
    svc = test_model(dsname, svc, params)
```

--------------------------------

### Test Model Function

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

A placeholder function for testing a model with a given dataset and parameter grid. It loads the dataset and splits it into training and test sets.

```python
def test_model(dataset, model, param_grid):
    # load data
    X, y, _ = load_regression_dataset(dataset)
    # split in training and test parts
    
```

--------------------------------

### Train Linear SVC with GridSearchCV

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Trains a Linear Support Vector Classifier with L1 penalty using GridSearchCV for hyperparameter tuning. The dual parameter is set to False for efficiency with many samples. Ignores warnings during fitting.

```python
%%time
with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    param_grid = {"C": np.logspace(-4, 4, 10)}
    clf = svm.LinearSVC(penalty="l1", class_weight="balanced", dual=False)
    model = GridSearchCV(clf, param_grid, cv=5)
    model.fit(X, y)
    print(model.score(X, y))
```

--------------------------------

### Test Ridge Regression Across Datasets

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_regression.ipynb

Applies a Ridge Regression model with a wide range of alpha parameters to multiple datasets using the test_model function.

```python
for dsname in datasets:
    print("####", dsname)
    rreg = Ridge()
    params = {
        "alpha": [
            0.00001,
            0.0001,
            0.001,
            0.01,
            0.1,
            1.0,
            2.5,
            5.0,
            10.0,
            25.0,
            50.0,
            100.0,
            250.0,
            500.0,
            1000.0,
            2500.0,
            5000.0,
            10000.0,
            25000.0,
            50000.0,
            100000.0,
        ]
    }
    rreg = test_model(dsname, rreg, params)
```

--------------------------------

### Generate Color Palette

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Generates a list of distinct RGB colors using the HSV color space. Useful for plotting multiple series with distinguishable colors.

```python
def get_colors(n=100):
    HSV_tuples = [(x * 1.0 / (n + 1), 1.0, 0.8) for x in range(n)]
    return [colorsys.hsv_to_rgb(*x) for x in HSV_tuples]
```

--------------------------------

### Test Autofeat for Feature Engineering

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Applies Autofeat for feature engineering on a classification dataset and evaluates its performance. It then trains and evaluates Logistic Regression and Random Forest models on the transformed data.

```python
def test_autofeat(dataset, feateng_steps=2):
    # load data
    X, y, units = load_classification_dataset(dataset)
    # split in training and test parts
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=12)
    # run autofeat
    afreg = AutoFeatClassifier(verbose=1, feateng_steps=feateng_steps, units=units)
    # fit autofeat on less data, otherwise ridge reg model with xval will overfit on new features
    X_train_tr = afreg.fit_transform(X_train, y_train)
    X_test_tr = afreg.transform(X_test)
    print("autofeat new features:", len(afreg.new_feat_cols_))
    print("autofeat Acc. on training data:", accuracy_score(y_train, afreg.predict(X_train_tr)))
    print("autofeat Acc. on test data:", accuracy_score(y_test, afreg.predict(X_test_tr)))
    # train rreg on transformed train split incl cross-validation for parameter selection
    print("# Logistic Regression")
    rreg = LogisticRegression(class_weight="balanced")
    param_grid = {"C": np.logspace(-4, 4, 10)}
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        gsmodel = GridSearchCV(rreg, param_grid, cv=5)
        gsmodel.fit(X_train_tr, y_train)
    print("best params:", gsmodel.best_params_)
    print("best score:", gsmodel.best_score_)
    print("Acc. on training data:", accuracy_score(y_train, gsmodel.predict(X_train_tr)))
    print("Acc. on test data:", accuracy_score(y_test, gsmodel.predict(X_test_tr)))
    print("# Random Forest")
    rforest = RandomForestClassifier(n_estimators=100, random_state=13)
    param_grid = {"min_samples_leaf": [0.0001, 0.001, 0.01, 0.05, 0.1, 0.2]}
    gsmodel = GridSearchCV(rforest, param_grid, cv=5)
    gsmodel.fit(X_train_tr, y_train)
    print("best params:", gsmodel.best_params_)
    print("best score:", gsmodel.best_score_)
    print("Acc. on training data:", accuracy_score(y_train, gsmodel.predict(X_train_tr)))
    print("Acc. on test data:", accuracy_score(y_test, gsmodel.predict(X_test_tr)))
    return gsmodel.best_estimator_
```

--------------------------------

### Test Random Forest Classifier on Datasets

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/autofeat_benchmark_classification.ipynb

Iterates through specified datasets and tests a Random Forest Classifier using the `test_model` function with a parameter grid for 'min_samples_leaf'.

```python
for dsname in datasets:
    print("####", dsname)
    rforest = RandomForestClassifier(n_estimators=100, random_state=13)
    params = {"min_samples_leaf": [0.0001, 0.001, 0.01, 0.05, 0.1, 0.2]}
    rforest = test_model(dsname, rforest, params)
```

--------------------------------

### Autofeat Citation

Source: https://github.com/cod3licious/autofeat/blob/main/docs/science.md

This is the BibTeX entry for citing the Autofeat paper in academic research.

```bibtex
@inproceedings{horn2019autofeat,
  title={The autofeat Python Library for Automated Feature Engineering and Selection},
  author={Horn, Franziska and Pack, Robert and Rieger, Michael},
  booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
  pages={111--120},
  year={2019},
  organization={Springer}
}
```

--------------------------------

### Calculate Precision and Recall for Autofeat

Source: https://github.com/cod3licious/autofeat/blob/main/notebooks/featsel_benchmark_linearmodels.ipynb

Calculates precision and recall for features selected by Autofeat. It compares the set of selected features against the set of true features to determine the overlap.

```python
def prec_rec_autofeat(selected_features, true_features):
    if not len(selected_features):
        return 0, 0
    TP = len(set(selected_features).intersection(true_features))
    recall = TP / len(true_features)
    precision = TP / len(selected_features)
    return precision, recall
```

--------------------------------

### Initialize AutoFeatRegressor for Categorical Features

Source: https://context7.com/cod3licious/autofeat/llms.txt

Initializes AutoFeatRegressor specifying categorical columns for automatic one-hot encoding. Use this when your dataset contains categorical variables.

```python
afreg = AutoFeatRegressor(
    categorical_cols=['category'],  # Columns to one-hot encode
    feateng_cols=None,              # Use all columns for engineering (default)
    feateng_steps=2,
    verbose=1
)

df_transformed = afreg.fit_transform(X, target)
print(f"Categorical mapping: {afreg.categorical_cols_map_}")
```