SKSurrogate

https://sksurrogate.readthedocs.io/en/latest/
Admin
SKSurrogate is a Python toolkit for designing and tuning machine learning pipelines and tracking the...
Tokens:71,533
Snippets:372
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install SKSurrogate via setup.py

Source: https://sksurrogate.readthedocs.io/en/latest/intro.html

Run this command in your terminal to install the SKSurrogate package from the source directory.

```bash
sudo python setup.py install
```

--------------------------------

### Parameter Initialization and Bounds Setup

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

Initializes parameter ranges, calculates search radii, and sets bounds for the optimization process.

```python
# from lightgbm.sklearn import LightGBMError
radius_list = []
self.scorer_ = check_scoring(self.estimator, scoring=self.scoring)
target_classes = []
if y is not None:
    target_classes = unique(y)
# if type(self.cv) is int:
#    cv = ShuffleSplit(n_splits=self.cv, test_size=.25)
# else:
cv = check_cv(self.cv, y, classifier=is_classifier(self.estimator))
x0_ = []
self.bounds = list(self.bounds)
for param in self.params_list:
    param_num_range = self.params[param]
    if param_num_range.VType != "hdreal":
        radius_list.append(
            (param_num_range.upper - param_num_range.lower) / 2.0
        )
        if param in self.init:
            if param_num_range.VType == "categorical":
                x0_.append(param_num_range.items.index(self.init[param]))
            else:
                x0_.append(self.init[param])
        else:
            x0_.append(uniform(param_num_range.lower, param_num_range.upper))
        self.bounds.append(param_num_range.bound_tuple)
    else:
        for i in range(param_num_range.n):
            radius_list.append(
                (
                        param_num_range.bound_tuple[i][1]
                        - param_num_range.bound_tuple[i][0]
                )
                / 2.0
            )
            if (param in self.init) and (i in self.init[param]):
                x0_.append(self.init[param][i])
            else:
                x0_.append(
                    uniform(
                        param_num_range.bound_tuple[i][0],
                        param_num_range.bound_tuple[i][1],
                    )
                )
        self.bounds = self.bounds + list(param_num_range.bound_tuple)
if self.radius is None:
    rds = 0.0
    for r in radius_list:
        rds += r ** 2
    self.radius = sqrt(rds)
self.x0 = array(x0_)
self.bounds = tuple(self.bounds)
cv_dat = list(cv.split(X, y))
```

--------------------------------

### Run EOA on a synthetic population

Source: https://sksurrogate.readthedocs.io/en/latest/eoa.html

Synthetic example on a small set of letters and a fitness based on ASCII code and length of individuals. It prepares the population, initiates the EOA instance, runs it, and prints the most fit individual found.

```python
# prepare the whole population
P = Words(['a', 'b', 'c', 'd', '1', '2'], last=['1', '2', '3'], repeat=True)
Pop = P.Generate(1) + P.Generate(2) + P.Generate(3) + P.Generate(4) +
      P.Generate(5) + P.Generate(6) + P.Generate(7)
# initiate the EOA instance
gen = EOA(population=Pop, fitness=sfit, num_parents=100, mutation_prob=.1, term_genes=['1', '2'])
# run the EOA
gen()
# get the most fit individual found and print
best = next(reversed(tst.children))
print(best, tst.children[best])
```

--------------------------------

### Regression pipeline optimization with AML.eoa_fit on Airfoil data

Source: https://sksurrogate.readthedocs.io/en/latest/opd.html

Complete example that finds an optimum regression pipeline using the Airfoil Self-Noise Data Set. It defines a configuration with regressors, preprocessors, and feature selectors, then runs evolutionary optimization with AML.eoa_fit and prints the top 5 models. Requires SKSurrogate, pandas, numpy, scikit-learn, and the UCI dataset URL.

```python
# Find an optimum regression pipeline

import pandas as pd
import numpy as np
from sklearn.model_selection import RandomizedSearchCV
from sklearn.kernel_ridge import KernelRidge
from sklearn.gaussian_process.kernels import Matern, Sum, ExpSineSquared
from SKSurrogate import *

config = {
    # Regressors
    "sklearn.linear_model.LinearRegression": {"normalize": Categorical([True, False])},
    "sklearn.kernel_ridge.KernelRidge": {
        "alpha": Real(1.0e-4, 10.0),
        "kernel": Categorical(
            [
                Sum(Matern(), ExpSineSquared(l, p))
                for l in np.logspace(-2, 2, 10)
                for p in np.logspace(0, 2, 10)
            ]
        ),
    },
    # Preprocesssors
    "sklearn.preprocessing.StandardScaler": {
        "with_mean": Categorical([True, False]),
        "with_std": Categorical([True, False]),
    },
    "sklearn.preprocessing.Normalizer": {"norm": Categorical(["l1", "l2", "max"])},
    # Feature Selectors
    "sklearn.feature_selection.VarianceThreshold": {"threshold": Real(0.0, 0.3)},
}
import warnings

warnings.filterwarnings("ignore", category=Warning)

df = pd.read_csv(
    "https://archive.ics.uci.edu/ml/machine-learning-databases/00291/airfoil_self_noise.dat",
    sep="\t",
    names=["Frequency", "Angle", "length", "velocity", "thickness", "level"],
)
X = df.drop("level", axis=1).values
y = df["level"].values

A = AML(
    config=config,
    length=3,
    check_point="./",
    verbose=2,
    scoring="neg_mean_squared_error",
)
A.eoa_fit(X, y, max_generation=12, num_parents=12)
print(A.get_top(5))
```

--------------------------------

### Output of the synthetic EOA example

Source: https://sksurrogate.readthedocs.io/en/latest/eoa.html

Output produced by the synthetic EOA example. Shows the progress bar and the most fit individual found with its fitness value.

```text
100%|##########| 50/50 [00:00<00:00, 769.22it/s]
('1', '1', '1', '1', '1', '1', '1') 10.149999999999999
```

--------------------------------

### SurrogateRandomCV Initialization

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

The __init__ method initializes the SurrogateRandomCV estimator with various hyperparameters for optimization, including search strategy, regressor settings, and solver configurations.

```python
    def __init__(
            self,
            estimator,
            params,
            scoring=None,
            fit_params=None,
            n_jobs=-1,
            refit=True,
            cv=None,
            verbose=0,
            pre_dispatch="2*n_jobs",
            error_score="raise",
            return_train_score=True,
            max_iter=50,
            min_evals=25,
            regressor=None,
            sampling=CompactSample,
            radius=None,
            contraction=0.95,
            search_sphere=False,
            optimizer="scipy",
            scipy_solver="SLSQP",
            task_name="optim_task",
            warm_start=True,
            Continue=False,
            max_itr_no_prog=10000,
            ineqs=(),
            init=None,
    ):
        super(SurrogateRandomCV, self).__init__(
            estimator=estimator,
            scoring=scoring,
            n_jobs=n_jobs,
            refit=refit,
            cv=cv,
            verbose=verbose,
            pre_dispatch=pre_dispatch,
            error_score=error_score,
            return_train_score=return_train_score,
        )
        self.params = params
        self.fit_params = fit_params
        self.params_list = list(params.keys())
        self.max_iter = max_iter
        self.min_evals = min_evals
        self.radius = radius
        self.contraction = contraction
        self.regressor = regressor
        self.sampling = sampling
        self.search_sphere = search_sphere
        self.optimizer = optimizer
        self.scipy_solver = scipy_solver
        self.task_name = task_name
        self.warm_start = warm_start
        self.Continue = Continue
        self.max_itr_no_prog = max_itr_no_prog
        self.bounds = []
        self.ineqs = ineqs
        self.init = init if init is not None else {}
        self.OPTIM = None
        self.scorer_ = {}
        self.x0 = ()
        self.best_estimator_ = None
        self.best_estimator_score = 0.0
        self.best_score_ = 0.
```

--------------------------------

### Configure and Run SKSurrogate Search

Source: https://sksurrogate.readthedocs.io/en/latest/mltrace.html

Sets up the search space for SVM estimators and preprocessing steps, then executes the search using AML.

```python
import numpy as np
from SKSurrogate import *
import warnings
warnings.filterwarnings("ignore", category=Warning)

from sklearn.model_selection import RandomizedSearchCV
from sklearn.kernel_ridge import KernelRidge
from sklearn.gaussian_process.kernels import Matern, Sum, ExpSineSquared
param_grid_krr = {"alpha": np.logspace(-4, 0, 5),
                  "kernel": [Sum(Matern(), ExpSineSquared(l, p))
                             for l in np.logspace(-2, 2, 10)
                             for p in np.logspace(0, 2, 10)]}
regressor = RandomizedSearchCV(KernelRidge(), param_distributions=param_grid_krr, n_iter=7, cv=2)

config = {
    # estimators
    'sklearn.svm.SVC': {
        "C": Real(1e-6, 20.),
        "gamma": Real(1e-6, 10.),
        "tol": Real(1e-6, 10.),
        "class_weight": HDReal((1.e-5, 1.e-5), (20., 20.))
    },
    'sklearn.svm.NuSVC': {
        'nu': Real(1.e-5, 1.),
        "gamma": Real(1e-6, 10.),
        "tol": Real(1e-6, 10.),
        "class_weight": HDReal((1.e-5, 1.e-5), (20., 20.))
    },
    # preprocessing
    'sklearn.preprocessing.StandardScaler': {
        'with_mean': Categorical([True, False]),
        'with_std': Categorical([True, False]),
    },
    'sklearn.feature_selection.VarianceThreshold': {
        'threshold': Real(0., .3)
    },
    'sklearn.preprocessing.Normalizer': {
        'norm': Categorical(['l1', 'l2', 'max'])
    },
}

MLTr = mltrack('sample', db_name="sample.db")
X, y = MLTr.get_data()


A_svm = AML(config=config, length=3, check_point='./svm/', verbose=1)
A_svm.eoa_fit(X, y, max_generation=10, num_parents=10)
print(A_svm.get_top(4))
```

--------------------------------

### _get_class static method

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/aml.html

Takes a string referring to a class in an accessible module and returns the associated class. Uses importlib to import the module and get the attribute.

```python
    @staticmethod
    def _get_class(library):
        """
        Takes a string that refers to a class in an accessible module and returns the associated class

        :param library: string referring to a class
        :return: the actual class
        """
        from importlib import import_module

        detail = library.split(".")
        module_str = ".".join(detail[:-1])
        module = import_module(module_str)
        clss = module.__getattribute__(detail[-1])
        return clss
```

--------------------------------

### Initialize mltrack and register data

Source: https://sksurrogate.readthedocs.io/en/latest/mltrace.html

Initializes the mltrace tracker with a task named 'sample' and a SQLite database file 'sample.db', registers the DataFrame with the target column, and updates the task description. This is the second step in the mltrace workflow.

```python
# initialize the tracker with a task called 'sample'
MLTr = mltrack('sample', db_name="sample.db")
# register the data
MLTr.RegisterData(df, 'target')
# modify the description of the task
MLTr.UpdateTask({'description': "This is a sample task to demonstrate\\
        capabilities of the mltrace."})
```

--------------------------------

### StructSearch Initialization Logic

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

Initializes the optimization parameters, sampling strategy, and regressor configuration.

```python
        self.objective = objective
        self.ineqs = kwargs.pop("ineq", [])
        self.bounds = kwargs.pop("bounds", None)
        self.MaxIter = kwargs.pop("max_iter", 50)
        self.radius = kwargs.pop("radius", 2.0)
        self.contraction = kwargs.pop("contraction", 0.9)
        sampling = kwargs.pop("sampling", SphereSample)
        self.Sampling = sampling(
            init_radius=self.radius,
            contraction=self.contraction,
            ineq=self.ineqs,
            bounds=self.bounds,
        )
        self.iteration = 0
        self.verbose = kwargs.pop("verbose", False)
        self.MaxIterNoProg = kwargs.pop("max_itr_no_prog", inf)
        self.NumIterNoProg = 0
        self.x0 = kwargs.pop("x0", None)
        self.evaluated = []
        self.NumFailedLocOptim = 0
        # init data storing protocol
        self.TaskName = kwargs.pop("task_name", "optim_task")
        self.Continue = kwargs.pop("Continue", False)
        self.warm_start = kwargs.pop("warm_start", False)
        self.optimizer = kwargs.get("optimizer", "scipy")
        self.__load()
        if self.x0 is not None:
            val = self.objective(self.x0)
            self.evaluated.append([self.x0, val])
            self.current = self.x0
            self.current_val = val
            n_ = len(self.x0)
        else:
            n_ = 1
        self.search_sphere = kwargs.pop("search_sphere", False)
        poly_deg = kwargs.pop("deg", 3)
        self.MinEvals = kwargs.pop("min_evals", binom(n_ + poly_deg, n_))
        self.regressor = kwargs.pop("regressor", None)
        self.scipy_solver = kwargs.get("scipy_solver", "COBYLA")
        if self.regressor is None:
            from .NpyProximation import HilbertRegressor

            self.regressor = HilbertRegressor(deg=3)
            new_n_evals = binom(n_ + 3, n_)
            if self.MinEvals < new_n_evals:
                self.MinEvals = new_n_evals
        self.other = kwargs
```

--------------------------------

### Words(letters, last=None, first=None, repeat=False)

Source: https://sksurrogate.readthedocs.io/en/latest/code.html

Class that takes a set as alphabet and generates words of a given length accordingly. Accepts parameters for allowed letters, ending letters, starting words, and whether consecutive repeats are allowed.

```APIDOC
## Words(letters, last=None, first=None, repeat=False)

### Description
This class takes a set as alphabet and generates words of a given length accordingly.

### Constructor Parameters
- **letters** (set) - Required - A set of letters (symbols) to make up the words.
- **last** (subset of letters) - Optional - A subset of letters that are allowed to appear at the end of a word.
- **first** (set of words) - Optional - A set of words that can only appear at the beginning of a word.
- **repeat** (bool) - Optional - Whether consecutive occurrence of a letter is allowed.
```

--------------------------------

### EOA.__init__

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/eoa.html

Initializes the EOA object with population, fitness function, and various configuration parameters for the evolutionary algorithm.

```APIDOC
## EOA.__init__

### Description
Initializes the EOA object with the given population and fitness function, along with optional parameters for initialization, recombination, mutation, termination, elitism, and other algorithm settings.

### Method
Constructor

### Signature
EOA(population, fitness, **kwargs)

### Parameters
- **population** (list) - Required - The whole possible population as a list.
- **fitness** (callable) - Required - The fitness evaluation function. Accepts an OrderedDict of individuals with their corresponding fitness and updates their fitness.
- **init_pop** (class) - Optional - The python class that initiates the initial population. Default: UniformRand.
- **recomb** (class) - Optional - The python class that defines how to combine parents to produce children. Default: UniformCrossover.
- **mutation** (class) - Optional - The python class that performs mutation on offspring population. Default: Mutation.
- **termination** (class) - Optional - The python class that determines the termination criterion. Default: MaxGenTermination.
- **elitism** (class) - Optional - The python class that decides how to handle elitism. Default: Elites.
- **num_parents** (int) - Optional - The size of initial parents population.
- **parents_porp** (float) - Optional - The size of initial parents population given as a portion of whole population (only used if num_parents is not given). Default: 0.1.
- **elits_porp** (float) - Optional - The proportion of offspring to be replaced by elite parents. Default: 0.2.
- **mutation_prob** (float) - Optional - The probability that a component will be mutated. Default: 0.05.
- **kwargs** - Optional - Additional keyword arguments.

### Returns
None
```

--------------------------------

### BoxSample __init__ method

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

Initializes a BoxSample instance, popping the 'bounds' key from kwargs and storing it as self.bounds.

```python
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.bounds = kwargs.pop("bounds", None)
```

--------------------------------

### UniformRand Population Initialization

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/eoa.html

Initializes the population by selecting random indices from the available population pool.

```python
[docs]class UniformRand(object):
    """
    Initial population initiation.
    """

    def __init__(self, **kwargs):
        pass

    def __call__(self, ref, *args, **kwargs):
        from collections import OrderedDict
        from random import randint

        cnt = 0
        indices = []
        while cnt < ref.num_parents:
            idx = randint(0, ref.population_size - 1)
            if idx not in indices:
                # print(ref.population[idx])
                indices.append(idx)
                cnt += 1
        return OrderedDict(
            [(ref.population[i], ref.evals[ref.population[i]]) for i in indices]
        )
```

--------------------------------

### StructSearch State Persistence Methods

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

Methods for saving and loading optimization progress using pickle.

```python
    def __save(self):
        """
        Logs state of the optimization progress at each iteration
        :return: None
        """
        from pickle import dumps

        fl = open(self.TaskName + ".pkl", "wb")
        data2store = dict(
            iteration=self.iteration,
            evaluated=self.evaluated,
            current=self.current,
            current_val=self.current_val,
        )
        fl.write(dumps(data2store))
        fl.close()
```

```python
    def __load(self):
        """
        Loads previous information saved, if any
        :return: None
        """
        from pickle import loads

        restored_data = {}
        try:
            fl = open(self.TaskName + ".pkl", "rb")
            restored_data = loads(fl.read())
            fl.close()
        except FileNotFoundError:
            pass
        if self.Continue:
            if "iteration" in restored_data:
                self.iteration = restored_data["iteration"]
            if "evaluated" in restored_data:
                self.evaluated = restored_data["evaluated"]
            if "current" in restored_data:
                self.current = restored_data["current"]
                self.x0 = None
            if "current_val" in restored_data:
                self.current_val = restored_data["current_val"]
        if self.warm_start:
            if "evaluated" in restored_data:
                self.evaluated = restored_data["evaluated"]
```

--------------------------------

### fit(X, y=None, groups=None, **fit_params)

Source: https://sksurrogate.readthedocs.io/en/latest/code.html

Runs the fit with all sets of parameters. This method is used to execute the optimization process with the provided training data and parameters.

```APIDOC
## fit(X, y=None, groups=None, **fit_params)

### Description
Run fit with all sets of parameters.

### Method
fit

### Parameters
- **X** (array-like, shape = [n_samples, n_features]) - Required - Training vector, where n_samples is the number of samples and n_features is the number of features.
- **y** (array-like, shape = [n_samples] or [n_samples, n_output], optional) - Optional - Target relative to X for classification or regression; None for unsupervised learning.
- **groups** (array-like, with shape (n_samples,), optional) - Optional - Group labels for the samples used while splitting the dataset into train/test set.
- **fit_params** (dict of string -> object) - Optional - Parameters passed to the fit method of the estimator.

### Returns
- self
```

--------------------------------

### Basis Construction Logic

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/NpyProximation.html

A snippet demonstrating the generation of a basis using trigonometric functions based on degree and dimension parameters.

```python
        B = [lambda x: 1.0]
        E = list(product([0, 1], repeat=n))
        RawCoefs = list(product(range(deg + 1), repeat=n))
        Coefs = set()
        for prt in RawCoefs:
            p_ = list(prt)
            p_.sort()
            Coefs.add(tuple(p_))
        for o in Coefs:
            if (sum(o) <= deg) and (sum(o) > 0):
                for ex in E:
                    if sum(ex) >= 0:
                        f_ = lambda x, o_=o, ex_=ex: prod(
                            [
                                sin(o_[i] * x[i] / l) ** ex_[i]
                                * cos(o_[i] * x[i] / l) ** (1 - ex_[i])
                                if o_[i] > 0
                                else 1.0
                                for i in range(n)
                            ]
                        )
                        B.append(f_)
        return B
```

--------------------------------

### eoa.UniformRand

Source: https://sksurrogate.readthedocs.io/en/latest/code.html

Initial population initiation class that generates the initial population.

```APIDOC
## eoa.UniformRand

### Description
Initial population initiation.

### Constructor
`eoa.UniformRand(**kwargs)`

### Parameters
- **kwargs** - Optional - Additional keyword arguments
```

--------------------------------

### StructSearch Parameter and Iteration Helpers

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

Helper methods for setting optimization parameters and managing iteration cycles.

```python
    def __optim_param(self):
        """
        Fetches and sets the Optimization's parameters
        :return: None
        """
        self.optimizer = self.other.pop("optimizer", "scipy")
        self.scipy_solver = self.other.pop("scipy_solver", "COBYLA")
```

```python
    def __iterate(self, x0):
        """
        Repeat a round of optimization.
        :param x0: init point for the iteration
        :return: None
        """
        from numpy import sqrt, array
```

--------------------------------

### Create sample classification dataset

Source: https://sksurrogate.readthedocs.io/en/latest/mltrace.html

Creates a synthetic classification dataset with 1000 samples and 10 features, then converts it to a pandas DataFrame using the SKSurrogate np2df helper. This is the first step in the mltrace workflow.

```python
# import requires libraries
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from SKSurrogate import *
# make up a classification dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=6, n_redundant=2)
Xy = np.hstack((X, np.reshape(y, (-1, 1))))
# make up some names for columns of the data
cols = ['cl%d'%(_+1) for _ in range(10)] + ['target']
# turn it into a pandas DataFrame
df = np2df(Xy, cols)()
```

--------------------------------

### BoxSample sample method

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

Samples a point from a box centered at the given centre, using uniform random values and constraint checking. The radius is contracted by the contraction factor until a valid candidate is found.

```python
    def sample(self, centre, cntrctn=1.0):
        """
        Samples a point out of a box centered at `centre`

        :param centre: `numpy.array` the center of the box
        :param cntrctn: `float` customized contraction factor
        :return: `numpy.array` a new sample
        """
        from random import uniform
        from numpy import array

        flag = False
        n = len(centre)
        candid = []
        radius = self.init_radius * cntrctn
        while not flag:
            candid = [uniform(-radius, radius) for _ in range(n)]
            candid = array([candid[i] for i in range(n)]) + centre
            flag = self.check_constraints(candid)
            radius = radius * self.contraction
        return candid
```

--------------------------------

### HilbertRegressor.__init__

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/NpyProximation.html

Initializes the HilbertRegressor estimator with optional parameters for degree, basis, measure, and function space.

```APIDOC
## HilbertRegressor.__init__

### Description
Initializes the HilbertRegressor estimator with optional parameters for degree, basis, measure, and function space.

### Method
Constructor

### Signature
`__init__(self, deg=3, base=None, meas=None, fspace=None)`

### Parameters
- **deg** (int, default=3) - Optional - The degree of polynomial regression. Only used if `base` is `None`.
- **base** (list, default=None) - Optional - A list of functions to form an orthogonal function basis.
- **meas** (NpyProximation.Measure, default=None) - Optional - The measure to form the L2(mu) space. If None, a discrete measure will be constructed based on fit inputs.
- **fspace** (NpyProximation.FunctionBasis, default=None) - Optional - The function subspace of L2(mu). If None, it will be initiated according to self.meas.

### Returns
None
```

--------------------------------

### Build and optimize a pipeline with SurrogateRandomCV

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/aml.html

Constructs a scikit-learn or imbalanced-learn Pipeline from a sequence of estimator names, handling feature selectors, classifiers, regressors, and FeatureUnion. Uses StackingEstimator for intermediate estimators and PermutationImportance with SelectFromModel for feature selection. Optimizes the pipeline with SurrogateRandomCV and returns the best estimator and score.

```python
if self.couldBfirst == []:
    from sklearn.pipeline import Pipeline
else:
    from imblearn.pipeline import Pipeline
OPTIM = None
n = len(seq)
idx = 0
ent_idx = 0
steps = []
config = {}
task_name = self.check_point + "_".join(seq)
while ent_idx < n:
    est = seq[ent_idx]
    clss = self._get_class(est)
    pre = "stp_%d" % idx
    if (
            self.config_types[est] in ["regressor", "classifier"]
            and ent_idx < n - 1
    ):
        mdl = clss()
        steps.append(
            (
                pre,
                StackingEstimator(
                    mdl,
                    res=self.stack_res,
                    probs=self.stack_probs,
                    decision=self.stack_decision,
                ),
            )
        )
        ent_idx += 1
    elif est == "sklearn.pipeline.FeatureUnion":
        self.config[est] = dict()
        int_idx = 1
        int_steps = []
        next_est = seq[ent_idx + int_idx]
        # Find the rest of the sequence before the last or the next feature selector
        while (
                (self.config_types[next_est] in ["regressor", "classifier"])
                or (next_est in self.known_feature_selectors)
        ) and (ent_idx + int_idx < n - 1):
            int_pre = "int_%d" % int_idx
            if next_est in self.known_feature_selectors:
                int_mdl = self._get_class(next_est)()
                # set the parameter's dictionary
                for kw in self.config[next_est]:
                    self.config[est][int_pre + "__" + kw] = self.config[
                        next_est
                    ][kw]
            else:
                from eli5.sklearn import PermutationImportance
                from sklearn.feature_selection import SelectFromModel
                from numpy import inf

                int_est = self._get_class(next_est)()
                int_mdl = SelectFromModel(
                    PermutationImportance(int_est, scoring=self.scoring, cv=3),
                    threshold=-inf,
                )
                self.config[est][int_pre + "__" + "max_features"] = Integer(
                    1, self.num_features
                )
                for kw in self.config[next_est]:
                    self.config[est][
                        int_pre + "__" + "estimator__estimator__" + kw
                        ] = self.config[next_est][kw]
            int_steps.append((int_pre, int_mdl))
            int_idx += 1
            next_est = seq[ent_idx + int_idx]
        if int_steps != []:
            mdl = clss(int_steps)
            steps.append((pre, mdl))
        ent_idx += int_idx
    else:
        mdl = clss()
        steps.append((pre, mdl))
        ent_idx += 1
    for kw in self.config[est]:
        config[pre + "__" + kw] = self.config[est][kw]
    idx += 1
ppln = Pipeline(steps)
if self.verbose > 0:
    print("=" * 90)
    print(seq)
    print("-" * 90)
for srgt in self.surrogates:
    OPTIM = SurrogateRandomCV(
        ppln,
        params=config,
        max_iter=srgt[1],
        min_evals=self.min_random_evals,
        scoring=self.scoring,
        cv=self.cv,
        verbose=max(self.verbose - 1, 0),
        sampling=srgt[2],
        regressor=srgt[0],
        scipy_solver=srgt[3],
        task_name=task_name,
        Continue=True,
        warm_start=True,
        n_jobs=self.n_jobs
    )
    OPTIM.fit(X, y)
return OPTIM.best_estimator_, OPTIM.best_estimator_score
```

--------------------------------

### Measure class initialization and methods

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/NpyProximation.html

Constructs a measure based on density and domain, providing methods for integration and norm calculation.

```python
[docs]class Measure(object):
    r"""
    Constructs a measure :math:`\mu` based on `density` and `domain`.

    :param density: the density over the domain:
            + if none is given, it assumes uniform distribution

            + if a callable `h` is given, then :math:`d\mu=h(x)dx`

            + if a dictionary is given, then :math:`\mu=\sum w_x\delta_x` a discrete measure.
              The points :math:`x` are the keys of the dictionary (tuples) and the weights :math:`w_x` are the values.
    :param domain: if `density` is a dictionary, it will be set by its keys. If callable, then `domain` must be a list
                   of tuples defining the domain's box. If None is given, it will be set to :math:`[-1, 1]^n`
    """

    def __init__(self, density=None, domain=None):

        # set the density
        if density is None:
            self.density = lambda x: 1.0
        elif callable(density):
            self.density = density
        elif isinstance(density, dict):
            self.density = density  # lambda x: density[x] if x in density else 0.
        else:
            raise Error(
                "The `density` must be either a callable or a dictionary of real numbers."
            )
        # check and set the domain
        self.continuous = True
        self.dim = 0
        if isinstance(domain, list):
            self.dim = len(domain)
            for intrvl in domain:
                if (not isinstance(intrvl, (list, tuple))) or (len(intrvl) != 2):
                    raise Error("`domain` should be a list of 2-tuples.")
            self.supp = domain
        elif isinstance(density, dict):
            self.supp = density.keys()
            self.continuous = False
        else:
            raise Error("No domain is specified.")


[docs]    def integral(self, f):
        r"""
        Calculates :math:`\int_{domain} fd\mu`.

        :param f: the integrand
        :return: the value of the integral
        """
        from types import FunctionType

        m = 0.0
        if not isinstance(
                f, (dict, FunctionType)
        ):  # if type(f) not in [dict, FunctionType]:
            raise Error("The integrand must be a `function` or a `dict`")
        if isinstance(f, dict):
            fn = lambda x: f[x] if x in f else 0.0
        else:
            fn = f
        if self.continuous:
            from scipy import integrate

            fw = lambda *x: fn(*x) * self.density(*x)
            m = integrate.nquad(fw, self.supp)[0]
        else:
            for p in self.supp:
                # print(p, self.density[p], fn)
                # print(p, self.density[p], fn(p))
                m += self.density[p] * fn(p)
        return m


[docs]    def norm(self, p, f):
        r"""
        Computes the norm-`p` of the `f` with respect to the current measure,
        i.e., :math:`(\int_{domain}|f|^p d\mu)^{1/p}`.

        :param p: a positive real number
        :param f: the function whose norm is desired.
        :return: :math:`\|f\|_{p, \mu}`
        """
        absfp = lambda *x: pow(abs(f(*x)), p)
        return pow(self.integral(absfp), 1.0 / p)
```

--------------------------------

### FunctionBasis class and methods

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/NpyProximation.html

Generates Polynomial and Fourier function bases for regression.

```python
[docs]class FunctionBasis(object):
    """
    This class generates two typical basis of functions: Polynomials and Trigonometric
    """

    def __init__(self):
        pass


[docs]    @staticmethod
    def Poly(n, deg):
        """
        Returns a basis consisting of polynomials in `n` variables of degree at most `deg`.

        :param n: number of variables
        :param deg: highest degree of polynomials in the basis
        :return: the raw basis consists of polynomials of degrees up to `n`
        """
        from itertools import product
        from numpy import prod

        B = []
        for o in product(range(deg + 1), repeat=n):
            if sum(o) <= deg:
                B.append(lambda x, e=o: prod([x[i] ** e[i] for i in range(n)]))
        return B


[docs]    @staticmethod
    def Fourier(n, deg, l=1.0):
        """
        Returns the Fourier basis of degree `deg` in `n` variables with period `l`

        :param n: number of variables
        :param deg: the maximum degree of trigonometric combinations in the basis
        :param l: the period
        :return: the raw basis consists of trigonometric functions of degrees up to `n`
        """

        from numpy import sin, cos, prod
        from itertools import product
```

--------------------------------

### SurrogateSearch(objective, **kwargs)

Source: https://sksurrogate.readthedocs.io/en/latest/code.html

Constructor for the SurrogateSearch class. Initializes the optimization task with the given objective function and various hyperparameters.

```APIDOC
## SurrogateSearch(objective, **kwargs)

### Description
An implementation of the Surrogate Random Search (SRS).

### Parameters
- **objective** (callable) - Required - The function to be minimized.
- **ineq** (list of callables) - Optional - Constraints (default: []).
- **task_name** (str) - Optional - A name to refer to the optimization task, store & restore previously acquired (default: 'optim_task').
- **bounds** (list of tuples of real numbers) - Optional - Bounds on each variable (default: None).
- **max_iter** (int) - Optional - Maximum number of iterations (default: 50).
- **radius** (float) - Optional - Initial radius of sampling region (default: 2.).
- **contraction** (float) - Optional - Rate of radius contraction (default: .9).
- **sampling** (BoxSample or SphereSample) - Optional - Sampling method (default: SphereSample).
- **search_sphere** (boolean) - Optional - Whether to fit the surrogate function on a neighbourhood of current candidate or over all sampled points (default: False).
- **deg** (int) - Optional - Degree of polynomial regressor if one chooses to fit polynomial surrogates (default: 3).
- **min_evals** (int) - Optional - Minimum number of samples before fitting a surrogate (default will be calculated as if the surrogate is a polynomial of degree 3).
- **regressor** (regressor) - Optional - A regressor (scikit-learn style) to find a surrogate.
- **scipy_solver** (str) - Optional - The scipy solver ('COBYLA' or 'SLSQP') to solve the local optimization problem at each iteration (default: 'COBYLA').
- **max_itr_no_prog** (int) - Optional - Maximum number of iterations with no progress (default: infinity).
- **Continue** (boolean) - Optional - Continues the progress from where it has been interrupted (default: False).
- **warm_start** (boolean) - Optional - Use data from the previous attempts, but starts from the first iteration (default: False).
- **verbose** (boolean) - Optional - Whether to report the progress on commandline or not (default: False).
```

--------------------------------

### Regression.__init__

Source: https://sksurrogate.readthedocs.io/en/latest/code.html

Initializes a Regression object with a set of points or a callable, and optionally specifies the dimension of the domain.

```APIDOC
## Regression.__init__

### Description
Initializes a Regression object with a set of points or a callable, and optionally specifies the dimension of the domain.

### Method
Constructor

### Parameters
- **points** (list or callable) - Required - A list of points to be fitted or a callable to be approximated.
- **dim** (int) - Optional - Dimension of the domain.

### Returns
None
```

--------------------------------

### class structsearch.CompactSample

Source: https://sksurrogate.readthedocs.io/en/latest/code.html

Sampling method that generates uniform samples from a box.

```APIDOC
## class structsearch.CompactSample(**kwargs)

### Description
Generates samples uniformly out of a box.

### Methods
#### sample(centre, cntrctn=1.0)
Samples a point out of a box centered at centre.

##### Parameters
- **centre** (numpy.array) - The center of the box.
- **cntrctn** (float) - Customized contraction factor.

##### Returns
- numpy.array - A new sample.
```

--------------------------------

### Load EOA Checkpoint

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/eoa.html

Loads population and task state from a binary file. If the file is not found, the operation is skipped.

```python
        try:
            fl = open(self.check_point + self.task_name + ".eoa", "rb")
            info = loads(fl.read())
            fl.close()
            self.population_size = info["population_size"]
            self.parents_porp = info["parents_porp"]
            self.num_parents = info["num_parents"]
            self.elits_porp = info["elits_porp"]
            self.num_elites = info["num_elites"]
            self.mutation_prob = info["mutation_prob"]
            self.max_generations = info["max_generations"]
            self.generation_num = info["generation_num"]
            self.genes = info["genes"]
            self.init_genes = info["init_genes"]
            self.term_genes = info["term_genes"]
            self.task_name = info["task_name"]
            self.check_point = info["check_point"]
            self.evals = info["evals"]
            self.parents = info["parents"]
            self.children = info["children"]
        except FileNotFoundError:
            pass
```

--------------------------------

### aml.AML constructor

Source: https://sksurrogate.readthedocs.io/en/latest/code.html

Initializes an AML object with configuration for pipeline optimization.

```APIDOC
## aml.AML(config=None, length=5, scoring='accuracy', cat_cols=None, surrogates=None, min_random_evals=15, cv=None, check_point='./', stack_res=True, stack_probs=True, stack_decision=True, verbose=1, n_jobs=-1)

### Description
Creates an instance of the AML class for finding an optimum combination of machine learning pipelines based on training data.

### Parameters
- **config** (dict) - Optional - A dictionary whose keys are scikit-learn-style objects (as strings) and values are dictionaries of parameters and their acceptable ranges/values.
- **length** (int) - Optional, default=5 - Maximum number of objects in generated pipelines.
- **scoring** (str) - Optional, default='accuracy' - The scoring method to be optimized. Must follow the sklearn scoring signature.
- **cat_cols** (list) - Optional, default=None - The list of indices of categorical columns.
- **surrogates** (list) - Optional, default=None - A list of 4-tuples determining surrogates. The first entity of each tuple is a scikit-learn regressor, the 2nd is the number of iterations, the 3rd is the sampling strategy, and the 4th is the scipy.optimize solver.
- **min_random_evals** (int) - Optional, default=15 - Number of randomly sampled initial values for hyper parameters.
- **cv** - Optional, default=ShuffleSplit(n_splits=3, test_size=.25) - The cross validation method.
- **check_point** (str) - Optional, default='./' - The path where the optimization results will be stored.
- **stack_res** (bool) - Optional, default=True - StackingEstimator's res.
- **stack_probs** (bool) - Optional, default=True - StackingEstimator's probs.
- **stack_decision** (bool) - Optional, default=True - StackingEstimator's decision.
- **verbose** (int) - Optional, default=1 - Level of output details.
- **n_jobs** (int) - Optional, default=-1 - Number of processes to run in parallel.

### Returns
An AML object.
```

--------------------------------

### CompactSample class definition and sample method

Source: https://sksurrogate.readthedocs.io/en/latest/_modules/structsearch.html

Defines the CompactSample class that inherits from BaseSample. Its sample method generates a uniform random point within the bounds, checking constraints until a valid point is found.

```python
class CompactSample(BaseSample):
    """
    Generates samples uniformly out of a box.

    """

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.cntrctn = None


[docs]    def sample(self, centre, cntrctn=1.0):
        """
        Samples a point out of a box centered at `centre`

        :param centre: `numpy.array` the center of the box
        :param cntrctn: `float` customized contraction factor
        :return: `numpy.array` a new sample
        """
        from random import uniform
        from numpy import array

        self.cntrctn = cntrctn
        flag = False
        n = len(centre)
        candid = []
        while not flag:
            candid = array(
                [uniform(self.bounds[_][0], self.bounds[_][1]) for _ in range(n)]
            )
            flag = self.check_constraints(candid)
        return candid
```
Tokens:
Raw