### Install TPOT using Conda

Source: https://context7.com/epistasislab/tpot/llms.txt

Recommended installation using conda, especially for M1/Arm Macs to ensure compatible lightgbm first. Then install TPOT.

```bash
conda install --yes -c conda-forge 'lightgbm>=3.3.3'
pip install tpot
```

--------------------------------

### Install TPOT via Pip

Source: https://context7.com/epistasislab/tpot/llms.txt

Standard installation of TPOT using pip. For Intel scikit-learn acceleration, use `pip install tpot[sklearnex]`.

```bash
pip install tpot
```

```bash
pip install tpot[sklearnex]
```

--------------------------------

### TPOT Example Output

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/1_Using_TPOT.ipynb

This is an example output from a TPOT run, showing the best accuracy achieved.

```text
0.999621947852182
```

--------------------------------

### Install TPOT from Local Repository

Source: https://github.com/epistasislab/tpot/blob/main/docs/installation.md

Install TPOT in editable mode from a local repository path, typically used for development or the latest branch.

```bash
pip install -e /path/to/tpotrepo
```

--------------------------------

### Dask-Jobqueue Setup for SGE

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/7_dask_parallelization.ipynb

Configure a Dask cluster using SGECluster for parallel execution. This example sets up a cluster with specified queues, cores, and memory, and uses adapt to auto-scale the number of jobs. Note that TPOT will ignore n_jobs and memory_limit when a Dask client is provided.

```python
from dask.distributed import Client, LocalCluster
import sklearn
import sklearn.datasets
import sklearn.metrics
import sklearn.model_selection
import tpot
from dask_jobqueue import SGECluster # or SLURMCluster, PBSCluster, etc. Replace SGE with your scheduler.
import os

if os.system("which qsub") != 0:
    print("Sun Grid Engine is not installed. This example requires Sun Grid Engine to be installed.")
else:
    print("Sun Grid Engine is installed.")

    
    cluster = SGECluster(
        queue='all.q',
        cores=2,
        memory="50 GB"

    )

    cluster.adapt(minimum_jobs=10, maximum_jobs=100)  # auto-scale between 10 and 100 jobs

    client = Client(cluster)

    scorer = sklearn.metrics.get_scorer('roc_auc_ovr')
    X, y = sklearn.datasets.load_digits(return_X_y=True)
    X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)

    graph_search_space = tpot.search_spaces.pipelines.GraphSearchPipeline(
    root_search_space= tpot.config.get_search_space(["KNeighborsClassifier", "LogisticRegression", "DecisionTreeClassifier"]),
    leaf_search_space = tpot.config.get_search_space("selectors"), 
    inner_search_space = tpot.config.get_search_space(["transformers"]),
    max_size = 10,
    )

    est = tpot.TPOTEstimator(
        client = client,
        scorers = ["roc_auc"],
        scorers_weights = [1],
        classification = True,
        cv = 10,
        search_space = graph_search_space,
        max_time_mins = 60,
        early_stop=10,
        verbose = 2,
        )
    est.fit(X_train, y_train)
    # this is equivalent to: 
    # est = tpot.TPOTClassifier(population_size= 8, generations=5, n_jobs=4, memory_limit="4GB", verbose=1)
    est.fit(X_train, y_train)
    print(scorer(est, X_test, y_test))

    #It is good to close the client and cluster when you are done with them
    client.close()
    cluster.close()

```

```text
Output:
Sun Grid Engine is not installed. This example requires Sun Grid Engine to be installed.


```

--------------------------------

### Developer Install TPOT

Source: https://context7.com/epistasislab/tpot/llms.txt

Install TPOT in editable mode from a local clone for development purposes.

```bash
pip install -e /path/to/tpot
```

--------------------------------

### GraphSearchPipeline with FSSNode Example

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/3_Feature_Set_Selector.ipynb

Illustrates setting up a GraphSearchPipeline where FSSNodes are the leaf search spaces. This example configures the pipeline to utilize two feature sets and visualizes the generated pipeline.

```python
graph_search_space = tpot.search_spaces.pipelines.GraphSearchPipeline(
    leaf_search_space = fss_search_space,
    inner_search_space = tpot.config.get_search_space(["transformers"]),
    root_search_space= tpot.config.get_search_space(["KNeighborsClassifier", "LogisticRegression", "DecisionTreeClassifier"]),
    max_size = 10,
)

graph_search_space.generate(rng=4).export_pipeline().plot()
```

--------------------------------

### Setup Dummy Dataset and Imports for AMLTK Parser

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/amltk_search_space_parser_example.ipynb

Imports necessary libraries and creates a dummy pandas DataFrame with both numerical and categorical columns, along with a target variable. This setup is required before using the tpot_parser with AMLTK search spaces.

```python
from sklearn.compose import make_column_selector
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.svm import SVC
from amltk.pipeline import Choice, Component, Sequential, Split
import tpot
from sklearn.preprocessing import FunctionTransformer
from sklearn.compose import make_column_transformer
import tpot
import numpy as np
import sklearn
import sklearn.datasets
import pandas as pd
# create dummy pandas dataset with both categorical and numerical columns
X, y = sklearn.datasets.make_classification(n_samples=100, n_features=5, n_informative=3, n_classes=2, random_state=42)
X = pd.DataFrame(X, columns=[f"num_{i}" for i in range(5)])
# add 5 categorical columns
for i in range(5):
    X[f"cat_{i}"] = np.random.choice(["A", "B", "C"], size=100)
y = y.flatten()
# train test split
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size=0.5)
```

--------------------------------

### Selected Features Output

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/4_Genetic_Feature_Selection.ipynb

Example output showing the index of the selected features after running the feature selection.

```text
selected features:  Index(['b', 'c', 'd', 'e', 'f', 'g'], dtype='object')

```

--------------------------------

### Install TPOT with scikit-learn Extensions

Source: https://github.com/epistasislab/tpot/blob/main/README.md

Use this command to install TPOT along with scikit-learn extensions for potential speedups. Note potential compatibility issues on Arm-based CPUs and recommended Python version 3.9.

```bash
pip install tpot[sklearnex]
```

--------------------------------

### Data Preparation and Scorer Setup

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/8_SH_and_cv_early_pruning.ipynb

Prepares a classification dataset and sets up a scorer for model evaluation. This is a standard setup for supervised learning tasks in scikit-learn and TPOT.

```python
import matplotlib.pyplot as plt
import tpot
import time
import sklearn
import sklearn.datasets

scorer = sklearn.metrics.make_scorer(sklearn.metrics.roc_auc_score, needs_proba=True, multi_class='ovr')

X, y = sklearn.datasets.make_classification(n_samples=5000, n_features=20, n_classes=5, random_state=1, n_informative=15, n_redundant=5, n_repeated=0, n_clusters_per_class=3, class_sep=.8)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)
```

--------------------------------

### Install LightGBM for Arm-based CPUs

Source: https://github.com/epistasislab/tpot/blob/main/docs/installation.md

For M1 Mac or other Arm-based CPU users, install a compatible version of lightgbm from conda-forge before installing TPOT.

```bash
conda install --yes -c conda-forge 'lightgbm>=3.3.3'
```

--------------------------------

### DynamicUnionPipeline with FSSNode Example

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/3_Feature_Set_Selector.ipynb

Demonstrates the use of DynamicUnionPipeline, which can select a variable number of feature sets. Examples show generating pipelines with different random seeds, resulting in different feature set selections.

```python
dynamic_fss_space = DynamicUnionPipeline(fss_search_space)
dynamic_fss_space.generate(rng=1).export_pipeline()
```

```python
dynamic_fss_space.generate(rng=3).export_pipeline()
```

--------------------------------

### TPOT Estimator with Dask Context Manager

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/7_dask_parallelization.ipynb

This example demonstrates initializing a Dask LocalCluster and Client using a context manager, which automatically handles closing them. It configures a TPOTEstimator with a graph search space and trains it on the Iris dataset.

```python
from dask.distributed import Client, LocalCluster
import tpot
import sklearn
import sklearn.datasets
import numpy as np

scorer = sklearn.metrics.get_scorer('roc_auc_ovr')
X, y = sklearn.datasets.load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)


n_jobs = 4
memory_limit = "4GB"

with LocalCluster(  
    n_workers=n_jobs,
    threads_per_worker=1,
    memory_limit='4GB',
) as cluster, Client(cluster) as client:
    graph_search_space = tpot.search_spaces.pipelines.GraphSearchPipeline(
        root_search_space= tpot.config.get_search_space(["KNeighborsClassifier", "LogisticRegression", "DecisionTreeClassifier"]),
        leaf_search_space = tpot.config.get_search_space("selectors"), 
        inner_search_space = tpot.config.get_search_space(["transformers"]),
        max_size = 10,
        )

    est = tpot.TPOTEstimator(
        client = client,
        scorers = ["roc_auc_ovr"],
        scorers_weights = [1],
        classification = True,
        cv = 5,
        search_space = graph_search_space,
        max_time_mins = 60,
        early_stop=10,
        verbose = 2,
        )
    est.fit(X_train, y_train)
    print(scorer(est, X_test, y_test))
```

--------------------------------

### Create Dummy Dataset

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/3_Feature_Set_Selector.ipynb

Generates a classification dataset with both informative and uninformative features, then splits it into training and testing sets. This setup is used for demonstrating feature selection.

```python
import tpot
import sklearn.datasets
from sklearn.linear_model import LogisticRegression
import numpy as np
import pandas as pd
import tpot
import sklearn.datasets
from sklearn.linear_model import LogisticRegression
import numpy as np
from tpot.search_spaces.nodes import *
from tpot.search_spaces.pipelines import *
from tpot.config import get_search_space


X, y = sklearn.datasets.make_classification(n_samples=1000, n_features=6, n_informative=6, n_redundant=0, n_repeated=0, n_classes=2, n_clusters_per_class=2, weights=None, flip_y=0.01, class_sep=1.0, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
X = np.hstack([X, np.random.rand(X.shape[0],6)]) #add six uninformative features
X = pd.DataFrame(X, columns=['a','b','c','d','e','f','g','h','i', 'j', 'k', 'l']) # a, b ,c the rest are uninformative
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)

X.head()
```

--------------------------------

### TPOT Training Output

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/4_Genetic_Feature_Selection.ipynb

Example output from TPOT during the training process, showing the progress of generations.

```text
Generation: 100%|██████████| 10/10 [00:47<00:00,  4.73s/it]


```

--------------------------------

### Create Dummy Dataset and Split

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/4_Genetic_Feature_Selection.ipynb

Generates a synthetic dataset with a mix of informative and uninformative features and splits it into training and testing sets. This setup is used for demonstrating feature selection.

```python
import tpot
from tpot.search_spaces.nodes import *
from tpot.search_spaces.pipelines import *
import tpot
import sklearn.datasets
from sklearn.linear_model import LogisticRegression
import numpy as np
import pandas as pd
import tpot
import sklearn.datasets
from sklearn.linear_model import LogisticRegression
import numpy as np
from tpot.search_spaces.nodes import *
from tpot.search_spaces.pipelines import *
from tpot.config import get_search_space


X, y = sklearn.datasets.make_classification(n_samples=1000, n_features=6, n_informative=6, n_redundant=0, n_repeated=0, n_classes=2, n_clusters_per_class=2, weights=None, flip_y=0.01, class_sep=1.0, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
X = np.hstack([X, np.random.rand(X.shape[0],6)]) #add six uninformative features
X = pd.DataFrame(X, columns=['a','b','c','d','e','f','g','h','i', 'j', 'k', 'l']) # a, b ,c the rest are uninformative
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)

X.head()
```

--------------------------------

### Sample and Export Pipeline from TPOT Search Space

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/amltk_search_space_parser_example.ipynb

Use `generate()` to sample a pipeline from the TPOT search space and `export_pipeline()` to get its string representation. This is useful for understanding the types of pipelines TPOT can create.

```python
tpot_search_space.generate().export_pipeline()
```

--------------------------------

### TPOT Classifier with Warm Start Enabled

Source: https://context7.com/epistasislab/tpot/llms.txt

Initializes a TPOTClassifier with warm_start=True. Subsequent calls to fit() will continue optimization from the last generation, rather than restarting.

```python
import tpot
import sklearn.datasets
import sklearn.model_selection

X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y)

if __name__ == "__main__":
    # warm_start: continue from last generation on subsequent fit() calls
    clf3 = tpot.TPOTClassifier(
        search_space="linear-light",
        max_time_mins=10,
        warm_start=True,
    )
    clf3.fit(X_train, y_train)   # first run
    clf3.fit(X_train, y_train)   # continues from where it left off
```

--------------------------------

### Initialize Dask Local Cluster and Client

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/7_dask_parallelization.ipynb

Manually initialize a Dask LocalCluster and Client for parallelization. This setup allows for more control and access to the Dask dashboard. TPOT will use this client if provided, ignoring n_jobs and memory_limit.

```python
from dask.distributed import Client, LocalCluster

n_jobs = 4
memory_limit = "4GB"

cluster = LocalCluster(n_workers=n_jobs, #if no client is passed in and no global client exists, create our own
                        threads_per_worker=1,
                        memory_limit=memory_limit)
client = Client(cluster)
```

--------------------------------

### TPOT Estimator with Dask Parallelization

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/7_dask_parallelization.ipynb

Configure and run TPOT with Dask parallelization by setting n_jobs and memory_limit. This example demonstrates setting up TPOT for classification with a specified search space and evaluation metrics.

```python
import tpot
import sklearn
import sklearn.datasets
import numpy as np
scorer = sklearn.metrics.get_scorer('roc_auc_ovr')
X, y = sklearn.datasets.load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)


graph_search_space = tpot.search_spaces.pipelines.GraphSearchPipeline(
    root_search_space= tpot.config.get_search_space(["KNeighborsClassifier", "LogisticRegression", "DecisionTreeClassifier"]),
    leaf_search_space = tpot.config.get_search_space("selectors"), 
    inner_search_space = tpot.config.get_search_space(["transformers"]),
    max_size = 10,
    )

est = tpot.TPOTEstimator(
    scorers = ["roc_auc_ovr"],
    scorers_weights = [1],
    classification = True,
    cv = 10,
    search_space = graph_search_space,
    max_time_mins = 60,
    verbose = 2,
    n_jobs=16,
    memory_limit="4GB"
)

est.fit(X_train, y_train)
print(scorer(est, X_test, y_test))
```

--------------------------------

### Run TPOT Unit Tests

Source: https://github.com/epistasislab/tpot/blob/main/docs/contribute.md

Execute all unit tests for TPOT to ensure your changes have not introduced any regressions. This requires the `pytest` package to be installed.

```bash
$ pytest
```

--------------------------------

### GraphPipeline with Cross-Validation

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/5_GraphPipeline.ipynb

This example demonstrates using the `cross_val_predict_cv` parameter in GraphPipeline to potentially improve model performance. The pipeline is configured with cross-validation and then fitted and evaluated.

```python
est = GraphPipeline(g, cross_val_predict_cv=10)
est.plot()
est.fit(X_train, y_train)
print("score")
print(sklearn.metrics.roc_auc_score(y_test, est.predict_proba(X_test)[:,1]))
```

--------------------------------

### TPOT Estimator Configuration for Early Pruning

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/8_SH_and_cv_early_pruning.ipynb

Configure the TPOTEstimator with parameters for generations, time limits, scorers, cross-validation, and various early pruning and evaluation strategies. This setup is crucial for controlling the search process and optimizing performance.

```python
est = tpot.TPOTEstimator(
                            generations=10,
                            max_time_mins=None,
                            scorers=['roc_auc_ovr'],
                            scorers_weights=[1],
                            classification=True,
                            search_space = search_space,
                            population_size=30,
                            n_jobs=3,
                            cv=cv,
                            verbose=3,

                            initial_population_size=initial_population_size,
                            population_scaling = population_scaling,
                            generations_until_end_population = generations_until_end_population,
                            
                            budget_range = budget_range,
                            generations_until_end_budget=generations_until_end_budget,
                            
                            threshold_evaluation_pruning = threshold_evaluation_pruning,
                            threshold_evaluation_scaling = threshold_evaluation_scaling,

                            selection_evaluation_pruning  = selection_evaluation_pruning,
                            selection_evaluation_scaling = selection_evaluation_scaling,
                            )


start = time.time()
est.fit(X_train, y_train)
print(f"total time: {time.time()-start}")
print("test score: ", scorer(est, X_test, y_test))
```

--------------------------------

### TPOT Estimator Configuration for Early Pruning

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/8_SH_and_cv_early_pruning.ipynb

Configure the TPOTEstimator with parameters for generations, time limits, scoring, and search space. This setup is suitable for classification tasks and utilizes multiple CPU cores for parallel processing.

```python
import tpot
import sklearn
import sklearn.datasets
import numpy as np
import time
import tpot
import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
import sklearn

X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)
scorer = sklearn.metrics.make_scorer(sklearn.metrics.roc_auc_score, needs_proba=True, multi_class='ovr')


est = tpot.TPOTEstimator(
    generations=50,
    max_time_mins=None,
    scorers=['roc_auc_ovr'],
    scorers_weights=[1],
    classification=True,
    search_space = 'linear',
    n_jobs=32,
    cv=10,
    verbose=3,

    population_size=population_size,
    initial_population_size=initial_population_size,
    population_scaling = population_scaling,
    generations_until_end_population = generations_until_end_population,
    
    budget_range = budget_range,
    generations_until_end_budget=generations_until_end_budget,
    )
```

--------------------------------

### Saving and Loading TPOT Best Pipeline with Dill

Source: https://context7.com/epistasislab/tpot/llms.txt

This snippet shows how to save the best pipeline found by TPOT using the `dill` library and then load it for production use. This is useful for deploying trained models without retraining. Ensure `dill` and `scikit-learn` are installed.

```python
import tpot
import dill as pickle
import sklearn.datasets
import sklearn.model_selection

X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y)

if __name__ == "__main__":
    clf = tpot.TPOTClassifier(
        search_space="linear-light",
        max_time_mins=10,
        n_jobs=2,
        verbose=1,
    )
    clf.fit(X_train, y_train)

    # Save only the fitted pipeline (not the TPOT object)
    with open("best_pipeline.pkl", "wb") as f:
        pickle.dump(clf.fitted_pipeline_, f)

    # Load and use in production
    with open("best_pipeline.pkl", "rb") as f:
        pipeline = pickle.load(f)

    predictions = pipeline.predict(X_test)
    probabilities = pipeline.predict_proba(X_test)
    print(predictions[:5])          # [1 0 1 1 0]
    print(probabilities[:5, 1])     # [0.97 0.02 0.95 ...]
```

--------------------------------

### Full Linear Pipeline with Inner Layers

Source: https://context7.com/epistasislab/tpot/llms.txt

Construct a comprehensive linear pipeline by combining scalers, selectors, dynamic unions of classifiers, and final classifiers. This example demonstrates nesting various pipeline components for a complete workflow.

```python
# Full linear pipeline with optional inner classifiers + stacking
inner_layer = UnionPipeline([
    ChoicePipeline([
        DynamicUnionPipeline(wrapped, max_estimators=3),
        get_search_space("SkipTransformer"),
    ]),
    get_search_space("Passthrough"),
])

full_pipeline = SequentialPipeline([
    get_search_space("scalers"),
    get_search_space("selectors"),
    inner_layer,
    get_search_space("classifiers"),
])
print(full_pipeline.generate().export_pipeline())
```

--------------------------------

### Run TPOT Classifier from Python Script

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/1_Using_TPOT.ipynb

Protect your TPOT code with `if __name__=="__main__":` when running from a Python script to ensure proper parallelization with Dask. This example demonstrates setting up a TPOT classifier, fitting it to data, and evaluating its performance.

```python
from dask.distributed import Client, LocalCluster
import tpot
import sklearn
import sklearn.datasets
import numpy as np

if __name__=="__main__":
    scorer = sklearn.metrics.get_scorer('roc_auc_ovo')
    X, y = sklearn.datasets.load_digits(return_X_y=True)
    X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, train_size=0.75, test_size=0.25)


    est = tpot.TPOTClassifier(n_jobs=4, max_time_mins=3, verbose=2, early_stop=3)
    est.fit(X_train, y_train)


    print(scorer(est, X_test, y_test))
```

--------------------------------

### ModuleNotFoundError Example

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/amltk_search_space_parser_example.ipynb

This is an example of a ModuleNotFoundError that occurs when the 'amltk' library is not installed or accessible in the Python environment. It indicates a missing dependency required for the subsequent code to run.

```python
from sklearn.preprocessing import OneHotEncoder
from sklearn.svm import SVC
from amltk.pipeline import Choice, Component, Sequential, Split
import tpot
from sklearn.preprocessing import FunctionTransformer

# This code block is intended to show an error, not a functional example.
```

--------------------------------

### Get Linear Search Space in TPOT

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Retrieves the 'linear' search space configuration from TPOT. This is useful for setting up a standard linear pipeline optimization. Ensure TPOT is installed and imported.

```python
linear_search_space = tpot.config.template_search_spaces.get_template_search_spaces("linear", inner_predictors=True, cross_val_predict_cv=5)
linear_search_space.generate().export_pipeline()
```

--------------------------------

### Instantiate TPOTClassifier

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/1_Using_TPOT.ipynb

Create an instance of TPOTClassifier to begin the optimization process. This sets up the classifier for use.

```python
classification_optimizer = TPOTClassifier()
```

--------------------------------

### Using Built-in TPOT Search Spaces

Source: https://context7.com/epistasislab/tpot/llms.txt

Demonstrates how to use predefined search space strings with TPOTClassifier. It iterates through 'linear-light' and 'graph-light' to find the best pipeline. Accessing template search spaces with custom options is also shown.

```python
import tpot
import sklearn.datasets
import sklearn.model_selection

X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y)

# Available built-in search space strings:
# "linear"        — Selector → (Transformers+Passthrough) → (inner classifiers+Passthrough) → classifier
# "linear-light"  — Same as linear but smaller, faster estimator set; no inner classifiers
# "graph"         — DAG-structured pipeline (returns GraphPipeline)
# "graph-light"   — Same as graph but faster
# "mdr"           — Specialized for genome-wide association studies (GWAS)

if __name__ == "__main__":
    for space in ["linear-light", "graph-light"]:
        clf = tpot.TPOTClassifier(
            search_space=space,
            max_time_mins=5,
            n_jobs=2,
            verbose=1,
        )
        clf.fit(X_train, y_train)
        print(f"{space}: {clf.fitted_pipeline_}")

    # Access template search spaces with extra options
    custom_space = tpot.config.template_search_spaces.get_template_search_spaces(
        search_space="linear",
        classification=True,
        inner_predictors=True,     # allow stacked classifiers
        cross_val_predict_cv=5,    # use OOF predictions for inner classifiers
    )
    clf = tpot.TPOTEstimator(
        search_space=custom_space,
        scorers=["roc_auc_ovr"],
        scorers_weights=[1],
        classification=True,
        max_time_mins=30,
        n_jobs=4,
    )
    clf.fit(X_train, y_train)
```

--------------------------------

### Sample and Export DynamicLinearPipeline

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Initializes a DynamicLinearPipeline with a specified search space and maximum length, then samples and exports a pipeline. This is useful for generating random preprocessing or feature engineering sequences.

```python
import tpot.config


linear_feature_engineering = tpot.search_spaces.pipelines.DynamicLinearPipeline(search_space = tpot.config.get_search_space(["all_transformers","selectors_classification"])), max_length=10)
print("sampled pipeline")
linear_feature_engineering.generate().export_pipeline()
```

--------------------------------

### Initialize and Run TPOT Estimator

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Initializes a TPOTEstimator with a custom search space and runs the optimization process on the training data. The `verbose` parameter controls the output during optimization.

```python
est = tpot.TPOTEstimator(
    scorers = ["roc_auc"],
    scorers_weights = [1],
    classification = True,
    cv = 5,
    search_space = search_space,
    max_time_mins=10,
    max_eval_time_mins = 60*5,
    verbose = 2,
    n_jobs=20,
)

est.fit(X_train, y_train)
```

--------------------------------

### Get Dask Dashboard Link

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/7_dask_parallelization.ipynb

Retrieve the URL for the Dask dashboard, which provides a live view of the parallelization performance.

```python
client.dashboard_link
```

--------------------------------

### Visualize Population Size and Budget Scaling with Beta Interpolation

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/8_SH_and_cv_early_pruning.ipynb

Illustrates how population size and budget change over generations using stepwise beta interpolation. Useful for understanding the impact of parameters like `initial_population_size`, `population_scaling`, `generations_until_end_population`, `budget_range`, `budget_scaling`, `generations_until_end_budget`, and `stepwise_steps`.

```python
import matplotlib.pyplot as plt
import tpot

population_size=30
initial_population_size=100
population_scaling = .5
generations_until_end_population = 50

budget_range = [.3,1]
generations_until_end_budget=50
budget_scaling = .5
stepwise_steps = 5

#Population and budget use stepwise
fig, ax1 = plt.subplots()
ax2 = ax1.twinx()

interpolated_values_population = tpot.utils.beta_interpolation(start=initial_population_size, end=population_size, n=generations_until_end_population, n_steps=stepwise_steps, scale=population_scaling)
interpolated_values_budget = tpot.utils.beta_interpolation(start=budget_range[0], end=budget_range[1], n=generations_until_end_budget, n_steps=stepwise_steps, scale=budget_scaling)
ax1.step(list(range(len(interpolated_values_population))), interpolated_values_population, label=f"population size")
ax2.step(list(range(len(interpolated_values_budget))), interpolated_values_budget, label=f"budget", color='r')
ax1.set_xlabel("generation")
ax1.set_ylabel("population size")
ax2.set_ylabel("bugdet")

ax1.legend(loc='center left', bbox_to_anchor=(1.1, 0.4))
ax2.legend(loc='center left', bbox_to_anchor=(1.1, 0.3))
plt.show()
```

--------------------------------

### TPOT Pipeline Exported Result

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/4_Genetic_Feature_Selection.ipynb

This is an example of a TPOT pipeline that has been exported, showcasing the selected features and the sequence of transformers and classifiers used.

```python
Pipeline(steps=[('maskselector',
                 MaskSelector(mask=array([False, False,  True, False, False, False, False, False, False,
        True, False, False]))),
                ('pipeline',
                 Pipeline(steps=[('normalizer', Normalizer(norm='l1')),
                                 ('selectpercentile',
                                  SelectPercentile(percentile=74.2561844719571)),
                                 ('featureunion-1',
                                  FeatureUnion(transformer_list=[('featureunion',
                                                                  FeatureUnion(transformer_list=[('binarizer',
                                                                                                  Binarizer(threshold=0.0935770250992))])),
                                                                 ('passthrough',
                                                                  Passthrough())])),
                                 ('featureunion-2',
                                  FeatureUnion(transformer_list=[('skiptransformer',
                                                                  SkipTransformer()),
                                                                 ('passthrough',
                                                                  Passthrough())])),
                                 ('adaboostclassifier',
                                  AdaBoostClassifier(algorithm='SAMME',
                                                     learning_rate=0.9665397922726,
                                                     n_estimators=320))]))])
```

--------------------------------

### Initialize and Plot TreePipeline

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Create an instance of the experimental TreePipeline, which is similar to GraphSearchPipeline but restricted to tree structures. Generate an individual and plot its pipeline.

```python
tree_search_space = tpot.search_spaces.pipelines.TreePipeline(
    root_search_space= tpot.config.get_search_space(["KNeighborsClassifier", "LogisticRegression", "DecisionTreeClassifier"]),
    leaf_search_space = tpot.config.get_search_space("selectors"), 
    inner_search_space = tpot.config.get_search_space(["transformers"]),
    max_size = 10,
)

ind = graph_search_space.generate()
exp = ind.export_pipeline()
exp.plot()

```

--------------------------------

### Get Search Space for All Classifiers

Source: https://context7.com/epistasislab/tpot/llms.txt

Retrieves a ChoicePipeline over all available classifiers. The generated pipeline will randomly select one of the listed estimators.

```python
classifiers = tpot.config.get_search_space("classifiers")
print(classifiers.generate().export_pipeline())
```

--------------------------------

### Import TPOT Configuration Modules

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/8_SH_and_cv_early_pruning.ipynb

Imports necessary modules from the TPOT library for configuring search spaces and estimators.

```python
import tpot.config
import tpot.config.template_search_spaces
import tpot.search_spaces
```

--------------------------------

### Get Objective Column Names

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/1_Using_TPOT.ipynb

Retrieves the names of the objective functions used by TPOT during optimization. These names correspond to the columns in the 'evaluated_individuals' DataFrame.

```python
est.objective_names
```

--------------------------------

### Get Search Space for Specific Classifiers

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Obtain a search space for a predefined list of classifiers. Use this to sample pipelines with specific algorithms.

```python
#same pipeline search space as before.
classifier_choice = tpot.config.get_search_space(["KNeighborsClassifier", "LogisticRegression", "DecisionTreeClassifier"])

print("sampled pipeline 1")
classifier_choice.generate().export_pipeline()
```

```python
print("sampled pipeline 2")
classifier_choice.generate().export_pipeline()
```

--------------------------------

### Constructing a Sequential Pipeline with Choices and Splits

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/amltk_search_space_parser_example.ipynb

This snippet demonstrates building a machine learning pipeline using AMLTK's Sequential, Split, and Choice components. It includes feature selection, transformation, and estimator choices, defining a search space for hyperparameter optimization.

```python
from tpot.builtin_modules import Passthrough, ZeroCount
from sklearn.preprocessing import PolynomialFeatures
from sklearn.decomposition import PCA

from sklearn.feature_selection import VarianceThreshold, SelectKBest

selectors = Choice(
    Component(VarianceThreshold, space={"threshold": (0.1,1)}),
    Component(SelectKBest, space={"k": (1, 10)}),
    name="selectors",
)


transformers = Split(
    {
        "passthrough": Passthrough(),
        "polynomial": Component(PolynomialFeatures, space={"degree": [2, 3]}),
        "zerocount" : ZeroCount(),
    },
    # config={"categories": select_categories, "numerics": select_numerical},
    name="transformers",
)

pipeline = (
    Sequential(name="my_pipeline")
    >> split_imputation
    # >> Component(SimpleImputer, space={"strategy": ["mean", "median"]})  # Choose either mean or median
    
    >> selectors
    >> transformers
    >> Choice(
        # Our pipeline can choose between two different estimators
        Component(
            RandomForestClassifier,
            space={"n_estimators": (10, 100), "criterion": ["gini", "log_loss"]},
            config={"max_depth": 3},
        ),
        Component(SVC, space={"kernel": ["linear", "rbf", "poly"]}),
        name="estimator",
    )
)

```

--------------------------------

### Get Search Space for All Classifiers

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Retrieve a search space encompassing all available classifiers in TPOT. Useful for exploring a wide range of classification models.

```python
#search space for all classifiers
classifier_choice = tpot.config.get_search_space("classifiers")

print("sampled pipeline 1")
classifier_choice.generate().export_pipeline()
```

```python
print("sampled pipeline 2")
classifier_choice.generate().export_pipeline()
```

--------------------------------

### Composing Pipeline Search Spaces

Source: https://context7.com/epistasislab/tpot/llms.txt

Shows how to construct pipelines with fixed or variable structures using SequentialPipeline, DynamicLinearPipeline, and UnionPipeline primitives.

```python
import tpot
from tpot.search_spaces.pipelines import (
    SequentialPipeline, DynamicLinearPipeline,
    UnionPipeline, DynamicUnionPipeline, ChoicePipeline
)
from tpot.config import get_search_space

# SequentialPipeline: fixed-order steps, each from a different search space
fixed_pipeline = SequentialPipeline([
    get_search_space("selectors"),
    get_search_space("transformers"),
    get_search_space("classifiers"),
])
print(fixed_pipeline.generate().export_pipeline())
# Pipeline(steps=[('selectpercentile', ...), ('pca', ...), ('lgbmclassifier', ...)])

# DynamicLinearPipeline: variable number of steps drawn from one search space
dynamic_feature_eng = DynamicLinearPipeline(
    search_space=get_search_space(["all_transformers", "selectors_classification"]),
    max_length=5,
)
print(dynamic_feature_eng.generate().export_pipeline())
# Pipeline(steps=[('pca', ...), ('quantiletransformer', ...)])
```

--------------------------------

### Get Search Space for Custom List of Classifiers

Source: https://context7.com/epistasislab/tpot/llms.txt

Creates a ChoicePipeline over a specified list of classifiers. This allows for a custom selection of estimators to be included in the search.

```python
custom = tpot.config.get_search_space(
    ["KNeighborsClassifier", "LogisticRegression", "XGBClassifier"],
    random_state=1,
)
print(custom.generate().export_pipeline())
```

--------------------------------

### Configure and Train TPOT Estimator

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/amltk_search_space_parser_example.ipynb

Instantiate a `TPOTEstimator` with custom search space, scorers, and training parameters. The `fit()` method trains the estimator on the provided data.

```python
est = tpot.TPOTEstimator(
    scorers = ["roc_auc"],
    scorers_weights = [1],
    classification = True,
    cv = 5,
    search_space = tpot_search_space, #converted search space goes here
    population_size= 10,
    generations = 2,
    max_eval_time_mins = 60*5,
    verbose = 5,
    n_jobs=10,
)

est.fit(X_train, y_train)
```

--------------------------------

### Get Search Space for RandomForestClassifier with Fixed Random State

Source: https://context7.com/epistasislab/tpot/llms.txt

Retrieves the search space configuration for RandomForestClassifier, ensuring reproducibility by setting a fixed random_state.

```python
rf_node_repro = tpot.config.get_search_space("RandomForestClassifier", random_state=42)
```

--------------------------------

### Initialize and run TPOT for symbolic classification

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/6_Symbolic_Regression_and_Classification.ipynb

Initializes a TPOTClassifier with custom search space, objectives, and scorers, then fits it to the training data and evaluates on the test set. The fitted pipeline's structure is also plotted.

```python
est = tpot.TPOTEstimator(  generations=20, 
                            max_time_mins=None,
                            scorers=['roc_auc_ovr'],
                            scorers_weights=[1],
                            other_objective_functions=[tpot.objectives.number_of_nodes_objective],
                            other_objective_functions_weights=[-1],
                            n_jobs=32,
                            classification=True,
                            search_space = symbolic_classification_search_space,
                            verbose=1,
                            )

scorer = sklearn.metrics.get_scorer('roc_auc_ovo')

est.fit(X_train, y_train)
print(scorer(est, X_test, y_test))
est.fitted_pipeline_.plot()
```

--------------------------------

### Define Pipeline with FSS and Classifier

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/3_Feature_Set_Selector.ipynb

Constructs a TPOT pipeline that starts with a FeatureSetSelector and is followed by a classifier. This integrates the FSS into a complete model training workflow.

```python
classification_search_space = get_search_space(["RandomForestClassifier"])
fss_and_classifier_search_space = SequentialPipeline([fss_search_space, classification_search_space])


est = tpot.TPOTEstimator(generations=5, 
                            scorers=["roc_auc_ovr", tpot.objectives.complexity_scorer],
                            scorers_weights=[1.0, -1.0],
                            n_jobs=32,
                            classification=True,
                            search_space = fss_and_classifier_search_space,
                            verbose=1,
                            )


scorer = sklearn.metrics.get_scorer('roc_auc_ovr')
est.fit(X_train, y_train)
print(scorer(est, X_test, y_test))
```

--------------------------------

### Exported Pipeline with FeatureSetSelector

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/3_Feature_Set_Selector.ipynb

This is an example of a TPOT pipeline that has been exported, showcasing the configuration of the FeatureSetSelector with a named subset 'group_two' and specified features ['d', 'e', 'f'].

```python
Pipeline(steps=[('featuresetselector',
                 FeatureSetSelector(name='group_two',
                                    sel_subset=['d', 'e', 'f'])),
                ('pipeline',
                 Pipeline(steps=[('maxabsscaler', MaxAbsScaler()),
                                 ('rfe',
                                  RFE(estimator=ExtraTreesClassifier(max_features=0.0390676831531,
                                                                     min_samples_leaf=8,
                                                                     min_samples_split=14,
                                                                     n_jobs=1), step=0.753983388654)),
                                 ('featureunion-1',
                                  FeatureUnion(transformer_lis...
                                  FeatureUnion(transformer_list=[('skiptransformer',
                                                                  SkipTransformer()),
                                                                 ('passthrough',
                                                                  Passthrough())])),
                                 ('histgradientboostingclassifier',
                                  HistGradientBoostingClassifier(early_stopping=True,
                                                                 l2_regularization=9.1304e-09,
                                                                 learning_rate=0.0036310282582,
                                                                 max_features=0.238877814721,
                                                                 max_leaf_nodes=1696,
                                                                 min_samples_leaf=59,
                                                                 n_iter_no_change=14,
                                                                 tol=0.0001,
                                                                 validation_fraction=None))]))])
```

--------------------------------

### SequentialPipeline with General Categories

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Build a `SequentialPipeline` where each step samples from a general category of estimators (e.g., 'selectors', 'transformers', 'classifiers'). This allows TPOT to choose from a broader set of options for each stage.

```python
selector_choicepipeline = tpot.config.get_search_space("selectors")
transformer_choicepipeline =  tpot.config.get_search_space("transformers")
classifier_choicepipeline = tpot.config.get_search_space("classifiers")

stc_pipeline = tpot.search_spaces.pipelines.SequentialPipeline([
    selector_choicepipeline,
    transformer_choicepipeline,
    classifier_choicepipeline,
])

print("sampled pipeline")
stc_pipeline.generate().export_pipeline()
```

```python
print("sampled pipeline")
stc_pipeline.generate().export_pipeline()
```

--------------------------------

### Generate and Mutate Pipeline from ChoicePipeline

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Demonstrates generating a random pipeline from the ChoicePipeline and then mutating it. This shows how TPOT samples and modifies pipelines within the defined search space.

```python
classifier_individual = classifier_node.generate()

print("sampled pipeline")
classifier_individual.export_pipeline()

print("mutated pipeline")
classifier_individual.mutate()
classifier_individual.export_pipeline()
```

--------------------------------

### UnionPipeline with FSSNode Example

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/3_Feature_Set_Selector.ipynb

This snippet shows how to create a UnionPipeline that selects exactly two feature sets. The generated pipeline can then be configured for pandas output, fitted, and transformed.

```python
union_fss_space = UnionPipeline([fss_search_space, fss_search_space])
```

```python
# this union search space will always select exactly two fss_search_space
selector1 = union_fss_space.generate(rng=1).export_pipeline()
```

```python
selector1.set_output(transform="pandas") 
selector1.fit(X_train)
selector1.transform(X_train)
```

--------------------------------

### TPOT Search Space Configuration

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/8_SH_and_cv_early_pruning.ipynb

Defines the search space for TPOT, including selectors, estimators, scalers, and transformers. This setup is crucial for TPOT's automated pipeline optimization.

```python
import tpot
from tpot.search_spaces.pipelines import *
from tpot.search_spaces.nodes import *
from tpot.config.get_configspace import get_search_space
import sklearn.model_selection
import sklearn


selectors = get_search_space(["selectors","selectors_classification", "Passthrough"], random_state=42,)
estimators = get_search_space(['XGBClassifier'],random_state=42,)

scalers = get_search_space(["scalers","Passthrough"],random_state=42,)

transformers_layer =UnionPipeline([
                        ChoicePipeline([
                            DynamicUnionPipeline(get_search_space(["transformers"], random_state=42,)),
                            get_search_space("SkipTransformer"),
                        ]),
                        get_search_space("Passthrough")
                        ]
                    )
    
search_space = SequentialPipeline(search_spaces=[
                                            scalers,
                                            selectors, 
                                            transformers_layer,
                                            estimators,
                                            ])
```

--------------------------------

### Initialize GraphSearchPipeline

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Instantiate a GraphSearchPipeline with specified search spaces for root, leaf, and inner nodes. This defines the structure and components TPOT can use to build DAG pipelines.

```python
graph_search_space = tpot.search_spaces.pipelines.GraphSearchPipeline(
    root_search_space= tpot.config.get_search_space(["KNeighborsClassifier", "LogisticRegression", "DecisionTreeClassifier"]),
    leaf_search_space = tpot.config.get_search_space("selectors"), 
    inner_search_space = tpot.config.get_search_space(["transformers"]),
    max_size = 10,
)

ind = graph_search_space.generate()

```

--------------------------------

### Get Linear Template Search Space

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Retrieves the search space configuration for linear models. Use 'inner_predictors=True' to include inner predictors and 'cross_val_predict_cv' to specify cross-validation folds.

```python
linear_search_space = tpot.config.template_search_spaces.get_template_search_spaces("linear", inner_predictors=True, cross_val_predict_cv=5)
```

--------------------------------

### Create WrapperPipeline for SelectFromModel

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Constructs a WrapperPipeline for SelectFromModel, integrating a custom search space for its own hyperparameters and an exported search space for its inner estimator (ExtraTreesClassifier).

```python
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.feature_selection import SelectFromModel

select_from_model_wrapper_searchspace = tpot.search_spaces.pipelines.WrapperPipeline(
            method=SelectFromModel,
            space = SelectFromModel_configspace_part,
            estimator_search_space= extratrees_estimator_node,
        )

select_from_model_wrapper_searchspace.generate().export_pipeline()
```

--------------------------------

### Feature Set Selector Configuration

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/3_Feature_Set_Selector.ipynb

Example of configuring a FeatureSetSelector within a TPOT pipeline. This specific configuration selects a subset of features named 'group_one' consisting of features 'a', 'b', and 'c'.

```python
Pipeline(steps=[('featuresetselector',
                 FeatureSetSelector(name='group_one',
                                    sel_subset=['a', 'b', 'c'])),
                ('randomforestclassifier',
                 RandomForestClassifier(max_features=0.30141491087,
                                        min_samples_leaf=4,
                                        min_samples_split=17, n_estimators=128,
                                        n_jobs=1))])
```

--------------------------------

### Clone TPOT Repository

Source: https://github.com/epistasislab/tpot/blob/main/docs/contribute.md

Clone your forked copy of the TPOT repository to your local machine and navigate into the directory.

```bash
$ git clone git@github.com:YourUsername/tpot.git
$ cd tpot
```

--------------------------------

### Configure Genetic Feature Selector

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/4_Genetic_Feature_Selection.ipynb

Configure the Genetic Feature Selector with specified scorers, weights, number of jobs, and search space. This setup is used for feature selection in classification tasks.

```python
scorers=["roc_auc_ovr", tpot.objectives.complexity_scorer],
                            scorers_weights=[1.0, -1.0],
                            n_jobs=32,
                            classification=True,
                            search_space = final_classification_search_space,
                            verbose=1,
                            )
```

--------------------------------

### TPOT Classifier Resuming from Checkpoint

Source: https://context7.com/epistasislab/tpot/llms.txt

Initializes a TPOTClassifier pointing to an existing checkpoint folder to resume a previously interrupted optimization run. This allows continuing the search without starting from scratch.

```python
import tpot
import sklearn.datasets
import sklearn.model_selection

X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y)

if __name__ == "__main__":
    # Resume from checkpoint (e.g., after crash or to extend search time)
    clf2 = tpot.TPOTClassifier(
        search_space="linear",
        max_time_mins=30,
        n_jobs=4,
        periodic_checkpoint_folder="./tpot_checkpoints",  # picks up where it left off
        verbose=2,
    )
    clf2.fit(X_train, y_train)
```

--------------------------------

### Initialize and Fit TPOTEstimator with Custom Search Space

Source: https://github.com/epistasislab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb

Initializes and fits the TPOTEstimator using a selected search space. Configure multiple scorers, weights, cross-validation, time limits, and verbosity for the optimization process.

```python
selected_search_space = all_search_spaces["stc_pipeline"] #change this to select a different search space


est = tpot.TPOTEstimator(
    scorers=["roc_auc_ovr", tpot.objectives.complexity_scorer],
    scorers_weights=[1.0, -1.0],
    classification = True,
    cv = 5,
    search_space = selected_search_space,
    max_time_mins=10,
    max_eval_time_mins = 10,
    early_stop = 2,
    verbose = 2,
    n_jobs=4,
)

est.fit(X_train, y_train)
```