### Install PySCIPOpt-ML from Source

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/basics.rst

After cloning the PySCIPOpt-ML repository, this command installs the package directly from the local source directory using pip.

```bash
python -m pip install .
```

--------------------------------

### Python Function for Wine Manufacturer Optimization Setup

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_advanced.rst

This Python function, `build_and_optimise_wine_manufacturer`, sets up and trains machine learning models (Random Forest or Gradient Boosting Regressor from scikit-learn, XGBoost, or LightGBM) to predict wine quality based on various features. It loads wine quality data, prepares features, and then trains the specified ML model, which will later be used within a MIP formulation.

```python
def build_and_optimise_wine_manufacturer(
        seed=42,
        n_vineyards=35,
        n_wines_to_produce=5,
        min_wine_quality=4.25,
        sklearn_xgboost_lightgbm="sklearn",
        gbdt_or_rf="rf",
        n_estimators=3,
        max_depth=3,
        epsilon=0.0001,
    ):
        assert sklearn_xgboost_lightgbm in ("sklearn", "xgboost", "lightgbm")
        assert gbdt_or_rf in ("gbdt", "rf")

        # Path to red wine data
        data_dict = read_csv_to_dict("./tests/data/wineQualityReds.csv")

        features = [
            "fixed.acidity",
            "volatile.acidity",
            "citric.acid",
            "residual.sugar",
            "chlorides",
            "free.sulfur.dioxide",
            "total.sulfur.dioxide",
            "density",
            "pH",
            "sulphates",
            "alcohol",
        ]
        n_features = len(features)
        budget = 1.7 * n_wines_to_produce

        # Generate the actual input data arrays for the ML predictors
        X = []
        quality = np.array([float(x) for x in data_dict["quality"]]).reshape(
            -1,
        )
        for feature in features:
            X.append(np.array([float(x) for x in data_dict[feature]]))
        X = np.swapaxes(np.array(X), 0, 1)

        # Train the ML predictor
        if sklearn_xgboost_lightgbm == "sklearn":
            if gbdt_or_rf == "rf":
                reg = RandomForestRegressor(
                    random_state=seed, n_estimators=n_estimators, max_depth=max_depth
                ).fit(X, quality)
            else:
                reg = GradientBoostingRegressor(
                    random_state=seed, n_estimators=n_estimators, max_depth=max_depth
                ).fit(X, quality)
        elif sklearn_xgboost_lightgbm == "xgboost":
            if gbdt_or_rf == "gbdt":
                reg = XGBRegressor(
                    random_state=seed, n_estimators=n_estimators, max_depth=max_depth
                ).fit(X, quality)
            else:
                reg = XGBRFRegressor(
                    random_state=seed, n_estimators=n_estimators, max_depth=max_depth
                ).fit(X, quality)
```

--------------------------------

### Install pytest and run PySCIPOpt-ML tests

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Installs the pytest framework within the activated virtual environment and then executes the test suite for PySCIPOpt-ML to verify functionality.

```Shell
(venv) pip install pytest
(venv) pytest
```

--------------------------------

### Install documentation build requirements for PySCIPOpt-ML

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Installs additional Python packages specified in 'docs/requirements.txt' that are necessary to build the PySCIPOpt-ML documentation locally using Sphinx.

```Shell
pip install -r docs/requirements.txt
```

--------------------------------

### Install PySCIPOpt-ML via pip

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Installs the PySCIPOpt-ML package and its core dependencies (numpy, pyscipopt) using pip within a virtual environment. This is the easiest and recommended installation method.

```Shell
(venv) pip install pyscipopt-ml
```

--------------------------------

### Install PySCIPOpt-ML via pip

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/index.rst

Installs the PySCIPOpt-ML library from PyPI using pip. This command fetches the latest stable version of the package. Python 3.8 or higher is required for installation.

```bash
pip install pyscipopt-ml
```

--------------------------------

### Install PySCIPOpt-ML via pip

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/basics.rst

This command installs the PySCIPOpt-ML package and its core dependencies like pyscipopt and numpy using pip, preferably within a virtual environment.

```console
(venv) pip install pyscipopt-ml
```

--------------------------------

### Build PySCIPOpt-ML documentation locally

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Executes the Sphinx build command to generate local documentation for PySCIPOpt-ML. This command should be run after installing the documentation requirements.

```Shell
sphinx-build docs docs/_build
```

--------------------------------

### Build and Optimize SCIP Model with XGBoost/Random Forest Predictors

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_advanced.rst

This Python code demonstrates how to build and optimize a SCIP model with embedded machine learning predictors using XGBoost and Random Forests. It initializes the SCIP model with specific parameters for a wine manufacturing scenario, optimizes it, and then performs an error check on the MIP embedding by comparing SKLearn and SCIP outputs.

```python
# Build the SCIP model with embedded ML predictors
scip = build_and_optimise_wine_manufacturer(
    seed=42,
    n_vineyards=35,
    n_wines_to_produce=5,
    min_wine_quality=4.25,
    sklearn_xgboost_lightgbm="xgboost",
    gbdt_or_rf="rf",
    n_estimators=3,
    max_depth=3,
    epsilon=0.0001
    )

# Optimise the SCIP model
scip.optimize()

# We can check the "error" of the MIP embedding via the difference between SKLearn and SCIP output
if np.max(pred_cons.get_error()) > 10**-3:
    error = np.max(pred_cons.get_error())
    raise AssertionError(f"Max error {error} exceeds threshold of {10 ** -3}")

return scip
```

--------------------------------

### Install PySCIPOpt-ML from cloned source

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Installs the PySCIPOpt-ML package from a locally cloned repository using pip. This command should be run from within the root directory of the cloned repository, typically in a virtual environment.

```Shell
(venv) python -m pip install .
```

--------------------------------

### Install development dependencies for PySCIPOpt-ML

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Installs pytest, pre-commit, and all supported machine learning frameworks (scikit-learn, torch, tensorflow, xgboost, lightgbm, onnx, onnxruntime) required for PySCIPOpt-ML development and testing.

```Shell
pip install pytest
pip install scikit-learn
pip install torch
pip install tensorflow
pip install xgboost
pip install lightgbm
pip install onnx
pip install onnxruntime
pip install pre-commit
pre-commit install
```

--------------------------------

### Build SCIP Optimization Model with LightGBM Predictor for Wine Blending (Python)

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_advanced.rst

This Python code constructs a SCIP optimization model for a wine blending problem. It first trains a LightGBM regressor (either GBDT or Random Forest) to predict wine quality. Then, it initializes a SCIP model, defines decision variables for wine features, quality, and vineyard mixtures, and adds various constraints including mixture proportions, vineyard limits, and a total budget. Crucially, the trained LightGBM model is integrated as a constraint using `add_predictor_constr` to enforce the predicted quality. Finally, the objective is set to maximize the average quality of the produced wines.

```python
        else:
            if gbdt_or_rf == "gbdt":
                reg = LGBMRegressor(
                    random_state=seed, n_estimators=n_estimators, max_depth=max_depth
                ).fit(X, quality)
            else:
                reg = LGBMRegressor(
                    random_state=seed,
                    n_estimators=n_estimators,
                    max_depth=max_depth,
                    boosting_type="rf",
                    bagging_freq=1,
                    bagging_fraction=0.5,
                ).fit(X, quality)

        # Create artificial data from some vineyards
        np.random.seed(seed)
        vineyard_order = np.arange(X.shape[0])
        np.random.shuffle(vineyard_order)
        vineyard_litre_limits = np.random.uniform(0.25, 0.35, n_vineyards)
        vineyard_costs = np.random.uniform(1, 2, n_vineyards)
        vineyard_features = []
        low_quality_vineyards_i = 0
        for i in vineyard_order:
            if low_quality_vineyards_i >= n_vineyards:
                break
            if quality[i] <= 5:
                low_quality_vineyards_i += 1
                vineyard_features.append(X[i])
        vineyard_features = np.array(vineyard_features)

        # Create the SCIP Model
        scip = Model()

        # Create variables deciding the features of each wine
        feature_vars = np.zeros((n_wines_to_produce, n_features), dtype=object)
        quality_vars = np.zeros((n_wines_to_produce, 1), dtype=object)
        wine_mixture_vars = np.zeros((n_wines_to_produce, n_vineyards), dtype=object)

        for i in range(n_wines_to_produce):
            quality_vars[i][0] = scip.addVar(vtype="C", lb=0, ub=10, name=f"quality_{i}")
            for j in range(n_features):
                max_val = np.max(X[:, j])
                min_val = np.min(X[:, j])
                lb = max(0, min_val - 0.1 * max_val)
                ub = 1.1 * max_val
                feature_vars[i][j] = scip.addVar(vtype="C", lb=lb, ub=ub, name=f"feature_{i}_{j}")
            for k in range(n_vineyards):
                wine_mixture_vars[i][k] = scip.addVar(
                    vtype="C", lb=0, ub=vineyard_litre_limits[k], name=f"mixture_{i}_{k}"
                )

        # Now create constraints on the wine blending
        for i in range(n_wines_to_produce):
            for j in range(n_features):
                scip.addCons(
                    feature_vars[i][j]
                    == quicksum(
                        wine_mixture_vars[i][k] * vineyard_features[k][j] for k in range(n_vineyards)
                    ),
                    name=f"mixture_cons_{i}_{j}",
                )
        for i in range(n_wines_to_produce):
            scip.addCons(
                quicksum(wine_mixture_vars[i][k] for k in range(n_vineyards)) == 1,
                name=f"wine_mix_{i}",
            )
        for k in range(n_vineyards):
            scip.addCons(
                quicksum(wine_mixture_vars[i][k] for i in range(n_wines_to_produce))
                <= vineyard_litre_limits[k],
                name=f"vineyard_limit_{k}",
            )

        # Add the budget constraint
        scip.addCons(
            quicksum(
                quicksum(wine_mixture_vars[i][k] * vineyard_costs[k] for k in range(n_vineyards))
                for i in range(n_wines_to_produce)
            )
            <= budget,
            name=f"budget_cons",
        )

        # Add the ML constraint. Add in a single batch!
        pred_cons = add_predictor_constr(
            scip, reg, feature_vars, quality_vars, unique_naming_prefix="wine_", epsilon=epsilon
        )

        # Add a constraint ensuring minimum wine quality on those produced
        for i in range(n_wines_to_produce):
            scip.addCons(quality_vars[i][0] >= min_wine_quality, name=f"min_quality_{i}")

        # Set the SCIP objective
        scip.setObjective(
            quicksum(quality_vars[i][0] for i in range(n_wines_to_produce)) / n_wines_to_produce,
            sense="maximize"
        )

        return scip
```

--------------------------------

### Decision Tree Internal Node Split Example

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Provides a concrete example of how decisions are formulated for an internal node split in a decision tree, based on a feature and a threshold value.

```APIDOC
x_4 \leq 5

x_4 \geq 5
```

--------------------------------

### Train ML Predictor for Function Approximation

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_basic.rst

This function trains a machine learning predictor (either Scikit-Learn MLPRegressor or PyTorch Sequential model) to approximate the previously generated non-linear functions. It takes parameters for the model type, number of inputs, samples, and layer sizes. The function uses `build_random_quadratic_functions` to get training data and then fits the chosen ML model.

```python
def build_and_optimise_function_approximation_model(
seed=42, n_inputs=5, n_samples=1000, sklearn_or_torch="sklearn", layers_sizes=(20, 20, 10)
):
    assert len(layers_sizes) == 3

    X, y_1, y_2 = build_random_quadratic_functions(
        seed=seed, n_inputs=n_inputs, n_samples=n_samples
    )

    if sklearn_or_torch == "sklearn":
        reg_1 = MLPRegressor(
            random_state=seed,
            hidden_layer_sizes=(layers_sizes[0], layers_sizes[1], layers_sizes[2]),
        ).fit(X, y_1.reshape(-1))
        reg_2 = MLPRegressor(
            random_state=seed,
            hidden_layer_sizes=(layers_sizes[0], layers_sizes[1], layers_sizes[2]),
        ).fit(X, y_2.reshape(-1))
    else:
        torch.random.manual_seed(seed)
        reg_1 = nn.Sequential(
            nn.Linear(n_inputs, layers_sizes[0]),
            nn.ReLU(),
            nn.Linear(layers_sizes[0], layers_sizes[1]),
            nn.ReLU(),
            nn.Linear(layers_sizes[1], layers_sizes[2]),
```

--------------------------------

### Clone PySCIPOpt-ML repository via HTTPS

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Clones the PySCIPOpt-ML GitHub repository using HTTPS. This method is suitable for users who prefer not to set up SSH keys and want to install from source.

```Shell
git clone https://github.com/Opt-Mucca/PySCIPOpt-ML/
```

--------------------------------

### Set up virtual environment and PYTHONPATH for PySCIPOpt-ML testing

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Creates and activates a Python virtual environment, then appends the current working directory to the PYTHONPATH. This prepares the environment for running PySCIPOpt-ML tests correctly.

```Shell
python -m venv venv
source venv/bin/activate
export PYTHONPATH="$(pwd):${PYTHONPATH}"
```

--------------------------------

### Initialize SCIP Model with Input/Output Variables and Constraints

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_basic.rst

This function sets up a basic SCIP optimization model. It initializes a SCIP Model instance, defines continuous input variables with specified bounds, and creates output variables. It also adds a fixed equality constraint on one output variable and sets the objective function to maximize another output variable.

```python
def build_basic_scip_model(n_inputs):
    # Initialise a SCIP Model
    scip = Model()

    # Create the input variables
    input_vars = np.zeros((1, n_inputs), dtype=object)
    for i in range(n_inputs):
        # Tight bounds are important for MIP formulations of neural networks. They often drastically improve
        # performance. As our training data is in the range [-10, 10], we pass that as bounds [-10, 10].
        # These bounds will then propagate to other variables.
        input_vars[0][i] = scip.addVar(name=f"x_{i}", vtype="C", lb=-10, ub=10)

    # Create the output variables. (Note that these variables will be automatically constructed if not specified)
    output_vars = np.zeros((2, 1), dtype=object)
    for i in range(2):
        output_vars[i] = scip.addVar(name=f"y_{i}", vtype="C", lb=None, ub=None)

    # Now set additional constraints and set the objective
    scip.addCons(output_vars[1][0] == 10, name="fix_output_reg_2")
    scip.setObjective(output_vars[0][0])

    return scip, input_vars, output_vars
```

--------------------------------

### Dense Layer ReLU Activation (SOS1 Formulation)

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Explains the non-standard SOS1 formulation for ReLU activation in dense layers, involving slack variables and SOS1 constraints. Notes its difference from big-M and potential performance implications.

```APIDOC
y_j = \sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j} + s_j \quad &\forall j
SOS1(y_j, s_j) \quad &\forall j
```

--------------------------------

### Dense Layer ReLU Activation (Big-M Formulation)

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Describes the traditional big-M formulation for ReLU, using binary variables and bounds. Highlights the requirement for user-provided input bounds and potential issues with large big-M values.

```APIDOC
y_j \geq 0

y_j \geq \sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j} \quad &\forall j

y_j \leq \sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j} - (1 - z_j) L_j \quad &\forall j

y_j \leq z_j U_j \quad &\forall j
```

--------------------------------

### API Documentation: pyscipopt_ml.sklearn.multi_output Module

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/MultiOutputConstr.rst

Provides an overview of the `pyscipopt_ml.sklearn.multi_output` module. This module encapsulates functionalities for integrating multi-output machine learning models with SCIP constraints, offering tools to formulate complex optimization problems.

```APIDOC
Module: pyscipopt_ml.sklearn.multi_output
```

--------------------------------

### Define and Train PyTorch Neural Networks for Regression

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_basic.rst

This snippet defines two multi-layer perceptron (MLP) models using PyTorch's `nn.Sequential` for regression tasks. It prepares synthetic data, converts it to PyTorch tensors, and uses `DataLoader` for efficient batch processing. The models are then trained using `nn.MSELoss` as the criterion and `optim.Adam` as the optimizer over 200 epochs, demonstrating a typical PyTorch training pipeline.

```python
                nn.ReLU(),
                nn.Linear(layers_sizes[2], 1),
            )
            reg_2 = nn.Sequential(
                nn.Linear(n_inputs, layers_sizes[0]),
                nn.ReLU(),
                nn.Linear(layers_sizes[0], layers_sizes[1]),
                nn.ReLU(),
                nn.Linear(layers_sizes[1], layers_sizes[2]),
                nn.ReLU(),
                nn.Linear(layers_sizes[2], 1),
            )

            # Convert data into PyTorch tensors
            X_tensor = torch.tensor(X, dtype=torch.float32)
            y_1_tensor = torch.tensor(y_1, dtype=torch.float32)
            y_2_tensor = torch.tensor(y_2, dtype=torch.float32)

            # Create a DataLoader for handling batches
            dataset_1 = TensorDataset(X_tensor, y_1_tensor)
            dataset_2 = TensorDataset(X_tensor, y_2_tensor)
            batch_size = 32
            dataloader_1 = DataLoader(dataset_1, batch_size=batch_size, shuffle=True)
            dataloader_2 = DataLoader(dataset_2, batch_size=batch_size, shuffle=True)

            # Initialise the loss function and optimizer
            criterion = nn.MSELoss()
            optimizer_1 = optim.Adam(reg_1.parameters(), lr=0.001, weight_decay=0.0001)
            optimizer_2 = optim.Adam(reg_2.parameters(), lr=0.001, weight_decay=0.0001)

            # Training loop
            for epoch in range(200):
                for batch_X, batch_y in dataloader_1:
                    # Forward pass
                    outputs = reg_1(batch_X)

                    # Calculate loss
                    loss = criterion(outputs, batch_y.view(-1, 1))  # Assuming y is a 1D array

                    # Backward pass and optimization
                    optimizer_1.zero_grad()
                    loss.backward()
                    optimizer_1.step()
                for batch_X, batch_y in dataloader_2:
                    # Forward pass
                    outputs = reg_2(batch_X)

                    # Calculate loss
                    loss = criterion(outputs, batch_y.view(-1, 1))  # Assuming y is a 1D array

                    # Backward pass and optimization
                    optimizer_2.zero_grad()
                    loss.backward()
                    optimizer_2.step()
```

--------------------------------

### API Documentation: MultiOutputConstr Class

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/MultiOutputConstr.rst

Documents the `MultiOutputConstr` class, which serves as a base or specific implementation for multi-output constraints within the `pyscipopt_ml` framework. The `:members:` directive indicates that all public methods and attributes of this class are relevant for documentation, providing a comprehensive view of its capabilities.

```APIDOC
Class: pyscipopt_ml.sklearn.multi_output.MultiOutputConstr
  Description: Base class or specific implementation for multi-output constraints.
  Members: All public methods and attributes are documented.
```

--------------------------------

### Dense Layer Softplus Activation Formulation

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Provides the mathematical formulation for a dense layer with a Softplus activation function.

```APIDOC
y_j = \log (1 + e^{\sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j}}) \quad \forall j
```

--------------------------------

### Generate Random Quadratic Functions for Training Data

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_basic.rst

This function generates synthetic training data for two non-linear quadratic functions. It takes a seed, number of inputs, and number of samples to create input features (X) and corresponding output values (y_1, y_2). The function ensures reproducibility by setting a random seed and uses NumPy for array operations.

```python
def build_random_quadratic_functions(seed=42, n_inputs=5, n_samples=1000):
    # Set the random seed so the results don't change
    np.random.seed(seed)

    # Generate two random quadratic functions f(x) = x^{t}Qx + Ax + c
    quadratic_coefficients = np.round(
        np.random.uniform(0, 5, size=(2, n_inputs, n_inputs)), decimals=3
    )
    linear_coefficients = np.round(np.random.uniform(0, 1, size=(2, n_inputs)), decimals=3)
    constant = np.round(np.random.uniform(0, 1, size=(2,)), decimals=3)

    # Generate data from the quadratic function
    X = np.random.uniform(-10, 10, size=(n_samples, n_inputs))
    y_1 = np.zeros((n_samples,))
    y_2 = np.zeros((n_samples,))
    for i in range(n_samples):
        y_1[i] = (
            X[i] @ quadratic_coefficients[0] @ X[i].T
            + linear_coefficients[0] @ X[i].T
            + constant[0]
        )
        y_2[i] = (
            X[i] @ quadratic_coefficients[1] @ X[i].T
            + linear_coefficients[1] @ X[i].T
            + constant[1]
        )

    return X, y_1, y_2
```

--------------------------------

### API Documentation for pyscipopt_ml.sklearn.support_vector Module

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/SupportVectorConstr.rst

Comprehensive API documentation for the `pyscipopt_ml.sklearn.support_vector` module, detailing its functions for adding support vector regressor and classifier constraints, and the `SupportVectorConstr` class.

```APIDOC
Module: pyscipopt_ml.sklearn.support_vector

Functions:
  pyscipopt_ml.sklearn.add_support_vector_regressor_constr(...)
  pyscipopt_ml.sklearn.add_support_vector_classifier_constr(...)

Classes:
  pyscipopt_ml.sklearn.support_vector.SupportVectorConstr:
    Members: All public members are documented.
```

--------------------------------

### API Reference: pyscipopt_ml.torch.sequential.SequentialConstr Class

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/SequentialConstr.rst

Documents the `SequentialConstr` class, which acts as a specialized constraint handler within the SCIP framework for PyTorch sequential neural networks. It manages the formulation and enforcement of constraints derived from the network's architecture and operations, as indicated by the `:members:` directive.

```APIDOC
class pyscipopt_ml.torch.sequential.SequentialConstr:
```

--------------------------------

### Document pyscipopt_ml.sklearn.pipeline Module

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/PipelineConstr.rst

Documents the `pyscipopt_ml.sklearn.pipeline` module, providing an overview of its contents and structure within the `pyscipopt_ml` library.

```APIDOC
.. automodule:: pyscipopt_ml.sklearn.pipeline
```

--------------------------------

### Dense Layer Softmax Activation Formulation

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Provides the mathematical formulation for a dense layer with a Softmax activation function.

```APIDOC
y_j = \frac{e^{\sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j}}}{\sum_{j'}e^{\sum_{i=1}^n \beta_{i,j'} x_i + \beta_{0,j'}}} \quad \forall j
```

--------------------------------

### Clone PySCIPOpt-ML latest sources for development

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Clones the PySCIPOpt-ML GitHub repository via SSH to obtain the most recent source code for development and contribution purposes.

```Shell
git clone git@github.com:Opt-Mucca/PySCIPOpt-ML.git
```

--------------------------------

### Dense Layer Sigmoid Activation Formulation

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Provides the mathematical formulation for a dense layer with a Sigmoid activation function.

```APIDOC
y_j = \frac{1}{1 + e^{-(\sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j})}} \quad \forall j
```

--------------------------------

### Argmax Formulation for Two-Dimensional Output with Indicator Constraints

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

This formulation uses indicator constraints for a two-dimensional output `y` to select the index corresponding to the larger pre-argmax value `y'`. It ensures one of the two outputs is 1, but is susceptible to tolerance issues if `y'_0` and `y'_1` are very close.

```APIDOC
y_0 = 1 \Rightarrow y'_0 \leq y'_1
y_1 = 1 \Rightarrow y'_1 \leq y'_0
y_0 + y_1 = 1
```

--------------------------------

### Cite PySCIPOpt-ML using BibTeX

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Provides the BibTeX entry for citing the PySCIPOpt-ML software in academic publications, referencing the associated arXiv paper.

```APIDOC
@misc{turner2024pyscipoptmlembeddingtrainedmachine,
      title={PySCIPOpt-ML: Embedding Trained Machine Learning Models into Mixed-Integer Programs},
      author={Mark Turner and Antonia Chmiela and Thorsten Koch and Michael Winkler},
      year={2024},
      eprint={2312.08074},
      archivePrefix={arXiv},
      primaryClass={math.OC},
      url={https://arxiv.org/abs/2312.08074},
}
```

--------------------------------

### Embed Trained PyTorch Models into SCIP Optimization Model

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_basic.rst

This code integrates the previously trained PyTorch neural networks (`reg_1`, `reg_2`) into a SCIP optimization model. It initializes a basic SCIP model using `build_basic_scip_model` and then uses `add_predictor_constr` to embed the neural networks as constraints. This allows SCIP to optimize problems where the neural network outputs are part of the objective or constraints.

```python
        # Now build the SCIP Model and embed the neural networks
        scip, input_vars, output_vars = build_basic_scip_model(n_inputs)
        mlp_cons_1 = add_predictor_constr(
            scip, reg_1, input_vars, output_vars[0], unique_naming_prefix="reg_1_"
        )
        mlp_cons_2 = add_predictor_constr(
            scip, reg_2, input_vars, output_vars[1], unique_naming_prefix="reg_2_"
        )

        return scip
```

--------------------------------

### Documenting `pyscipopt_ml.sklearn.mlp` Module

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/MLPConstr.rst

Provides an overview and documentation for the `pyscipopt_ml.sklearn.mlp` module, which integrates Multi-Layer Perceptron models from scikit-learn with SCIP optimization, allowing for the formulation of MLP-based constraints within optimization problems.

```APIDOC
Module:
  Name: pyscipopt_ml.sklearn.mlp
  Purpose: Provides functionalities to integrate Multi-Layer Perceptron (MLP) models from scikit-learn into SCIP optimization problems as constraints.
```

--------------------------------

### API Reference for ONNX ModelProto Constraint

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/ONNXConstr.rst

This snippet provides the API reference for the `pyscipopt_ml.onnx` module, detailing the `add_onnx_constr` function and the `ONNXConstr` class with its members.

```APIDOC
Function: pyscipopt_ml.onnx.add_onnx_constr
Class: pyscipopt_ml.onnx.neural_net_onnx.ONNXConstr
  Members: All
```

--------------------------------

### API Documentation for pyscipopt_ml.sklearn.sktransformer.SKtransformer

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/SKGetter.rst

Documents the `SKtransformer` class, a helper designed to perform data transformations following Scikit-Learn patterns within the `pyscipopt_ml` framework. It exposes all its public members for use, as indicated by the `autoclass` directive.

```APIDOC
Class: pyscipopt_ml.sklearn.sktransformer.SKtransformer
  Purpose: Provides functionality to transform data using Scikit-Learn patterns.
  Members: All public members are documented.
```

--------------------------------

### Argmax Formulation for Multi-Dimensional Output with SOS Constraints

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

This formulation handles three or more dimensions using SOS (Special Ordered Sets) constraints, along with auxiliary variables `m` (maximum value) and `s` (slack variables). It identifies the maximum value and sets the corresponding output `y_i` to 1, but can have tolerance issues for values close to the maximum.

```APIDOC
y'_i + s_i + m = 0 \quad \forall i \in \{0,...,n-1\}
SOS1(s_i, y_i) \quad \forall i \in \{0,...,n-1\}
\sum_{i \in \{0,...,n-1\}} y_i = 1
```

--------------------------------

### XGBoost Constraint Class Reference

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/XGBoostConstr.rst

This API documentation describes the `XGBoostConstr` class. It serves as a core component for handling and managing XGBoost-related constraints within the `pyscipopt_ml` framework. The `:members:` directive indicates that all public members of this class are documented.

```APIDOC
class pyscipopt_ml.xgboost.xgboost_constr.XGBoostConstr:
```

--------------------------------

### XGBoost Getter Class Reference

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/XGBoostConstr.rst

This API documentation describes the `XGBgetter` class. It provides functionalities for retrieving information or specific components related to XGBoost models within the `pyscipopt_ml` library. The `:members:` directive indicates that all public members of this class are documented.

```APIDOC
class pyscipopt_ml.xgboost.xgbgetter.XGBgetter:
```

--------------------------------

### Embed PyTorch Sequential Models in SCIP

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/supported.rst

This section details how to embed PyTorch `torch.nn.Sequential` models into a SCIP optimization model using `pyscipopt_ml.torch.add_sequential_constr`. It lists the specific layer types supported for integration and explains how to handle classification tasks by setting `output_type="classification"` to simplify the modeling of final activation functions like Softmax.

```APIDOC
Function: pyscipopt_ml.torch.add_sequential_constr
Supported Layers:
- torch.nn.Linear
- torch.nn.ReLU
- torch.nn.Sigmoid
- torch.nn.Tanh
- torch.nn.Softmax
- torch.nn.Softplus
Note: For classification, set output_type="classification" when inserting the constraint.
```

--------------------------------

### Dense Layer Tanh Activation Formulation

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Provides the mathematical formulation for a dense layer with a Tanh activation function.

```APIDOC
y_j = \frac{1 - e^{-2(\sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j})}}{1 + e^{-2(\sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j})}} \quad \forall j
```

--------------------------------

### API Documentation for pyscipopt_ml.sklearn.skgetter.SKgetter

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/SKGetter.rst

Documents the `SKgetter` class, a helper designed to retrieve information or objects related to Scikit-Learn within the `pyscipopt_ml` framework. It exposes all its public members for use, as indicated by the `autoclass` directive.

```APIDOC
Class: pyscipopt_ml.sklearn.skgetter.SKgetter
  Purpose: Provides functionality to retrieve Scikit-Learn related information.
  Members: All public members are documented.
```

--------------------------------

### Embed LightGBM Scikit-Learn Models in SCIP

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/supported.rst

This section details the embedding of LightGBM models from its Scikit-Learn interface into a SCIP model. It presents a table listing the supported LightGBM objects (e.g., `LGBMRegressor`, `LGBMClassifier`) and their respective `pyscipopt-ml` insertion functions. It also specifies that "gbdt" and "rf" boosters are currently supported for integration.

```APIDOC
Supported LightGBM Models and Insertion Functions:
| LightGBM object          | Function to insert                                      |
|--------------------------|---------------------------------------------------------|
| lightgbm.LGBMRegressor   | pyscipopt_ml.lightgbm.add_lgbregressor_constr           |
| lightgbm.LGBMClassifier  | pyscipopt_ml.lightgbm.add_lgbclassifier_constr          |
Supported Boosters: "gbdt", "rf"
```

--------------------------------

### pyscipopt-ml Linear Regression API Reference

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/LinearRegressionConstr.rst

This section outlines the key API components for integrating linear regression constraints into SCIP models using pyscipopt-ml. It references the main module, a utility function for adding constraints, and the core class for representing linear regression constraints.

```APIDOC
Module: pyscipopt_ml.sklearn.linear_regression

Function: pyscipopt_ml.sklearn.add_linear_regression_constr

Class: pyscipopt_ml.sklearn.linear_regression.LinearRegressionConstr
  (All public members are documented)
```

--------------------------------

### Scikit-learn Model Integration API for pyscipopt-ml

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/supported.rst

This section details the supported Scikit-learn machine learning models, their corresponding Python objects, and the `pyscipopt-ml` functions used to integrate them as constraints within a SCIP optimization model. It covers various regression, classification, and clustering models.

```APIDOC
Machine Learning Model: Ordinary Least Square
  Scikit-learn objects:
    - sklearn.linear_model.LinearRegression
    - sklearn.linear_model.Ridge
    - sklearn.linear_model.ElasticNet
    - sklearn.linear_model.Lasso
  Functions to insert:
    - pyscipopt_ml.sklearn.linear_regression.add_linear_regression_constr

Machine Learning Model: Partial Least Square
  Scikit-learn objects:
    - sklearn.cross_decomposition.PLSRegression
    - sklearn.cross_decomposition.PLSCanonical
  Functions to insert:
    - pyscipopt_ml.sklearn.pls.add_pls_regression_constr

Machine Learning Model: Logistic Regression
  Scikit-learn objects:
    - sklearn.linear_model.LogisticRegression
  Functions to insert:
    - pyscipopt_ml.sklearn.logistic_regression.add_logistic_regression_constr

Machine Learning Model: Neural-network
  Scikit-learn objects:
    - sklearn.neural_network.MLPRegressor
    - sklearn.neural_network.MLPClassifier
  Functions to insert:
    - pyscipopt_ml.sklearn.mlp.add_mlp_regressor_constr
    - pyscipopt_ml.sklearn.mlp.add_mlp_classifier_constr

Machine Learning Model: Decision tree
  Scikit-learn objects:
    - sklearn.tree.DecisionTreeRegressor
    - sklearn.tree.DecisionTreeClassifier
  Functions to insert:
    - pyscipopt_ml.sklearn.decision_tree.add_decision_tree_regressor_constr
    - pyscipopt_ml.sklearn.decision_tree.add_decision_tree_classifier_constr

Machine Learning Model: Gradient boosting
  Scikit-learn objects:
    - sklearn.ensemble.GradientBoostingRegressor
    - sklearn.ensemble.GradientBoostingClassifier
  Functions to insert:
    - pyscipopt_ml.sklearn.gradient_boosting.add_gradient_boosting_regressor_constr
    - pyscipopt_ml.sklearn.gradient_boosting.add_gradient_boosting_classifier_constr

Machine Learning Model: Random Forest
  Scikit-learn objects:
    - sklearn.ensemble.RandomForestRegressor
    - sklearn.ensemble.RandomForestClassifier
  Functions to insert:
    - pyscipopt_ml.sklearn.random_forest.add_random_forest_regressor_constr
    - pyscipopt_ml.sklearn.random_forest.add_random_forest_classifier_constr

Machine Learning Model: Support Vector Machines
  Scikit-learn objects:
    - sklearn.svm.SVR
    - sklearn.svm.SVC
    - sklearn.svm.LinearSVR
    - sklearn.svm.LinearSVC
  Functions to insert:
    - pyscipopt_ml.sklearn.support_vector.add_support_vector_regressor_constr
    - pyscipopt_ml.sklearn.support_vector.add_support_vector_classifier_constr

Machine Learning Model: Centroid Clustering
  Scikit-learn objects:
    - sklearn.cluster.KMeans
    - sklearn.cluster.MiniBatchKMeans
  Functions to insert:
    - pyscipopt_ml.sklearn.centroid_cluster.add_centroid_cluster_constr

Machine Learning Model: Pipeline
  Scikit-learn objects:
    - sklearn.pipeline.Pipeline
  Functions to insert:
    - pyscipopt_ml.sklearn.pipeline.add_pipeline_constr

Machine Learning Model: MultiOutput
  Scikit-learn objects:
    - sklearn.multioutput.MultiOutputClassifier
    - sklearn.multioutput.MultiOutputRegressor
  Functions to insert:
    - pyscipopt_ml.sklearn.multi_output.add_multi_output_classifier_constr
    - pyscipopt_ml.sklearn.multi_output.add_multi_output_regressor_constr
```

--------------------------------

### Linear Regression Model Formulation

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

This formulation defines the linear regression model, where `y` is the output, `x` are the input features, and `beta` are the learned weights. Since it's linear, it can be directly represented using linear constraints in SCIP, supporting techniques like Ridge, Lasso, and ElasticNet.

```APIDOC
y = \sum_{i=1}^n \beta_i x_i + \beta_0.
```

--------------------------------

### Argmax Formulation for Single-Dimensional Output

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

This formulation interprets argmax for a single-dimensional output by checking if the value is greater than 0.5. It uses linear inequalities to constrain the output `y` based on the pre-argmax value `y'`, but has tolerance issues around 0.5.

```APIDOC
y \leq y' + 0.5
y \geq y' - 0.5
```

--------------------------------

### LightGBM Constraint Class Definition (Python APIDOC)

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/LightGBMConstr.rst

Documents the `LightGBMConstr` class, which is central to handling LightGBM model constraints within PySCIPOpt-ML. This class likely encapsulates the methods and properties required to translate LightGBM model structures into SCIP constraints. The `:members:` directive indicates that all public members of this class are documented.

```APIDOC
class pyscipopt_ml.lightgbm.lightgbm_constr.LightGBMConstr:
  # All public members are documented.
```

--------------------------------

### Dense Layer Classification Output Formulation (Argmax)

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Describes the formulation used for classification purposes when the final activation layer is not explicitly modeled, typically for maximum preservation.

```APIDOC
y'_j =& argmax(\sum_{i=1}^n \beta_{i,j} x_i + \beta_{0,j}) \quad \forall j

y =& argmax(y')
```

--------------------------------

### Embed XGBoost Scikit-Learn Models in SCIP

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/supported.rst

This section outlines the process of embedding XGBoost models, specifically those from its Scikit-Learn interface, into a SCIP model. It provides a comprehensive table detailing the supported XGBoost objects (e.g., `XGBRegressor`, `XGBClassifier`) and their corresponding `pyscipopt-ml` insertion functions. It also specifies that only "gbtree" boosters are currently supported for integration.

```APIDOC
Supported XGBoost Models and Insertion Functions:
| XGBoost object           | Function to insert                                      |
|--------------------------|---------------------------------------------------------|
| xgboost.XGBRegressor     | pyscipopt_ml.xgboost.add_xgbregressor_constr            |
| xgboost.XGBClassifier    | pyscipopt_ml.xgboost.add_xgbclassifier_constr           |
| xgboost.XGBRFRegressor   | pyscipopt_ml.xgboost.add_xgbregressor_rf_constr         |
| xgboost.XGBRFClassifier  | pyscipopt_ml.xgboost.add_xgbclassifier_rf_constr        |
Supported Boosters: "gbtree"
```

--------------------------------

### Clone PySCIPOpt-ML repository via SSH

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/README.md

Clones the PySCIPOpt-ML GitHub repository using SSH. This method is often preferred by developers for authenticated access and easier contributions.

```Shell
git clone git@github.com:Opt-Mucca/PySCIPOpt-ML.git
```

--------------------------------

### Optimize and Validate SCIP Model with Embedded Predictors

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/example_basic.rst

This snippet demonstrates how to execute the function that builds the SCIP model with embedded neural networks, optimize it using `scip.optimize()`, and then validate the accuracy of the neural network embedding. It checks the maximum error of the embedded predictors (`mlp_cons_1`, `mlp_cons_2`) against a predefined threshold (10^-3), raising an `AssertionError` if the error is too high, ensuring the quality of the MIP embedding.

```python
    # Get the SCIP Model with the embedded trained predictors
    scip = build_and_optimise_function_approximation_model()

    # Optimize the model
    scip.optimize()

    # We can check the "error" of the MIP embedding via the difference between SKLearn / Torch and SCIP output
    if np.max(mlp_cons_1.get_error()) > 10**-3:
        error = np.max(mlp_cons_1.get_error())
        raise AssertionError(f"Max error {error} exceeds threshold of {10 ** -3}")
    if np.max(mlp_cons_2.get_error()) > 10**-3:
        error = np.max(mlp_cons_2.get_error())
        raise AssertionError(f"Max error {error} exceeds threshold of {10 ** -3}")
```

--------------------------------

### API Documentation for Random Forest Constraints

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/api/RandomForestConstr.rst

This section provides an overview of the API for handling Random Forest constraints within the `pyscipopt_ml` library, specifically focusing on the `pyscipopt_ml.sklearn.random_forest` module. It details functions for adding regressor and classifier constraints and the `RandomForestConstr` class.

```APIDOC
Module: pyscipopt_ml.sklearn.random_forest

Functions:
  - add_random_forest_regressor_constr(...)
    Purpose: Adds constraints for a Random Forest Regressor model to a SCIP model.
  - add_random_forest_classifier_constr(...)
    Purpose: Adds constraints for a Random Forest Classifier model to a SCIP model.

Class: RandomForestConstr
  Module: pyscipopt_ml.sklearn.random_forest
  Purpose: Represents a Random Forest constraint within the SCIP model.
  Members: All public members are documented.
```

--------------------------------

### Decision Tree Leaf Selection Constraint

Source: https://github.com/opt-mucca/pyscipopt-ml/blob/main/docs/formulations.rst

Explains the constraint that exactly one leaf must be chosen in a decision tree, formulated as a sum of binary decision variables.

```APIDOC
\sum_{l} \delta_l = 1
```