Lightwood

https://mindsdb.github.io/lightwood
Admin
Lightwood is an AutoML framework that generates and customizes machine learning pipelines using...
Tokens:254,306
Snippets:2,010
Trust Score:9.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Lightwood using pip

Source: https://mindsdb.github.io/lightwood/_sources/index.rst.txt

Install the Lightwood package using pip. Depending on your environment, you might need to use pip3.

```bash
pip3 install lightwood
```

--------------------------------

### Install Lightwood using pip

Source: https://mindsdb.github.io/lightwood/index.html

Install the Lightwood library using pip. Depending on your environment, you may need to use pip3. It is recommended to use a Python virtual environment.

```bash
pip3 install lightwood

```

--------------------------------

### Import Libraries and Get Lightwood Version

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_explainer/custom_explainer.ipynb.txt

Imports necessary libraries and retrieves the installed Lightwood version. This is a common setup step for Lightwood projects.

```python
from typing import Dict, Tuple
import pandas as pd
import lightwood
lightwood.__version__
```

--------------------------------

### Make Train/Test Splits and Predict Examples

Source: https://mindsdb.github.io/lightwood/index.html

This Python code snippet demonstrates how to preprocess a DataFrame, create train/test splits using a predictor, and then generate predictions for the first 10 test examples.

```python
test_df = predictor.split(predictor.preprocess(df))["test"]
preds = predictor.predict(test_df).iloc[:10]
print(preds)
```

--------------------------------

### Instantiate ModelCorrelationHeatmap

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_explainer/custom_explainer.ipynb.txt

Instantiates the ModelCorrelationHeatmap analysis block. This is a basic setup for a new analysis block.

```python
ModelCorrelationHeatmap()
```

--------------------------------

### Import Lightwood ProblemDefinition

Source: https://mindsdb.github.io/lightwood/tutorials/tutorial_time_series/tutorial_time_series.html

Imports the necessary class for defining machine learning problems with Lightwood. No additional setup is required if Lightwood is installed.

```python
from lightwood.api.high_level import ProblemDefinition
```

--------------------------------

### Example JSON AI Syntax Structure

Source: https://mindsdb.github.io/lightwood/tutorials/custom_cleaner/custom_cleaner.html

This is an example of the JSON AI syntax generated by Lightwood. It defines encoders, data types, model configurations, and problem-specific parameters.

```json
{
    "encoders": {
        "target": {
            "module": "NumericEncoder",
            "args": {
                "is_target": "True",
                "positive_domain": "$statistical_analysis.positive_domain"
            }
        },
        "excerpt": {
            "module": "PretrainedLangEncoder",
            "args": {
                "output_type": "$dtype_dict[$target]",
                "stop_after": "$problem_definition.seconds_per_encoder"
            }
        }
    },
    "dtype_dict": {
        "excerpt": "rich_text",
        "target": "float"
    },
    "dependency_dict": {},
    "model": {
        "module": "BestOf",
        "args": {
            "submodels": [
                {
                    "module": "Neural",
                    "args": {
                        "fit_on_dev": true,
                        "stop_after": "$problem_definition.seconds_per_mixer",
                        "search_hyperparameters": true
                    }
                },
                {
                    "module": "XGBoostMixer",
                    "args": {
                        "stop_after": "$problem_definition.seconds_per_mixer",
                        "fit_on_dev": true
                    }
                },
                {
                    "module": "Regression",
                    "args": {
                        "stop_after": "$problem_definition.seconds_per_mixer"
                    }
                },
                {
                    "module": "RandomForest",
                    "args": {
                        "stop_after": "$problem_definition.seconds_per_mixer",
                        "fit_on_dev": true
                    }
                }
            ]
        }
    },
    "problem_definition": {
        "target": "target",
        "pct_invalid": 2,
        "unbias_target": true,
        "seconds_per_mixer": 21384.0,
        "seconds_per_encoder": 85536.0,
        "expected_additional_time": 15.390582799911499,
        "time_aim": 259200,
        "target_weights": null,
        "positive_domain": false,
        "timeseries_settings": {
            "is_timeseries": false,
            "order_by": null,
            "window": null,
            "group_by": null,
            "use_previous_target": true,
            "horizon": null,
            "historical_columns": null,
            "target_type": "",
            "allow_incomplete_history": true,
            "eval_incomplete": false,
            "interval_periods": []
        },
        "anomaly_detection": false,
        "use_default_analysis": true,
        "embedding_only": false,
        "dtype_dict": {},
        "ignore_features": [
            "url_legal",
            "license",
            "standard_error"
        ],
        "fit_on_all": true,
        "strict_mode": true,
        "seed_nr": 1
    },
    "identifiers": {
        "id": "Hash-like identifier"
    },
    "imputers": [],
    "accuracy_functions": [
        "r2_score"
    ]
}
```

--------------------------------

### Setup Problem Definition and Generate JSON-AI

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_splitter/custom_splitter.ipynb.txt

This snippet sets up the problem definition for Lightwood, specifying the target column, and then generates the JSON-AI syntax using the provided data and definition. Ensure 'data' is defined and 'json_ai_from_problem' function is available.

```python
# Setup the problem definition
problem_definition = {
    'target': 'Class',
}

# Generate the j{ai}son syntax
json_ai = json_ai_from_problem(data, problem_definition)

```

--------------------------------

### Import Libraries and Check Lightwood Version

Source: https://mindsdb.github.io/lightwood/tutorials/custom_explainer/custom_explainer.html

Imports necessary libraries and displays the installed Lightwood version. Ensure you have the correct version for compatibility.

```python
from typing import Dict, Tuple
import pandas as pd
import lightwood
lightwood.__version__
```

```text
'25.3.3.3'
```

--------------------------------

### Setup JSON-AI Problem Definition

Source: https://mindsdb.github.io/lightwood/tutorials/custom_splitter/custom_splitter.html

Initializes a dictionary to define the target column for a machine learning problem in JSON-AI format. The 'target' key specifies the column to be predicted.

```python
# Setup the problem definition
problem_definition = {
    'target': 'Class',
}
```

--------------------------------

### Log Mixer Training and Fit Model

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_cleaner/custom_cleaner.ipynb.txt

Logs the start of the mixer training phase and fits the model if embedding is not the only option. Otherwise, initializes an Embedder.

```python
log.info(f"[Learn phase 6/{n_phases}] - Mixer training")
        if not self.problem_definition.embedding_only:
            self.fit(enc_train_test)
        else:
            self.mixers = []
            self.ensemble = Embedder(
                self.target, mixers=list(), data=enc_train_test["train"]
            )
            self.supports_proba = self.ensemble.supports_proba
```

--------------------------------

### Lightwood Type Inference Logs

Source: https://mindsdb.github.io/lightwood/tutorials/custom_mixer/custom_mixer.html

Example log output from Lightwood's type inference process, showing detected data types for various columns.

```log
INFO:lightwood-3231:No torchvision detected, image helpers not supported.
INFO:lightwood-3231:No torchvision/pillow detected, image encoder not supported
INFO:type_infer-3231:Analyzing a sample of 298
INFO:type_infer-3231:from a total population of 303, this is equivalent to 98.3% of your data.
INFO:type_infer-3231:Infering type for: age
INFO:type_infer-3231:Column age has data type integer
INFO:type_infer-3231:Infering type for: sex
INFO:type_infer-3231:Column sex has data type binary
INFO:type_infer-3231:Infering type for: cp
INFO:type_infer-3231:Column cp has data type categorical
INFO:type_infer-3231:Infering type for: trestbps
INFO:type_infer-3231:Column trestbps has data type integer
INFO:type_infer-3231:Infering type for: chol
INFO:type_infer-3231:Column chol has data type integer
INFO:type_infer-3231:Infering type for: fbs
INFO:type_infer-3231:Column fbs has data type binary
INFO:type_infer-3231:Infering type for: restecg
INFO:type_infer-3231:Column restecg has data type categorical
INFO:type_infer-3231:Infering type for: thalach
INFO:type_infer-3231:Column thalach has data type integer
INFO:type_infer-3231:Infering type for: exang
INFO:type_infer-3231:Column exang has data type binary
INFO:type_infer-3231:Infering type for: oldpeak
INFO:type_infer-3231:Column oldpeak has data type float
INFO:type_infer-3231:Infering type for: slope
INFO:type_infer-3231:Column slope has data type categorical
INFO:type_infer-3231:Infering type for: ca
INFO:type_infer-3231:Column ca has data type categorical
INFO:type_infer-3231:Infering type for: thal
INFO:type_infer-3231:Column thal has data type categorical
INFO:type_infer-3231:Infering type for: target
INFO:type_infer-3231:Column target has data type binary
INFO:dataprep_ml-3231:Starting statistical analysis
INFO:dataprep_ml-3231:Finished statistical analysis
```

--------------------------------

### Instantiate Post-Training Evaluation Blocks

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_splitter/custom_splitter.ipynb.txt

Initializes a list of analysis blocks for post-training evaluation. These blocks include statistical analysis and feature importance calculations.

```python
self.analysis_blocks = [
            ICP(fixed_significance=None, confidence_normalizer=False, deps=[]),
            ConfStats(deps=["ICP"]),
            AccStats(deps=["ICP"]),
            PermutationFeatureImportance(deps=["AccStats"]),
        ]
```

--------------------------------

### XGBoostMixer Initialization

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/mixer/xgboost.html

Initializes the XGBoostMixer with specified parameters, including time budget, target column, data types, and configuration for optuna and GPU usage.

```python
class XGBoostMixer(BaseMixer):
    model: Union[xgb.XGBClassifier, xgb.XGBRegressor]
    ordinal_encoder: OrdinalEncoder
    label_set: List[str]
    max_bin: int
    device: torch.device
    device_str: str
    num_iterations: int
    use_optuna: bool
    supports_proba: bool

    """
    Gradient boosting mixer with an XGBoost backbone.

    This mixer is a good all-rounder, due to the generally great performance of tree-based ML algorithms for supervised learning tasks with tabular data.
    If you want more information regarding the techniques that set apart XGBoost from other gradient boosters, please refer to their technical paper: "XGBoost: A Scalable Tree Boosting System" (2016).

    We can basically think of this mixer as a wrapper to the XGBoost Python package. To do so, there are a few caveats the user may want to be aware about:
        * If you seek GPU utilization, XGBoost must be compiled from source instead of being installed through `pip`.
        * Integer, float, and quantity `dtype`s are treated as regression tasks with `reg:squarederror` loss. All other supported `dtype`s is casted as a multiclass task with `multi:softmax` loss.
        * A partial fit can be performed with the `dev` data split as part of `fit`, if specified with the `fit_on_dev` argument.

    There are a couple things in the backlog that will hopefully be added soon:
        * An automatic optuna-based hyperparameter search. This procedure triggers when a single iteration of XGBoost is deemed fast enough (given the time budget).
        * Support for "unknown class" as a possible answer for multiclass tasks.
    """  # noqa

    def __init__(
            self, stop_after: float, target: str, dtype_dict: Dict[str, str],
            input_cols: List[str],
            fit_on_dev: bool, use_optuna: bool, target_encoder: BaseEncoder):
        """
        :param stop_after: time budget in seconds.
        :param target: name of the target column that the mixer will learn to predict.
        :param dtype_dict: dictionary with dtypes of all columns in the data.
        :param input_cols: list of column names.
        :param fit_on_dev: whether to perform a `partial_fit()` at the end of `fit()` using the `dev` data split.
        :param use_optuna: whether to activate the automated hyperparameter search (optuna-based). Note that setting this flag to `True` does not guarantee the search will run, rather, the speed criteria will be checked first (i.e., if a single iteration is too slow with respect to the time budget, the search will not take place). 
        """  # noqa
        super().__init__(stop_after)
        self.model = None
        self.ordinal_encoder = None
        self.positive_domain = False
        self.label_set = []
        self.target = target
        self.dtype_dict = dtype_dict
        self.input_cols = input_cols
        self.use_optuna = use_optuna
        self.params = {}
        self.fit_on_dev = fit_on_dev
        self.cls_dtypes = [dtype.categorical, dtype.binary, dtype.cat_tsarray]
        self.float_dtypes = [dtype.float, dtype.quantity, dtype.num_tsarray]
        self.num_dtypes = [dtype.integer] + self.float_dtypes
        self.supports_proba = dtype_dict[target] in self.cls_dtypes
        self.stable = True
        self.target_encoder = target_encoder

        gpu_works = check_gpu_support()
        if not gpu_works:
            self.device = torch.device('cpu')
            self.device_str = 'cpu'
            log.warning('XGBoost running on CPU')
        else:
            self.device = torch.device('cuda')
            self.device_str = 'gpu'

        self.max_bin = 255

    def _to_dataset(self, ds: EncodedDs, output_dtype: str, mode='train'):
        """
        Helper method to wrangle a datasource into the format that the underlying model requires.
        """
```

--------------------------------

### Example JSON-AI Structure

Source: https://mindsdb.github.io/lightwood/tutorials/custom_encoder_rulebased/custom_encoder_rulebased.html

This is an example of the structure of a JSON-AI object, detailing encoders, data types, and model configurations. It is generated by the `json_ai_from_problem` function.

```json
{
    "encoders": {
        "price": {
            "module": "NumericEncoder",
            "args": {
                "is_target": "True",
                "positive_domain": "$statistical_analysis.positive_domain"
            }
        },
        "model": {
            "module": "CategoricalAutoEncoder",
            "args": {
                "stop_after": "$problem_definition.seconds_per_encoder"
            }
        },
        "year": {
            "module": "NumericEncoder",
            "args": {}
        },
        "transmission": {
            "module": "OneHotEncoder",
            "args": {}
        },
        "mileage": {
            "module": "NumericEncoder",
            "args": {}
        },
        "fuelType": {
            "module": "OneHotEncoder",
            "args": {}
        },
        "tax": {
            "module": "NumericEncoder",
            "args": {}
        },
        "mpg": {
            "module": "NumericEncoder",
            "args": {}
        },
        "engineSize": {
            "module": "NumericEncoder",
            "args": {}
        }
    },
    "dtype_dict": {
        "model": "categorical",
        "year": "integer",
        "price": "integer",
        "transmission": "categorical",
        "mileage": "integer",
        "fuelType": "categorical",
        "tax": "integer",
        "mpg": "float",
        "engineSize": "float"
    },
    "dependency_dict": {},
    "model": {
        "module": "BestOf",
        "args": {
            "submodels": [
                {
                    "module": "Neural",
                    "args": {
                        "fit_on_dev": true,
                        "stop_after": "$problem_definition.seconds_per_mixer",
                        "search_hyperparameters": true
                    }
                },
                {
                    "module": "XGBoostMixer",
                    "args": {
                        "stop_after": "$problem_definition.seconds_per_mixer",
                        "fit_on_dev": true
                    }
                },
                {
                    "module": "Regression",
                    "args": {
                        "stop_after": "$problem_definition.seconds_per_mixer"
                    }
                },
                {
                    "module": "RandomForest",
                    "args": {
                        "stop_after": "$problem_definition.seconds_per_mixer",
                        "fit_on_dev": true
                    }
                }
            ]
        }
    },
    "problem_definition": {
        "target": "price",
        "pct_invalid": 2,
        "unbias_target": true,
        "seconds_per_mixer": 21384.0,
        "seconds_per_encoder": 85536.0,
        "expected_additional_time": 11.070627927780151,
        "time_aim": 259200,
        "target_weights": null,
        "positive_domain": false,
        "timeseries_settings": {
            "is_timeseries": false,
            "order_by": null,
            "window": null,
            "group_by": null,
            "use_previous_target": true,
            "horizon": null,
            "historical_columns": null,
            "target_type": "",
            "allow_incomplete_history": true,
            "eval_incomplete": false,
            "interval_periods": []
        },
        "anomaly_detection": false,
        "use_default_analysis": true,
        "embedding_only": false,
        "dtype_dict": {},
        "ignore_features": [],
        "fit_on_all": true
    }
}
```

--------------------------------

### Prepare Img2VecEncoder

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/encoder/image/img_2_vec.html

Sets up the Img2Vec model and prepares the encoder for use. This method must be called once before encoding or decoding.

```python
def prepare(self, train_priming_data: Iterable[str], dev_priming_data: Iterable[str]):
    # @TODO: finetune here? depending on time aim
    """
    Sets an `Img2Vec` object (model) and sets the expected size for encoded representations.
    """
    if self.is_prepared:
        raise Exception('You can only call "prepare" once for a given encoder.')

    self.model = Img2Vec(device=self.device_name)
    self.device = self.model.device
    self.device_name = f'{self.device}'
    self.output_size = self.model.output_size
    self.is_prepared = True
```

--------------------------------

### Visualize Time Series Forecasts

Source: https://mindsdb.github.io/lightwood/tutorials/tutorial_time_series/tutorial_time_series.html

Generate a plot to visualize the forecasted values along with their lower and upper confidence bounds. Ensure matplotlib is installed (`pip install matplotlib`).

```python
plt.figure(figsize=(12, 8))
plt.plot([None for _ in range(forecasts.shape[0])] + forecasts.iloc[-1]['prediction'], color='purple', label='point prediction')
plt.plot([None for _ in range(forecasts.shape[0])] + forecasts.iloc[-1]['lower'], color='grey')
plt.plot([None for _ in range(forecasts.shape[0])] + forecasts.iloc[-1]['upper'], color='grey')
plt.xlabel('timestep')
plt.ylabel('# sunspots')
plt.title("Forecasted amount of sunspots for the next semester")
plt.legend()
plt.show()
```

--------------------------------

### Initialize Statistical Analysis

Source: https://mindsdb.github.io/lightwood/tutorials/tutorial_data_analysis/tutorial_data_analysis.html

Instantiate the StatisticalAnalysis class with dataset information, including data types, problem definition, and identifiers. This prepares Lightwood to perform statistical analysis.

```python
from dataprep_ml.insights import StatisticalAnalysis
from dataprep_ml.insights import statistical_analysis

stan = statistical_analysis(df,
                            type_information.dtypes,
                            problem_definition.to_dict(),
                            type_information.identifiers)
```

--------------------------------

### Initialize ETSMixer

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/mixer/ets.html

Initializes the ETSMixer with various parameters for time series forecasting. Use this to configure the AutoETS model with specific settings for error handling, trend, seasonality, and more.

```python
from typing import Dict, Optional

from lightwood.mixer.sktime import SkTime



[docs]class ETSMixer(SkTime):
    def __init__(
            self,
            stop_after: float,
            target: str,
            dtype_dict: Dict[str, str],
            horizon: int,
            ts_analysis: Dict,
            model_path: str = 'ets.AutoETS',
            auto_size: bool = True,
            sp: int = None,
            use_stl: bool = True,
            error: str = 'add',
            trend: Optional[str] = None,
            damped_trend: bool = False,
            seasonal: Optional[str] = None,
            initialization_method: str = 'estimated',
            initial_level: Optional[float] = None,
            initial_trend: Optional[float] = None,
            initial_seasonal: Optional[list] = None,
            bounds: Optional[dict] = None,
            start_params: Optional[list] = None,
            maxiter: int = 1000,
            auto: bool = False,
            information_criterion: str = 'aic',
            allow_multiplicative_trend: bool = False,
            restrict: bool = True,
            additive_only: bool = False,
            ignore_inf_ic: bool = True,
            n_jobs: Optional[int] = None,
            random_state: Optional[int] = None
    ):
        """
        Wrapper for SkTime's AutoETS interface.

        :param stop_after: time budget in seconds
        :param target: column containing target time series
        :param dtype_dict: data types for each dataset column
        :param horizon: forecast length
        :param ts_analysis: lightwood-produced stats about input time series
        :param auto_size: whether to filter out old data points if training split is bigger than a certain threshold (defined by the dataset sampling frequency). Enabled by default to avoid long training times in big datasets.
        :param use_stl: Whether to use de-trenders and de-seasonalizers fitted in the timeseries analysis phase.

        For the rest of the parameters, please refer to SkTime's documentation.
        """  # noqa

        hyperparam_search = False
        model_kwargs = {
            'error': error,
            'trend': trend,
            'damped_trend': damped_trend,
            'seasonal': seasonal,
            'initialization_method': initialization_method,
            'initial_level': initial_level,
            'initial_trend': initial_trend,
            'initial_seasonal': initial_seasonal,
            'bounds': bounds,
            'start_params': start_params,
            'maxiter': maxiter,
            'auto': auto,
            'information_criterion': information_criterion,
            'allow_multiplicative_trend': allow_multiplicative_trend,
            'restrict': restrict,
            'additive_only': additive_only,
            'ignore_inf_ic': ignore_inf_ic,
            'n_jobs': n_jobs,
            'random_state': random_state
        }
        super().__init__(stop_after, target, dtype_dict, horizon, ts_analysis,
                         model_path, model_kwargs, auto_size, sp, hyperparam_search, use_stl)
        self.name = 'AutoETS'
        self.stable = False

```

--------------------------------

### get_devices

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/helpers/device.html

Gets the appropriate Torch device(s) based on CUDA availability and compatibility.

```APIDOC
## get_devices

### Description
Get the appropriate Torch device(s) based on CUDA availability and compatibility.

This function determines the appropriate Torch device(s) to be used for computations based on the availability of CUDA and compatible devices. It checks if CUDA is available and if the available CUDA devices are compatible according to the 'is_cuda_compatible()' function. If compatible devices are found, the function selects either the first available CUDA device or a randomly selected one based on the 'RANDOM_GPU' environment variable. If CUDA is not available or no compatible devices are found, the function returns the CPU device.

### Method
N/A (This is a Python function, not an API endpoint)

### Endpoint
N/A

### Parameters
None

### Request Example
```python
import torch
import os
from random import randint

def is_cuda_compatible():
    compatible_device_count = 0
    if torch.version.cuda is not None:
        for d in range(torch.cuda.device_count()):
            capability = torch.cuda.get_device_capability(d)
            major = capability[0]
            minor = capability[1]
            current_arch = major * 10 + minor
            min_arch = min((int(arch.split("_")[1]) for arch in torch.cuda.get_arch_list()), default=35)
            if (not current_arch < min_arch
                    and not torch._C._cuda_getCompiledVersion() <= 9000):
                compatible_device_count += 1

    if compatible_device_count > 0:
        return True
    return False

def get_devices():
    if torch.cuda.is_available() and is_cuda_compatible():
        device_str = "cuda"
        available_devices = torch.cuda.device_count()

        if available_devices > 1:
            if os.environ.get('RANDOM_GPU', False) in ['1', 'true', 'True', True, 1]:
                device_str = 'cuda:' + str(randint(0, available_devices - 1))
                available_devices = 1
    else:
        device_str = "cpu"
        available_devices = 0

    return torch.device(device_str), available_devices


device, num_devices = get_devices()
print(device)
print(num_devices)
```

### Response
#### Success Response
- **Tuple**: A tuple containing the selected Torch device and the number of available devices.

#### Response Example
```
cuda:0
1
```
```

--------------------------------

### Prepare SimpleLabelEncoder

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/encoder/categorical/simple_label.html

Prepares the encoder by mapping unique labels from priming data to integers. Labels must be exact matches. The `unknown` category is mapped to index 0.

```python
def prepare(self, priming_data: Union[list, pd.Series]) -> None:
        if not isinstance(priming_data, pd.Series):
            priming_data = pd.Series(priming_data)

        for i, v in enumerate(priming_data.unique()):
            if v is not None:
                self.label_map[str(v)] = int(i + 1)  # leave 0 for UNK
        self.n_labels = len(self.label_map)
        for k, v in self.label_map.items():
            self.inv_label_map[v] = k
        self.is_prepared = True
```

--------------------------------

### get_device_from_name

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/helpers/device.html

Gets a Torch device based on the specified device name or default behavior.

```APIDOC
## get_device_from_name

### Description
Get a Torch device based on the specified device name or default behavior.

This function returns a Torch device based on the specified device name or the default behavior, which is to return the output of the 'get_devices()' function.

### Method
N/A (This is a Python function, not an API endpoint)

### Endpoint
N/A

### Parameters
#### Query Parameters
- **device_name** (str) - Optional - Name of the device to use. Default is an empty string.

### Request Example
```python
import torch
import os
from random import randint

def is_cuda_compatible():
    compatible_device_count = 0
    if torch.version.cuda is not None:
        for d in range(torch.cuda.device_count()):
            capability = torch.cuda.get_device_capability(d)
            major = capability[0]
            minor = capability[1]
            current_arch = major * 10 + minor
            min_arch = min((int(arch.split("_")[1]) for arch in torch.cuda.get_arch_list()), default=35)
            if (not current_arch < min_arch
                    and not torch._C._cuda_getCompiledVersion() <= 9000):
                compatible_device_count += 1

    if compatible_device_count > 0:
        return True
    return False

def get_devices():
    if torch.cuda.is_available() and is_cuda_compatible():
        device_str = "cuda"
        available_devices = torch.cuda.device_count()

        if available_devices > 1:
            if os.environ.get('RANDOM_GPU', False) in ['1', 'true', 'True', True, 1]:
                device_str = 'cuda:' + str(randint(0, available_devices - 1))
                available_devices = 1
    else:
        device_str = "cpu"
        available_devices = 0

    return torch.device(device_str), available_devices

def get_device_from_name(device_name=''):
    if(device_name != ''):
        device = torch.device(device_name)
    else:
        device, _ = get_devices()
    return device


device = get_device_from_name('cuda:1')
print(device)
```

### Response
#### Success Response
- **torch.device**: The selected Torch device.

#### Response Example
```
cuda:1
```
```

--------------------------------

### Preprocess, Split, Prepare, and Featurize Data

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_encoder_rulebased/custom_encoder_rulebased.ipynb.txt

This sequence of commands preprocesses raw data, splits it into training and testing sets, prepares encoders, and then featurizes the data for machine learning models.

```python
cleaned_data = predictor.preprocess(data=df)
```

```python
split_data = predictor.split(cleaned_data)
```

```python
predictor.prepare(split_data)
```

```python
ft_data = predictor.featurize(split_data)
```

--------------------------------

### Get Length - ConcatedEncodedDs

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/data/encoded_ds.html

Returns the length of the ConcatedEncodedDs object. The behavior is noted as potentially unintuitive.

```python
def __len__(self):
        """
        See `lightwood.data.encoded_ds.EncodedDs.__len__()`.
        """
        # @TODO: behavior here is not intuitive
        return max(0, np.sum(self.encoded_ds_lengths) - 2)
```

--------------------------------

### Initialize WeightedMeanEnsemble

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/ensemble/weighted_mean_ensemble.html

Initializes the WeightedMeanEnsemble. It checks if the target is suitable for regression and calculates initial weights based on mixer accuracies if 'fit' is True. Raises an exception for non-regression tasks.

```python
class WeightedMeanEnsemble(BaseEnsemble):
    """
    This ensemble determines a weight vector to return a weighted mean of the underlying mixers.

    More specifically, each model is evaluated on the validation dataset and assigned an accuracy score (as per the fixed accuracy function at the JsonAI level).

    Afterwards, all the scores are softmaxed to obtain the final weights.

    Note: this ensemble only supports regression tasks.
    """  # noqa
    def __init__(self, target, mixers: List[BaseMixer], data: EncodedDs, args: PredictionArguments,
                 dtype_dict: dict, accuracy_functions, ts_analysis: Optional[dict] = None,
                 fit: Optional[bool] = True, **kwargs) -> None:
        super().__init__(target, mixers, data)
        if dtype_dict[target] not in (dtype.float, dtype.integer, dtype.quantity):
            raise Exception(
                f'This ensemble can only be used regression problems! Got target dtype {dtype_dict[target]} instead!')

        if fit:
            score_list = []
            for _, mixer in enumerate(mixers):
                score_dict = evaluate_accuracies(
                    data.data_frame,
                    mixer(data, args)['prediction'],
                    target,
                    accuracy_functions,
                    ts_analysis=ts_analysis
                )
                avg_score = np.mean(list(score_dict.values()))
                log.info(f'Mixer: {type(mixer).__name__} got accuracy: {avg_score}')

                if is_nan_numeric(avg_score):
                    log.warning(f'Could not compute a valid accuracy for mixer: {type(mixer).__name__}, \ 
                                  functions: {accuracy_functions}, yielded invalid average score {avg_score}, \ 
                                  resetting that to -pow(2,63) instead.')
                    avg_score = -pow(2, 63)

                score_list.append(avg_score)

            self.weights = self.accuracies_to_weights(np.array(score_list))
            self.prepared = True
```

--------------------------------

### BaseAnalysisBlock Initialization

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/analysis/base.html

Initializes the BaseAnalysisBlock with optional dependencies. Dependencies can be used for parallelization.

```python
from typing import Tuple, Dict, Optional

import pandas as pd
from lightwood.helpers.log import log



[docs]class BaseAnalysisBlock:
    """Class to be inherited by any analysis/explainer block."""
    def __init__(self,
                 deps: Optional[Tuple] = ()
                 ):

        self.dependencies = deps  # can be parallelized when there are no dependencies @TODO enforce
```

--------------------------------

### Get Encoded Column Data

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/data/encoded_ds.html

Retrieves the encoded data for a specified column across all encoded datasets.

```APIDOC
## GET /api/data/encoded_column

### Description
Retrieves the encoded data for a specified column by concatenating data from multiple encoded datasets.

### Method
GET

### Endpoint
/api/data/encoded_column

### Parameters
#### Query Parameters
- **column_name** (str) - Required - The name of the column for which to retrieve encoded data.

### Response
#### Success Response (200)
- **encoded_data** (torch.Tensor) - The concatenated encoded data for the specified column.

#### Response Example
```json
{
  "encoded_data": "[tensor data]"
}
```
```

--------------------------------

### Get Length of EncodedDs Dataset

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/data/encoded_ds.html

Returns the total number of rows in the original dataframe, representing the dataset's length.

```python
    def __len__(self):
        """
        The length of an `EncodedDs` datasource equals the amount of rows of the original dataframe.

        :return: length of the `EncodedDs`
        """
        return int(self.data_frame.shape[0])

```

--------------------------------

### Initialize XGBoostArrayMixer

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/mixer/xgboost_array.html

Initializes the XGBoostArrayMixer with forecasting parameters and sets up individual XGBoost models for each timestep in the horizon. Requires a TimeseriesSettings object for configuration.

```python
class XGBoostArrayMixer(BaseMixer):
    """XGBoost-based model, intended for usage in forecasting tasks."""
    models: List[XGBoostMixer]
    submodel_stop_after: float
    target: str
    supports_proba: bool
    ts_analysis: Dict
    tss: TimeseriesSettings

    def __init__(
            self,
            stop_after: float,
            target: str,
            dtype_dict: Dict[str, str],
            input_cols: List[str],
            fit_on_dev: bool,
            target_encoder: BaseEncoder,
            ts_analysis: Dict[str, object],
            use_stl: bool,
            tss: TimeseriesSettings
    ):
        super().__init__(stop_after)
        self.tss = tss
        self.horizon = tss.horizon
        self.submodel_stop_after = stop_after / self.horizon
        self.target = target
        self.offset_pred_cols = [f'{self.target}_timestep_{i}' for i in range(1, self.horizon)]
        if set(input_cols) != {self.tss.order_by}:
            input_cols.remove(self.tss.order_by)
        for col in self.offset_pred_cols:
            dtype_dict[col] = dtype_dict[self.target]
        self.models = [XGBoostMixer(self.submodel_stop_after,
                                    target_col,
                                    dtype_dict,
                                    input_cols,
                                    False,  # fit_on_dev,
                                    False,  # use_optuna
                                    target_encoder)
                       for _, target_col in zip(range(self.horizon), [target] + self.offset_pred_cols)]
        self.ts_analysis = ts_analysis
        self.supports_proba = False
        self.use_stl = False
        self.stable = False
```

--------------------------------

### AutoARIMA Model Initialization

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/mixer/arima.html

Initializes the AutoARIMA model with various configuration parameters. This is part of the model's setup process.

```python
super().__init__(stop_after, target, dtype_dict, horizon, ts_analysis,
                         model_path, model_kwargs, auto_size, sp, hyperparam_search, use_stl)
        self.name = 'AutoARIMA'
        self.stable = False
```

--------------------------------

### Statistical Analysis Logs

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_mixer/custom_mixer.ipynb.txt

Logs indicating the start and end of the statistical analysis phase. This is a crucial step before model training.

```text
[32mINFO:dataprep_ml-3231:Starting statistical analysis[0m

```

```text
[32mINFO:dataprep_ml-3231:Finished statistical analysis[0m

```

--------------------------------

### Initialize ConfStats Class

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/analysis/helpers/conf_stats.html

Initializes the ConfStats class with dependencies and ECE bin count. Use this to set up the confidence statistics computation.

```python
from copy import deepcopy
from typing import Dict
from types import SimpleNamespace

from sklearn.preprocessing import OrdinalEncoder

from lightwood.analysis.base import BaseAnalysisBlock



class ConfStats(BaseAnalysisBlock):
    """
    Computes confidence-related statistics on the held-out validation dataset.

    TODO: regression & forecasting tasks
    """

    def __init__(self, deps=('ICP',), ece_bins: int = 10):
        super().__init__(deps=deps)
        self.ece_bins = ece_bins
        self.ordenc = OrdinalEncoder()
        self.n_decimals = 3
```

--------------------------------

### Get Time Series Groups

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/helpers/ts.html

Identifies and returns group combinations for time series data based on specified grouping columns.

```python
from typing import Tuple, Dict
from datetime import datetime

import numpy as np
import pandas as pd


def get_ts_groups(df: pd.DataFrame, tss) -> list:
    group_combinations = ['__default']
    if tss.group_by:
        groups = [tuple([g]) if not isinstance(g, tuple) else g
                  for g in list(df.groupby(by=tss.group_by).groups.keys())]
        group_combinations.extend(groups)
    return group_combinations
```

--------------------------------

### Initialize RandomForest Mixer

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/mixer/random_forest.html

Initializes the RandomForest mixer with configuration parameters. Supports classification and regression based on target dtype. Optionally uses Optuna for hyperparameter tuning.

```python
import time
import math
import torch
import numpy as np
import pandas as pd
import optuna
from typing import Dict, Union, Optional
from optuna import trial as trial_module
from sklearn import clone
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import check_cv, cross_val_predict
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier

from type_infer.dtype import dtype
from lightwood.helpers.log import log
from lightwood.encoder.base import BaseEncoder
from lightwood.data.encoded_ds import EncodedDs, ConcatedEncodedDs
from lightwood.mixer.base import BaseMixer
from lightwood.api.types import PredictionArguments



class RandomForest(BaseMixer):
    model: Union[RandomForestClassifier, RandomForestRegressor]
    dtype_dict: dict
    target: str
    fit_on_dev: bool
    use_optuna: bool
    supports_proba: bool

    def __init__(
            self,
            stop_after: float,
            target: str,
            dtype_dict: Dict[str, str],
            fit_on_dev: bool,
            target_encoder: BaseEncoder,
            use_optuna: bool = False,
    ):
        """
        The `RandomForest` mixer supports both regression and classification tasks. 
        It inherits from sklearn.ensemble.RandomForestRegressor and sklearn.ensemble.RandomForestClassifier.
        (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)
        (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)

        :param stop_after: time budget in seconds.
        :param target: name of the target column that the mixer will learn to predict.
        :param dtype_dict: dictionary with dtypes of all columns in the data.
        :param fit_on_dev: whether to fit on the dev dataset.
        :param use_optuna: whether to activate the automated hyperparameter search (optuna-based). Note that setting this flag to `True` does not guarantee the search will run, rather, the speed criteria will be checked first (i.e., if a single iteration is too slow with respect to the time budget, the search will not take place). 
        """  # noqa
        super().__init__(stop_after)
        self.target = target
        self.dtype_dict = dtype_dict
        self.fit_on_dev = fit_on_dev
        self.use_optuna = use_optuna
        self.target_encoder = target_encoder

        self.model = None
        self.positive_domain = False
        self.num_trials = 5
        self.cv = 3
        self.map = {}

        self.cls_dtypes = [dtype.categorical, dtype.binary, dtype.cat_tsarray]
        self.float_dtypes = [dtype.float, dtype.quantity, dtype.num_tsarray]
        self.num_dtypes = [dtype.integer] + self.float_dtypes
        self.supports_proba = dtype_dict[target] in self.cls_dtypes
        self.is_classifier = self.supports_proba

        self.stable = True

```

--------------------------------

### Get Original Column Data - ConcatedEncodedDs

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/data/encoded_ds.html

Retrieves the original data for a specified column by concatenating the results from all underlying EncodedDs objects.

```python
def get_column_original_data(self, column_name: str) -> pd.Series:
        """
        See `lightwood.data.encoded_ds.EncodedDs.get_column_original_data()`.
        """
        encoded_df_arr = [x.get_column_original_data(column_name) for x in self.encoded_ds_arr]
```

--------------------------------

### Prepare Trainable Encoders with Dependencies

Source: https://mindsdb.github.io/lightwood/tutorials/custom_splitter/custom_splitter.html

This code prepares trainable encoders, including those with dependencies and those that use the target. It constructs keyword arguments for the encoder's prepare method.

```python
# Prepare any non-target encoders that are learned
for col_name, encoder in self.encoders.items():
    if col_name != self.target and encoder.is_trainable_encoder:
        priming_data = pd.concat([data["train"Показать больше], data["dev"]])
        kwargs = {}
        if self.dependencies[col_name]:
            kwargs["dependency_data"] = {}
            for col in self.dependencies[col_name]:
                kwargs["dependency_data"][col] = {
                    "original_type": self.dtype_dict[col],
                    "data": priming_data[col],
                }

        # If an encoder representation requires the target, provide priming data
        if hasattr(encoder, "uses_target"):
            kwargs["encoded_target_values"] = self.encoders[self.target].encode(
                priming_data[self.target]
            )

        encoder.prepare(
            data["train"][col_name], data["dev"][col_name], **kwargs
        )
```

--------------------------------

### Get Item - ConcatedEncodedDs

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/data/encoded_ds.html

Retrieves an item at a specific index from the ConcatedEncodedDs. It iterates through the underlying EncodedDs objects to find the correct one.

```python
def __getitem__(self, idx: int) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        See `lightwood.data.encoded_ds.EncodedDs.__getitem__()`.
        """
        for ds_idx, length in enumerate(self.encoded_ds_lengths):
            if idx - length < 0:
                return self.encoded_ds_arr[ds_idx][idx]
            else:
                idx -= length
        raise StopIteration()
```

--------------------------------

### Load and Display CSV Data with Pandas

Source: https://mindsdb.github.io/lightwood/tutorials/tutorial_time_series/tutorial_time_series.html

Loads a CSV file from a URL into a pandas DataFrame and displays its contents. Ensure pandas is installed.

```python
import pandas as pd

df = pd.read_csv("https://raw.githubusercontent.com/mindsdb/benchmarks/main/benchmarks/datasets/monthly_sunspots/data.csv")
df
```

--------------------------------

### Loading Custom Module

Source: https://mindsdb.github.io/lightwood/tutorials/custom_encoder_rulebased/custom_encoder_rulebased.html

Demonstrates how to load the custom LabelEncoder module into the lightwood environment.

```APIDOC
## Loading Custom Module

### Description

This snippet shows how to load the `LabelEncoder.py` file as a custom module within the lightwood framework, making it available for use.

### Method

```python
from lightwood import load_custom_module
load_custom_module('LabelEncoder.py')
```

### Endpoint

N/A

### Parameters

N/A

### Request Body

N/A

### Response

N/A
```

--------------------------------

### Helpers

Source: https://mindsdb.github.io/lightwood/mixer.html

General documentation for Helper utilities.

```APIDOC
## Helpers

### Description
General documentation for Helper utilities in Lightwood. Specific endpoints or methods are not detailed in the provided text.
```

--------------------------------

### Add Custom Analysis Block

Source: https://mindsdb.github.io/lightwood/_sources/tutorials/custom_explainer/custom_explainer.ipynb.txt

Appends a custom analysis block to the Json AI object. This example adds a 'ModelCorrelationHeatmap' from the 'model_correlation' module.

```python
# add the custom list of analysis blocks; in this case, composed of a single block
json_ai.analysis_blocks = [{
    'module': 'model_correlation.ModelCorrelationHeatmap',
    'args': {}
}]
```

--------------------------------

### Import Core Lightwood and DataPrep ML Libraries

Source: https://mindsdb.github.io/lightwood/tutorials/custom_splitter/custom_splitter.html

Imports essential modules from the Lightwood library, including analysis, API, data handling, encoders, ensembles, and various helper utilities. It also imports functionalities from DataPrep ML for data insights, cleaning, and splitting.

```python
import lightwood
from lightwood import __version__ as lightwood_version
from lightwood.analysis import *
from lightwood.api import *
from lightwood.data import *
from lightwood.encoder import *
from lightwood.ensemble import *
from lightwood.helpers.device import *
from lightwood.helpers.general import *
from lightwood.helpers.ts import *
from lightwood.helpers.log import *
from lightwood.helpers.numeric import *
from lightwood.helpers.parallelism import *
from lightwood.helpers.seed import *
from lightwood.helpers.text import *
from lightwood.helpers.torch import *
from lightwood.mixer import *

from dataprep_ml.insights import statistical_analysis
from dataprep_ml.cleaners import cleaner
from dataprep_ml.splitters import splitter
from dataprep_ml.imputers import *
```

--------------------------------

### Prepare MultiHotEncoder for Priming Data

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/encoder/categorical/multihot.html

Prepares the encoder by fitting the `MultiLabelBinarizer` to the provided `priming_data`. Handles potential None values and converts all elements to strings. The `output_size` is determined after encoding the first sample.

```python
    @staticmethod
    def _clean_col_data(column_data):
        column_data = [(arr if arr is not None else []) for arr in column_data]
        column_data = [[str(x) for x in arr] for arr in column_data]
        return column_data


[docs]
    def prepare(self, priming_data, max_dimensions=100):
        priming_data = self._clean_col_data(priming_data)
        self._binarizer.fit(priming_data + [('None')])
        for arr in priming_data:
            for x in arr:
                self._seen.add(x)
        self.is_prepared = True
        self.output_size = len(self.encode(priming_data[0:1])[0])
```

--------------------------------

### Get Item from EncodedDs Dataset

Source: https://mindsdb.github.io/lightwood/_modules/lightwood/data/encoded_ds.html

Retrieves and returns the encoded features (X) and target (Y) for a given row index. Uses cache if available.

```python
    def __getitem__(self, idx: int) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        The getter yields a tuple (X, y), where:
          - `X `is a concatenation of all encoded representations of the row. Size: (B, n_features)
          - `y` is the encoded target. Size: (B, n_features)
          
        :param idx: index of the row to access.
        
        :return: tuple (X, y) with encoded data.
        
        """  # noqa
        if self.use_cache and self.X_cache[idx] is not torch.nan:
            X = self.X_cache[idx, :]
            Y = self.Y_cache[idx]
        else:
            X, Y = self._encode_idxs([idx, ])
            if self.use_cache:
                self.X_cache[idx, :] = X
                self.Y_cache[idx, :] = Y

        return X, Y

```
Tokens:
Raw