### Setup Local Development Environment Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/contributing.md Creates a virtual environment and installs the package in editable mode with development dependencies. ```bash mkdir .env python3 -m venv .env/tabular_env source .env/tabular_env/bin/activate pip install -e .[dev] ``` -------------------------------- ### Clone and Install from Source Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/README.md Commands to clone the repository from GitHub and perform a local installation. This is useful for users who want to work with the source code directly. ```bash git clone git://github.com/manujosephv/pytorch_tabular cd pytorch_tabular && pip install .[extra] ``` -------------------------------- ### Clone and Install from Source Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/gs_installation.md Clones the repository from GitHub and installs the package from the local source directory. Useful for developers or users needing the latest development version. ```bash git clone git://github.com/manujosephv/pytorch_tabular pip install . # Alternatively python setup.py install ``` -------------------------------- ### Install PyTorch Tabular Bare Essentials Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/gs_installation.md Installs only the core requirements for the library. This is suitable for minimal environments where extra features are not needed. ```bash pip install "pytorch_tabular" ``` -------------------------------- ### Custom Optimizer Example Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/02-Exploring Advanced Features with PyTorch Tabular.ipynb Demonstrates how to use a custom optimizer like Lamb from `torch_optimizer` with the `fit` method. ```APIDOC ## Custom Optimizer Example ### Description This example shows how to use a custom optimizer, such as `Lamb` from `torch_optimizer`, with the `fit` method. ### Method `TabularModel.fit()` ### Endpoint N/A (In-memory Python API) ### Parameters #### `optimizer` - **optimizer** (Callable) - The custom optimizer class (e.g., `Lamb`). #### `optimizer_params` - **optimizer_params** (Dict) - Dictionary of parameters for the custom optimizer. ### Request Body Example (Conceptual) ```python from torch_optimizer import Lamb # Assuming 'model' is an instance of TabularModel model.fit( ..., optimizer=Lamb, optimizer_params={'lr': 1e-3, 'betas': (0.9, 0.999)}, ... ) ``` ### Code Definition ```python # Import the custom optimizer from torch_optimizer import Lamb # Assign the optimizer class to a variable for clarity CustomOptimizer = Lamb ``` ### Response Example (Conceptual) Training proceeds using the specified custom optimizer. ``` -------------------------------- ### GANDALF Model Configuration and Initialization Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/01-Approaching Any Tabular Problem with PyTorch Tabular.ipynb Example of defining configurations for the GANDALF model and initializing the TabularModel. ```APIDOC ## GANDALF Model Configuration and Initialization This example demonstrates how to configure `DataConfig`, `TrainerConfig`, `OptimizerConfig`, and `GANDALFConfig` and then initialize the `TabularModel`. ### Method ```python from pytorch_tabular.models import GANDALFConfig from pytorch_tabular.config import ( DataConfig, OptimizerConfig, TrainerConfig, ) from pytorch_tabular import TabularModel # Assuming target_col, num_col_names, cat_col_names are defined target_col = "your_target_column" num_col_names = ["col1", "col2"] cat_col_names = ["cat1", "cat2"] # Define Data Configuration data_config = DataConfig( target=[target_col], continuous_cols=num_col_names, categorical_cols=cat_col_names, ) # Define Trainer Configuration trainer_config = TrainerConfig( batch_size=1024, max_epochs=100, ) # Define Optimizer Configuration (using defaults) optimizer_config = OptimizerConfig() # Define GANDALF Model Specific Configuration model_config = GANDALFConfig( task="classification", gflu_stages=6, gflu_feature_init_sparsity=0.3, gflu_dropout=0.0, learning_rate=1e-3, ) # Initialize the TabularModel tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, verbose=True ) ``` ### Parameters - **`data_config`** (DataConfig) - Configuration for data handling. - **`model_config`** (GANDALFConfig) - Configuration specific to the GANDALF model. - **`optimizer_config`** (OptimizerConfig) - Configuration for the optimizer. - **`trainer_config`** (TrainerConfig) - Configuration for the training process. - **`verbose`** (bool) - Controls the verbosity of the model output during initialization and training. ``` -------------------------------- ### Install PyTorch Tabular with Extra Dependencies Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/gs_installation.md Installs the full library including optional dependencies like Weights&Biases, Plotly, and Captum. This is recommended for users requiring experiment tracking and visualization features. ```bash pip install "pytorch_tabular[extra]" ``` -------------------------------- ### Initializing TabularModel Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tabular_model.md Demonstrates the basic and advanced usage for initializing the TabularModel, which orchestrates the entire model setup process. ```APIDOC ## Initializing Tabular Model ### Description The `TabularModel` is the central component that parses configurations, initializes the model, sets up experiment tracking, data handling, callbacks, and the PyTorch Lightning Trainer. It enables training, saving, loading, and prediction. ### Basic Usage Initialize `TabularModel` by providing configuration objects or paths to YAML files for data, model, optimizer, trainer, and experiment. ### Parameters - **data_config** (DataConfig or str) - DataConfig object or path to the data configuration yaml file. - **model_config** (ModelConfig or str) - ModelConfig object or path to the model configuration yaml file. - **optimizer_config** (OptimizerConfig or str) - OptimizerConfig object or path to the optimizer configuration yaml file. - **trainer_config** (TrainerConfig or str) - TrainerConfig object or path to the trainer configuration yaml file. - **experiment_config** (ExperimentConfig or str) - ExperimentConfig object or path to the experiment configuration yaml file. ### Request Example ```python tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, experiment_config=experiment_config, ) ``` ### Advanced Usage - **config** (DictConfig) - Initialize with an `omegaconf.DictConfig`. This bypasses validation and is primarily used for loading saved models. - **model_callable** (Optional[Callable]) - For custom models, pass the class (not the object) to override config-based initialization. ``` -------------------------------- ### Install PyTorch Tabular via Pip Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/README.md Commands to install the library using pip. Users can choose between the full version with extra dependencies like Weights&Biases and Plotly, or the bare essentials. ```bash pip install -U "pytorch_tabular[extra]" ``` ```bash pip install -U "pytorch_tabular" ``` -------------------------------- ### Custom Callback Example Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/02-Exploring Advanced Features with PyTorch Tabular.ipynb Demonstrates how to integrate custom callbacks with PyTorch Lightning and the `fit` method. ```APIDOC ## Custom Callback Example ### Description This example illustrates how to use custom callbacks, which are directly added to the PyTorch Lightning Trainer via the `fit` method. ### Method `TabularModel.fit()` ### Endpoint N/A (In-memory Python API) ### Parameters #### `callbacks` - **callbacks** (List[pytorch_lightning.callbacks.Callback]) - A list that can include custom callback instances. ### Request Body Example (Conceptual) ```python from pytorch_lightning.callbacks import Callback class CustomCallback(Callback): def on_train_start(self, trainer, pl_module): print("Custom callback: Training is starting!") # Assuming 'model' is an instance of TabularModel model.fit( ..., callbacks=[CustomCallback()], ... ) ``` ### Response Example (Conceptual) Custom messages or actions will be triggered at different stages of the training process as defined by the callback. ``` -------------------------------- ### Initialize TabularModel with Configurations Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/02-Exploring Advanced Features with PyTorch Tabular.ipynb Initializes the TabularModel by passing various configuration objects and a YAML file for trainer settings. This model orchestrates data transformations, model setup, experiment tracking, and PyTorch Lightning Trainer configuration. It supports training, saving, loading, and prediction functionalities. ```python from pytorch_tabular import TabularModel tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config="trainer_config.yml", verbose=True, suppress_lightning_logger=False ) ``` -------------------------------- ### PyTorch Tabular Model Configuration Examples Source: https://context7.com/pytorch-tabular/pytorch_tabular/llms.txt Illustrates how to configure various deep learning architectures available in PyTorch Tabular for tabular data. Each configuration class allows customization of task type, architecture-specific hyperparameters, and common model parameters. ```python from pytorch_tabular.models import ( CategoryEmbeddingModelConfig, # Simple MLP with embeddings TabNetModelConfig, # Google's TabNet NodeConfig, # Neural Oblivious Decision Ensembles FTTransformerConfig, # Feature Tokenizer + Transformer GANDALFConfig, # Gated Adaptive Network for Deep Automated Learning GatedAdditiveTreeEnsembleConfig, # GATE model AutoIntConfig, # Automatic Feature Interaction TabTransformerConfig, # Tab Transformer DANetConfig, # Deep Abstract Networks MDNConfig # Mixture Density Networks (probabilistic) ) # CategoryEmbeddingModel - Simple but effective baseline ce_config = CategoryEmbeddingModelConfig( task="regression", layers="256-128-64", activation="LeakyReLU", dropout=0.1, use_batch_norm=True, initialization="kaiming" ) # GANDALF - High performance with feature selection gandalf_config = GANDALFConfig( task="classification", gflu_stages=6, gflu_dropout=0.0, gflu_feature_init_sparsity=0.3, learnable_sparsity=True ) # FT-Transformer - Transformer architecture for tabular data ft_config = FTTransformerConfig( task="classification", num_heads=8, num_attn_blocks=6, attn_dropout=0.1, ff_dropout=0.1, embedding_initialization="kaiming_uniform" ) ``` -------------------------------- ### Launch TensorBoard for PyTorch Tabular Experiments Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/05.1-Experiment_Tracking_using_Tensorboard.ipynb Instructions on how to launch TensorBoard to visualize experiment runs. This involves navigating to the project directory in the terminal and running the tensorboard command with the log directory specified. This allows for interactive monitoring of training progress and results. ```bash tensorboard --logdir ``` -------------------------------- ### Configure Data, Trainer, and Optimizer Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/16-Model Stacking.ipynb Sets up the configuration objects required for training, including data column definitions, trainer hyperparameters, and optimizer settings. ```python data_config = DataConfig( target=["target"], continuous_cols=num_col_names, categorical_cols=cat_col_names, ) trainer_config = TrainerConfig( batch_size=1024, max_epochs=20, early_stopping="valid_accuracy", early_stopping_mode="max", early_stopping_patience=3, checkpoints="valid_accuracy", load_best=True, ) optimizer_config = OptimizerConfig() ``` -------------------------------- ### Configure Metrics and Parameters Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/models.md Shows how to define a list of metrics, specify whether they require probability inputs, and pass additional parameters to those metrics. ```python metrics = ["accuracy", "f1_score"] metrics_prob_input = [False, False] metrics_params = [{}, {"num_classes": 2}] ``` -------------------------------- ### Cross-Validation Setup with Scikit-learn Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/12-Bagged Predictions.ipynb Initializes a KFold object from scikit-learn for performing cross-validation. It specifies the number of splits, whether to shuffle the data, and a random state for reproducibility. This setup is crucial for evaluating model performance across different subsets of the data. ```python from sklearn.model_selection import KFold from sklearn.metrics import accuracy_score, f1_score kf = KFold(n_splits=5, shuffle=True, random_state=42) ``` -------------------------------- ### Configure Data, Trainer, and Optimizer Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/15-Search Best Architecture and Hyperparameter.ipynb Sets up the configuration objects for data handling, model training, and optimization. This includes defining target columns, feature types, batch size, maximum epochs, early stopping criteria, and checkpointing. ```python data_config = DataConfig( target=[ "target" ], continuous_cols=num_col_names, categorical_cols=cat_col_names, ) trainer_config = TrainerConfig( batch_size=32, max_epochs=50, early_stopping="valid_accuracy", early_stopping_mode="max", early_stopping_patience=3, checkpoints="valid_accuracy", load_best=True, progress_bar="none" ) optimizer_config = OptimizerConfig() ``` -------------------------------- ### GET /model/load Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/01-Approaching Any Tabular Problem with PyTorch Tabular.ipynb Loads a previously saved model from a directory. ```APIDOC ## GET /model/load ### Description Loads a saved model and its associated datamodule from the specified path. ### Method GET ### Endpoint TabularModel.load_model(path) ### Parameters #### Request Body - **path** (string) - Required - The directory path where the model was saved. ### Request Example ```python loaded_model = TabularModel.load_model("path/to/save") ``` ``` -------------------------------- ### GET /models/load Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/16-Model Stacking.ipynb Loads a previously saved stacking model from a specified directory. ```APIDOC ## GET /models/load ### Description Loads a saved TabularModel from the disk, allowing for inference or continued usage. ### Method GET ### Endpoint TabularModel.load_model(path) ### Parameters #### Query Parameters - **path** (string) - Required - The directory path containing the saved model artifacts. ### Request Example loaded_model = TabularModel.load_model("stacking_model") ### Response #### Success Response (200) - **model** (object) - Returns an instance of the loaded TabularModel. ``` -------------------------------- ### Configure and Initialize TabularModel Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/09-Cross Validation.ipynb Sets up data, trainer, optimizer, and model configurations, then initializes the TabularModel. This process defines the architecture and training parameters for the model. ```python from pytorch_tabular import TabularModel from pytorch_tabular.models import CategoryEmbeddingModelConfig from pytorch_tabular.config import DataConfig, OptimizerConfig, TrainerConfig from pytorch_tabular.models.common.heads import LinearHeadConfig data_config = DataConfig(target=["target"], continuous_cols=num_col_names, categorical_cols=cat_col_names) trainer_config = TrainerConfig(batch_size=1024, max_epochs=100, early_stopping="valid_loss", early_stopping_mode="min", early_stopping_patience=5, checkpoints="valid_loss", load_best=True, progress_bar="none", trainer_kwargs=dict(enable_model_summary=False)) optimizer_config = OptimizerConfig() head_config = LinearHeadConfig(layers="", dropout=0.1, initialization="kaiming").__dict__ model_config = CategoryEmbeddingModelConfig(task="classification", layers="1024-512-512", activation="LeakyReLU", learning_rate=1e-3, head="LinearHead", head_config=head_config) tabular_model = TabularModel(data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, verbose=False) ``` -------------------------------- ### Download and Prepare Bank Marketing Dataset Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/examples/PyTorch Tabular with Bank Marketing Dataset.ipynb Downloads the 'Bank marketing data set' from OpenML, joins features and target, and displays the first few rows. This step is crucial for setting up the dataset for subsequent analysis. ```python np.random.seed(42) X, y = fetch_openml("Bank_marketing_data_set_UCI", version=1, as_frame=True, return_X_y=True) data = X.join(y) del X, y data.head() ``` -------------------------------- ### GET /tabular/evaluate Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/16-Model Stacking.ipynb Evaluates a trained model on a test dataset and returns performance metrics. ```APIDOC ## GET /tabular/evaluate ### Description Runs the evaluation loop on a provided test dataset using the best weights loaded from the trained model. ### Method GET ### Endpoint /tabular/evaluate ### Parameters #### Query Parameters - **test_data** (object) - Required - The dataset to evaluate against. ### Response #### Success Response (200) - **test_accuracy** (float) - The accuracy metric calculated on the test set. #### Response Example { "test_accuracy": 0.4586 } ``` -------------------------------- ### GET /model/evaluate Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/05.1-Experiment_Tracking_using_Tensorboard.ipynb Evaluates a trained tabular model on a test dataset and returns performance metrics. ```APIDOC ## GET /model/evaluate ### Description Runs evaluation on the test dataset using the trained model instance to calculate metrics like accuracy and loss. ### Method GET ### Endpoint /model/evaluate ### Parameters #### Query Parameters - **test_data** (object) - Required - The dataset to evaluate against ### Request Example { "test_data": "test_dataframe_or_dataloader" } ### Response #### Success Response (200) - **test_accuracy** (float) - The accuracy score on the test set - **test_loss** (float) - The loss value on the test set #### Response Example { "test_accuracy": 0.912, "test_loss": 0.213 } ``` -------------------------------- ### Target Transform Example Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/02-Exploring Advanced Features with PyTorch Tabular.ipynb Demonstrates how to apply custom transformations to the target variable using `target_transform`. ```APIDOC ## Target Transform Example ### Description This example shows how to use the `target_transform` parameter to apply custom transformations to the target variable before it's used in the loss function and after the model's output. ### Method `TabularModel.fit()` ### Endpoint N/A (In-memory Python API) ### Parameters #### `target_transform` - **target_transform** (Tuple[Callable, Callable]) - A tuple containing two callables. The first transforms the target before the loss calculation, and the second transforms the model's output. ### Request Body Example (Conceptual) ```python import numpy as np # Assuming 'model' is an instance of TabularModel model.fit( ..., target_transform=[np.log, np.exp], # Example: log transform target, exp transform prediction ... ) ``` ### Response Example (Conceptual) The target variable and model predictions will be transformed according to the specified functions during training and evaluation. ``` -------------------------------- ### Initialize Basic Model Configuration Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/models.md Demonstrates the simplest way to initialize a model configuration for a classification task. The system automatically infers remaining parameters using intelligent defaults. ```python model_config = (task="classification") ``` -------------------------------- ### Initialize TabularModel with Configurations Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/01-Approaching Any Tabular Problem with PyTorch Tabular.ipynb This snippet shows how to combine the previously defined configuration objects into the main TabularModel class for training. ```python from pytorch_tabular import TabularModel tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, verbose=True ) ``` -------------------------------- ### Configure and Initialize GANDALF Model Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/14-Explainability.ipynb Sets up the necessary configurations for data, training, optimization, and the GANDALF model architecture, then initializes the TabularModel. ```python from pytorch_tabular import TabularModel from pytorch_tabular.models import GANDALFConfig from pytorch_tabular.config import DataConfig, OptimizerConfig, TrainerConfig from pytorch_tabular.models.common.heads import LinearHeadConfig data_config = DataConfig( target=["target"], continuous_cols=num_col_names, categorical_cols=cat_col_names, ) trainer_config = TrainerConfig( auto_lr_find=True, batch_size=1024, max_epochs=100, accelerator="auto", ) optimizer_config = OptimizerConfig() head_config = LinearHeadConfig( layers="", dropout=0.1, initialization="kaiming", ).__dict__ model_config = GANDALFConfig( task="classification", gflu_stages=3, gflu_dropout=0.0, gflu_feature_init_sparsity=0.1, head="LinearHead", head_config=head_config, learning_rate=1e-3, ) tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, verbose=False, ) ``` -------------------------------- ### Custom Metric Function Example Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/02-Exploring Advanced Features with PyTorch Tabular.ipynb Demonstrates how to define and use a custom metric function with the `fit` method. ```APIDOC ## Custom Metric Function Example ### Description This example shows how to create a custom metric function and pass it to the `fit` method. ### Method `TabularModel.fit()` ### Endpoint N/A (In-memory Python API) ### Parameters #### `metrics` - **metrics** (List[Callable]) - A list containing the custom metric function. #### `metrics_prob_inputs` - **metrics_prob_inputs** (List[bool]) - A list indicating whether the metric expects probability inputs (True) or class inputs (False). ### Request Body Example (Conceptual) ```python # Assuming 'model' is an instance of TabularModel model.fit( ..., metrics=[custom_metric], metrics_prob_inputs=[False], # Assuming the metric does not need probabilities ... ) ``` ### Code Definition ```python def custom_metric(y_true, y_pred): # Example: Mean of cubed differences return torch.mean(torch.pow(y_true - y_pred, 3)) ``` ### Response Example (Conceptual) Training proceeds with the custom metric being calculated. ``` -------------------------------- ### Initialize TabularModel in Python Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tabular_model.md Demonstrates the basic usage of initializing the TabularModel by passing configuration objects for data, model, optimizer, trainer, and experiment tracking. This model orchestrates the entire training and prediction pipeline. ```python tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, experiment_config=experiment_config, ) ``` -------------------------------- ### Custom Loss Function Example Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/02-Exploring Advanced Features with PyTorch Tabular.ipynb Demonstrates how to define and use a custom loss function with the `fit` method. ```APIDOC ## Custom Loss Function Example ### Description This example shows how to create a custom Mean Squared Error (MSE) loss function and pass it to the `fit` method. ### Method `TabularModel.fit()` ### Endpoint N/A (In-memory Python API) ### Request Body Example (Conceptual) ```python # Assuming 'model' is an instance of TabularModel model.fit( ..., loss=CustomLoss(), ... ) ``` ### Code Definition ```python import torch import torch.nn as nn class CustomLoss(nn.Module): def __init__(self): super(CustomLoss, self).__init__() def forward(self, inputs, targets): # Example: Mean Squared Error loss = torch.mean((inputs - targets) ** 2) return loss.mean() ``` ### Response Example (Conceptual) Training proceeds with the custom loss function. ``` -------------------------------- ### Configure Model and Trainer Parameters Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/05.1-Experiment_Tracking_using_Tensorboard.ipynb Sets up the data, trainer, and model configurations required for the TabularModel. It includes settings for early stopping, learning rate finding, and model architecture. ```python data_config = DataConfig(target=[target_col], continuous_cols=num_col_names, categorical_cols=cat_col_names) trainer_config = TrainerConfig(auto_lr_find=True, batch_size=1024, max_epochs=100, early_stopping="valid_loss", early_stopping_mode="min", early_stopping_patience=5, checkpoints="valid_loss", load_best=True) optimizer_config = OptimizerConfig() head_config = LinearHeadConfig(layers="", dropout=0.1, initialization="kaiming").__dict__ model_config = CategoryEmbeddingModelConfig(task="classification", layers="1024-512-512", activation="LeakyReLU", learning_rate=1e-3, head="LinearHead", head_config=head_config) ``` -------------------------------- ### Debugging Utilities Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/training.md PyTorch Tabular adopts PyTorch Lightning's debugging features to help identify performance bottlenecks and setup issues. ```APIDOC ## Debugging Utilities PyTorch Tabular provides several debugging utilities inherited from PyTorch Lightning to assist in model development and performance analysis. ### Profiler - **profiler** (Optional[str]): Enables profiling of individual training steps to identify performance bottlenecks. - Choices: `None`, `simple`, `advanced` - Defaults to `None` ### Fast Development Run - **fast_dev_run** (Optional[bool]): Performs a quick debug run using a small subset of data to check if the entire setup runs without errors. - Defaults to `False` ### Overfitting Batches - **overfit_batches** (float): Uses a specified fraction of the training data to quickly test if the model can overfit. If set to a non-zero value, the same data is used for training, validation, and testing. Shuffling is automatically disabled for training dataloaders. - Defaults to `0` ### Gradient Norm Tracking - **track_grad_norm** (bool): Tracks and logs gradient norms if experiment tracking is enabled. Useful for diagnosing learning issues. A falling gradient norm to zero quickly indicates a problem. - Values: `False` (no tracking), `1` (L1 norm), `2` (L2 norm), etc. - Defaults to `False` ``` -------------------------------- ### Initialize Experiment Tracking Dependencies Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/05.1-Experiment_Tracking_using_Tensorboard.ipynb This snippet imports necessary libraries for data handling, evaluation metrics, and experiment tracking integration. It sets up the environment for training and monitoring machine learning models. ```python from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score, f1_score import random from pytorch_tabular.utils import load_covertype_dataset, print_metrics import pandas as pd import wandb ``` -------------------------------- ### Configure and Train a Tabular Model Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/07-Probabilistic Regression with MDN.ipynb This snippet demonstrates how to define the training, optimizer, and model configurations, initialize the TabularModel, and execute the training process using the fit method. ```python trainer_config = TrainerConfig( auto_lr_find=True, batch_size=batch_size, max_epochs=epochs, early_stopping="valid_loss", early_stopping_patience=5, checkpoints="valid_loss" ) optimizer_config = OptimizerConfig(lr_scheduler="ReduceLROnPlateau", lr_scheduler_params={"patience":3}) model_config = CategoryEmbeddingModelConfig( task="regression", layers="16-8", activation="ReLU", head="LinearHead", learning_rate=1e-3, ) tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config ) tabular_model.fit(train=df_train, validation=df_valid) ``` -------------------------------- ### Configure TabularModel for Classification Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/15-Multi Target Classification.ipynb Defines the training, optimization, and model architecture configurations. This setup includes early stopping, learning rate finding, and a CategoryEmbeddingModel with a linear head. ```python from pytorch_tabular.config import DataConfig, OptimizerConfig, TrainerConfig from pytorch_tabular.models import CategoryEmbeddingModelConfig from pytorch_tabular.models.common.heads import LinearHeadConfig trainer_config = TrainerConfig(auto_lr_find=True, batch_size=1024, max_epochs=100, early_stopping="valid_loss", early_stopping_patience=5, checkpoints="valid_loss", load_best=True) optimizer_config = OptimizerConfig() head_config = LinearHeadConfig(layers="", dropout=0.1, initialization="kaiming").__dict__ model_config = CategoryEmbeddingModelConfig(task="classification", layers="1024-512-512", activation="LeakyReLU", head="LinearHead", head_config=head_config, learning_rate=1e-3, metrics=["f1_score","accuracy","auroc"], metrics_prob_input=[True, False, True]) ``` -------------------------------- ### Generate Non-Linear Example Data Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/07-Probabilistic Regression with MDN.ipynb Generates a synthetic dataset for demonstrating non-linear regression tasks. It returns training, validation, and testing dataframes, along with the target column name. ```python df_train, df_valid, df_test, target_col = generate_non_linear_example() ``` -------------------------------- ### Initialize Synthetic Dataset for PyTorch Tabular Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/04-Implementing New Architectures.ipynb Demonstrates how to generate a synthetic dataset and split it into training, validation, and testing sets using PyTorch Tabular utilities and scikit-learn. ```python from pytorch_tabular.utils import make_mixed_dataset from sklearn.model_selection import train_test_split data, cat_col_names, num_col_names = make_mixed_dataset( task="regression", n_samples=10000, n_features=20, n_categories=4 ) train, test = train_test_split(data, random_state=42) train, val = train_test_split(train, random_state=42) ``` -------------------------------- ### Model Sweep with Preset and Custom Models Source: https://context7.com/pytorch-tabular/pytorch_tabular/llms.txt Demonstrates using the `model_sweep` function with predefined model lists ('lite', 'standard', 'full', 'high_memory') and custom model configurations. It shows how to initiate a model search and print comparison results. ```python from pytorch_tabular.models import CategoryEmbeddingModelConfig, GANDALFConfig # Using preset model list results, best_model = model_sweep( task="classification", train=train_df, test=test_df, data_config=data_config, optimizer_config=optimizer_config, trainer_config=trainer_config, model_list="lite", # Presets: "lite", "standard", "full", "high_memory" rank_metric=("accuracy", "higher_is_better"), return_best_model=True, progress_bar=True, verbose=True ) print("Model Comparison Results:") print(results[['model', '# Params', 'test_accuracy', 'test_loss', 'epochs']]) # Use custom model configurations custom_models = [ CategoryEmbeddingModelConfig(task="classification", layers="128-64"), CategoryEmbeddingModelConfig(task="classification", layers="256-128-64"), GANDALFConfig(task="classification", gflu_stages=6), GANDALFConfig(task="classification", gflu_stages=10), ] results, best_model = model_sweep( task="classification", train=train_df, test=test_df, data_config=data_config, optimizer_config=optimizer_config, trainer_config=trainer_config, model_list=custom_models, return_best_model=True ) ``` -------------------------------- ### Configure Tabular Model for Classification Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/11-Test Time Augmentation.ipynb Sets up the configuration for a tabular model, including data parameters, training parameters, optimizer, and model head. This example focuses on a classification task with a LinearHead. ```python data_config = DataConfig( target=[ target_col ], # target should always be a list. Multi-targets are only supported for regression. Multi-Task Classification is not implemented continuous_cols=num_col_names, categorical_cols=cat_col_names, ) trainer_config = TrainerConfig( batch_size=1024, max_epochs=100, early_stopping="valid_loss", # Monitor valid_loss for early stopping early_stopping_mode="min", # Set the mode as min because for val_loss, lower is better early_stopping_patience=5, # No. of epochs of degradation training will wait before terminating checkpoints="valid_loss", # Save best checkpoint monitoring val_loss load_best=True, # After training, load the best checkpoint # progress_bar="none", # Turning off Progress bar # trainer_kwargs=dict( # enable_model_summary=False # Turning off model summary # ) ) optimizer_config = OptimizerConfig() head_config = LinearHeadConfig( layers="", dropout=0.1, initialization="kaiming" # No additional layer in head, just a mapping layer to output_dim ).__dict__ # Convert to dict to pass to the model config (OmegaConf doesn't accept objects) model1_config = CategoryEmbeddingModelConfig( task="classification", layers="1024-512-512", # Number of nodes in each layer activation="LeakyReLU", # Activation between each layers learning_rate=1e-3, head="LinearHead", # Linear Head head_config=head_config, # Linear Head Config ) model_config = CategoryEmbeddingModelConfig( task="classification", layers="1024-512-512", # Number of nodes in each layer activation="LeakyReLU", # Activation between each layers learning_rate=1e-3, head="LinearHead", # Linear Head head_config=head_config, # Linear Head Config ) tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, verbose=False ) ``` -------------------------------- ### Train PyTorch Tabular Model Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/06-Imbalanced Classification.ipynb This code initiates the training process for a PyTorch Tabular model. It logs the start of training and indicates the environment configuration, such as the local rank and visible CUDA devices. ```python tabular_model.fit(train, validation_data=valid) ``` -------------------------------- ### Optimizer Configuration for Fine-tuning Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/08-Self-Supervised Learning-DAE.ipynb Sets up the optimizer configuration for the fine-tuning process, specifying the learning rate scheduler and its parameters. It also defines the custom optimizer (QHAdam) and its specific parameters. ```python ft_optimizer_config = OptimizerConfig( lr_scheduler="OneCycleLR", lr_scheduler_params={ "max_lr":1e-3, "epochs": epochs, "steps_per_epoch":steps_per_epoch } ) finetune_model = ssl_tabular_model.create_finetune_model( task="classification", train=finetune_train, validation=finetune_val, target=[target_name], #Provide the column name of the target as a list head="LinearHead", head_config={ "layers": "256-64", "activation": "ReLU", }, trainer_config=ft_trainer_config,# Overriding previous trainer config optimizer_config=ft_optimizer_config, optimizer=QHAdam, # Using a custom optimizer optimizer_params={"nus": (0.7, 1.0), "betas": (0.95, 0.998)} ) ``` -------------------------------- ### Display DataFrame of Results (Pandas) Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/08-Self-Supervised Learning-DAE.ipynb Converts a list of results into a Pandas DataFrame for structured display and analysis. This allows for easy comparison of metrics across different experimental setups, such as supervised versus self-supervised learning. ```python pd.DataFrame(results) ``` -------------------------------- ### Model Training and Evaluation in PyTorch Tabular Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/16-Model Stacking.ipynb This snippet demonstrates the typical flow of training a model using PyTorch Tabular. It includes steps like preparing data loaders, setting up the model and trainer, initiating training, and loading the best model post-training. It also shows how to report test metrics. ```python print("Stacking Model Test Accuracy: {}".format(stacking_acc)) print("Category Embedding Model Test Accucacy: {}".format(ce_acc)) print("FT Transformer Model Test Accuracy: {}".format(ft_acc)) print("TabNet Model Test Accuracy: {}".format(tab_acc)) ``` -------------------------------- ### Model Explainability with Feature Attributions Source: https://context7.com/pytorch-tabular/pytorch_tabular/llms.txt Explains how to obtain feature attributions and model interpretability using PyTorch Tabular's integration with Captum. It covers getting model-specific feature importance and generating SHAP-like attributions. ```python from pytorch_tabular import TabularModel # After training tabular_model.fit(train=train_df) # Get feature importance (model-specific, where supported) try: importance = tabular_model.feature_importance() print(importance.sort_values(ascending=False)) except NotImplementedError: print("Feature importance not available for this model") # Get SHAP-like attributions using Captum ``` -------------------------------- ### Finish Weights & Biases Run Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/05-Experiment_Tracking_using_WandB.ipynb Explicitly calls the `wandb.finish()` function to end the current Weights & Biases run. This is a good practice to ensure that all logging and resources are properly closed before starting a new experiment or exiting the script. ```python wandb.finish() ``` -------------------------------- ### Initialize and Tune TabularModelTuner in Python Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/10-Hyperparameter Tuning.ipynb This snippet demonstrates how to initialize the TabularModelTuner with various configurations and then use its 'tune' method to search for optimal hyperparameters. It includes setting up data, model, optimizer, and trainer configurations, and specifies the search strategy, metric, and mode. ```python from pytorch_tabular.tabular_model_tuner import TabularModelTuner tuner = TabularModelTuner( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config ) with warnings.catch_warnings(): warnings.simplefilter("ignore") result = tuner.tune( train=train, validation=test, search_space=search_space, strategy="grid_search", # cv=5, # Uncomment this to do a 5 fold cross validation metric="accuracy", mode="max", progress_bar=True, verbose=False # Make True if you want to log metrics and params each iteration ) ``` -------------------------------- ### Configure and Initialize TabularModel with Custom Model Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/04-Implementing New Architectures.ipynb Shows how to set up configuration objects and initialize TabularModel using the model_callable parameter to inject a custom model class. ```python data_config = DataConfig(target=["target"], continuous_cols=num_col_names, categorical_cols=cat_col_names) trainer_config = TrainerConfig(auto_lr_find=True, batch_size=1024, max_epochs=100, accelerator="auto", devices=-1) optimizer_config = OptimizerConfig() head_config = LinearHeadConfig(layers="32", activation="ReLU", dropout=0.1, initialization="kaiming").__dict__ model_config = MyAwesomeModelConfig(task="regression", use_batch_norm=False, head="LinearHead", head_config=head_config, learning_rate=1e-3) tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, model_callable=MyAwesomeRegressionPTModel ) ``` -------------------------------- ### PyTorch Tabular Model Training Output and Logging Source: https://github.com/pytorch-tabular/pytorch_tabular/blob/main/docs/tutorials/04-Implementing New Architectures.ipynb This section shows typical output logs during PyTorch Tabular model training. It includes information about data preparation, model setup, trainer configuration, and auto LR finding. ```text Seed set to 42\n\n2023-12-26 16:43:10,447 - {pytorch_tabular.tabular_model:506} - INFO - Preparing the DataLoaders \n\n2023-12-26 16:43:10,455 - {pytorch_tabular.tabular_datamodule:431} - INFO - Setting up the datamodule for \nregression task \n\n2023-12-26 16:43:10,490 - {pytorch_tabular.tabular_model:556} - INFO - Preparing the Model: Model \n\n2023-12-26 16:43:10,541 - {pytorch_tabular.tabular_model:322} - INFO - Preparing the Trainer \n\nGPU available: True (cuda), used: True\nTPU available: False, using: 0 TPU cores\nIPU available: False, using: 0 IPUs\nHPU available: False, using: 0 HPUs\n\n2023-12-26 16:43:10,722 - {pytorch_tabular.tabular_model:612} - INFO - Auto LR Find Started \n\nFinding best initial lr: 0%| | 0/100 [00:00