### Install Transtab from GitHub using Pip

Source: https://transtab.readthedocs.io/en/latest/install.html

Install transtab directly from its GitHub repository using pip.

```bash
pip install git+https://github.com/RyanWangZf/transtab.git

```

--------------------------------

### Example Dataset Configuration

Source: https://transtab.readthedocs.io/en/latest/data_preparation.html

Provides an example of a dataset configuration dictionary, specifying binary, categorical, and numerical columns, as well as data splitting indices.

```python
EXAMPLE_DATACONFIG = {
    "example": { # dataset name
        "bin": ["bin1", "bin2"], # binary column names
        "cat": ["cat1", "cat2"], # categorical column names
        "num": ["num1", "num2"], # numerical column names
        "cols": ["bin1", "bin2", "cat1", "cat2", "num1", "num2"], # all column names
        "binary_indicator": ["1", "yes", "true", "positive", "t", "y"], # binary indicators in the binary columns, which will be converted to 1
        "data_split_idx": {
            "train":[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], # row indices for training set
            "val":[10, 11, 12, 13, 14, 15, 16, 17, 18, 19], # row indices for validation set
            "test":[20, 21, 22, 23, 24, 25, 26, 27, 28, 29], # row indices for test set
            }
        }
    }

}
```

--------------------------------

### Install Transtab from Local Clone

Source: https://transtab.readthedocs.io/en/latest/install.html

Clone the transtab repository and install it locally. This is useful for development or if you need to modify the source code.

```bash
git clone https://github.com/RyanWangZf/transtab.git
cd transtab
pip install .

```

--------------------------------

### Install Transtab using Pip

Source: https://transtab.readthedocs.io/en/latest/install.html

Use this command to install the transtab library directly from the Python Package Index.

```bash
pip install transtab

```

--------------------------------

### Basic Usage of transtab

Source: https://transtab.readthedocs.io/en/latest/index.html

Demonstrates the fundamental steps for using transtab, including loading data, building a classifier, training the model, and making predictions. Ensure you have the transtab library installed and necessary data loaded.

```python
import transtab

# load dataset by specifying dataset name
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \
    = transtab.load_data('credit-g')

# build classifier
model = transtab.build_classifier(cat_cols, num_cols, bin_cols)

# start training
transtab.train(model, trainset, valset, **training_arguments)

# make predictions, df_x is a pd.DataFrame with shape (n, d)
# return the predictions ypred with shape (n, 1) if binary classification;
# (n, n_class) if multiclass classification.
ypred = transtab.predict(model, df_x)

```

--------------------------------

### Trainer Initialization and Training

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/transtab.html

Initializes the Trainer with a model, training set, validation set, and various training arguments. The train() method is then called to start the training process.

```python
        'warmup_ratio':warmup_ratio,
        'warmup_steps':warmup_steps,
        'eval_metric':eval_metric,
        'output_dir':output_dir,
        'collate_fn':collate_fn,
        'num_workers':num_workers,
        'balance_sample':balance_sample,
        'load_best_at_last':load_best_at_last,
        'ignore_duplicate_cols':ignore_duplicate_cols,
        'eval_less_is_better':eval_less_is_better,
    }
    trainer = Trainer(
        model,
        trainset,
        valset,
        **train_args,
    )
    trainer.train()
```

--------------------------------

### Load Data and Train Transtab Classifier

Source: https://transtab.readthedocs.io/en/latest/example_encode.html

Loads a dataset, builds a contrastive learner model, and trains it with specified arguments. Ensure the 'transtab' library is installed.

```python
import transtab

# load a dataset and start vanilla supervised training
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \
    = transtab.load_data('credit-g')

# build transtab classifier model
model, collate_fn = transtab.build_contrastive_learner(cat_cols, num_cols, bin_cols)

# start training
training_arguments = {
    'num_epoch':50,
    'batch_size':64,
    'lr':1e-4,
    'eval_metric':'val_loss',
    'eval_less_is_better':True,
    'output_dir':'./checkpoint'
    }
transtab.train(model, trainset, valset, collate_fn=collate_fn, **training_arguments)
```

--------------------------------

### Start Contrastive Pretraining Training

Source: https://transtab.readthedocs.io/en/latest/example_pretrain.html

Initiates the contrastive pretraining process. The training_arguments dictionary specifies hyperparameters like epochs, batch size, learning rate, and output directory for saving the model. The collate_fn is passed to the train function.

```python
# start contrastive pretraining training
training_arguments = {
    'num_epoch':50,
    'batch_size':64,
    'lr':1e-4,
    'eval_metric':'val_loss',
    'eval_less_is_better':True,
    'output_dir':'./checkpoint' # save the pretrained model
    }

# pass the collate function to the train function
transtab.train(model, trainset, valset, collate_fn=collate_fn, **training_arguments)

```

--------------------------------

### build_contrastive_learner

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/transtab.html

Builds a contrastive learner for pretraining based on TransTab. This function supports both supervised and unsupervised contrastive learning setups.

```APIDOC
## build_contrastive_learner

### Description
Build a contrastive learner for pretraining based on TransTab. If no cat/num/bin specified, the model takes ALL as categorical columns, which may undermine the performance significantly.

If there is one column assigned to more than one type, e.g., the feature age is both nominated as categorical and binary columns, the model will raise errors. set ``ignore_duplicate_cols=True`` to avoid this error as the model will ignore this duplicate feature.

### Parameters
#### Path Parameters
- **categorical_columns** (list) - Optional - a list of categorical feature names.
- **numerical_columns** (list) - Optional - a list of numerical feature names.
- **binary_columns** (list) - Optional - a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1).
- **hidden_dim** (int) - Optional - the dimension of hidden embeddings.
- **num_layer** (int) - Optional - the number of transformer layers used in the encoder.
- **num_attention_head** (int) - Optional - the number of heads of multihead self-attention layer in the transformers.
- **hidden_dropout_prob** (float) - Optional - the dropout ratio in the transformer encoder.
- **ffn_dim** (int) - Optional - the dimension of feed-forward layer in the transformer layer.
- **projection_dim** (int) - Optional - the dimension of projection head on the top of encoder.
- **overlap_ratio** (float) - Optional - the overlap ratio of columns of different partitions when doing subsetting.
- **num_partition** (int) - Optional - the number of partitions made for vertical-partition contrastive learning.
- **supervised** (bool) - Optional - indicates whether to use supervised contrastive learning.
- **device** (str) - Optional - the device, ``"cpu"`` or ``"cuda:0"``.
- **checkpoint** (str) - Optional - the directory to load the pretrained TransTab model.
- **ignore_duplicate_cols** (bool) - Optional - if True, ignore duplicate columns across feature types to avoid errors.

### Returns
- An instance of a contrastive learner model.
```

--------------------------------

### Troubleshoot Tokenizers Build Error on MAC/Linux

Source: https://transtab.readthedocs.io/en/latest/install.html

If you encounter a 'Failed building wheel for tokenizers' error on MAC or Linux, run this command to install Rust, then restart your terminal and try the pip install again.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

```

--------------------------------

### Project File Structure for Custom Datasets

Source: https://transtab.readthedocs.io/en/latest/data_preparation.html

Illustrates the recommended directory structure for organizing custom datasets, including processed data and feature files.

```text
project
|
├── run_your_model.py
|
└─── data
     |
     ├── dataset1
     |   |    data_processed.csv
     |   |    binary_feature.txt
     |   └─── numerical_feature.txt
     |
     ├── dataset2
     |
    ...
```

--------------------------------

### Get Activation Function

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Returns the corresponding PyTorch functional activation function based on the input string. Supports 'relu', 'gelu', 'selu', and 'leakyrelu'.

```python
def _get_activation_fn(activation):
    if activation == "relu":
        return F.relu
    elif activation == "gelu":
        return F.gelu
    elif activation == 'selu':
        return F.selu
    elif activation == 'leakyrelu':
        return F.leaky_relu
    raise RuntimeError("activation should be relu/gelu/selu/leakyrelu, not {}".format(activation))
```

--------------------------------

### Load Multiple Datasets and Build Contrastive Learner

Source: https://transtab.readthedocs.io/en/latest/example_pretrain.html

Loads multiple datasets and initializes a contrastive learner model. Set supervised=True for supervised VPCL. The num_partition and overlap_ratio parameters control column partitioning for sampling.

```python
import transtab

# load multiple datasets by passing a list of data names
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols     = transtab.load_data(['credit-g','credit-approval'])

# build contrastive learner, set supervised=True for supervised VPCL
model, collate_fn = transtab.build_contrastive_learner(
    cat_cols, num_cols, bin_cols,
    supervised=True, # if take supervised CL
    num_partition=4, # num of column partitions for pos/neg sampling
    overlap_ratio=0.5, # specify the overlap ratio of column partitions during the CL
)

```

--------------------------------

### Get Evaluation Metric Function

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/evaluator.html

Retrieves the appropriate evaluation function based on the metric name. Supported metrics include 'acc', 'auc', and 'mse'.

```python
def get_eval_metric_fn(eval_metric):
    fn_dict = {
        'acc': acc_fn,
        'auc': auc_fn,
        'mse': mse_fn,
        'val_loss': None,
    }
    return fn_dict[eval_metric]

```

--------------------------------

### Loading a Custom Dataset with Transtab

Source: https://transtab.readthedocs.io/en/latest/data_preparation.html

Demonstrates how to load a custom dataset from a specified directory using the transtab library.

```python
transtab.load_data('./data/dataset1')
```

--------------------------------

### Load Data and Train Classifier

Source: https://transtab.readthedocs.io/en/latest/example_transfer.html

Load a dataset and train a vanilla supervised classifier using transtab. Ensure the 'transtab' library is imported.

```python
import transtab

# load a dataset and start vanilla supervised training
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols     = transtab.load_data('credit-g')

# build transtab classifier model
model = transtab.build_classifier(cat_cols, num_cols, bin_cols)

# start training
training_arguments = {
    'num_epoch':50,
    'eval_metric':'val_loss',
    'eval_less_is_better':True,
    'output_dir':'./checkpoint'
    }
transtab.train(model, trainset, valset, **training_arguments)
```

--------------------------------

### TransTabProjectionHead Initialization

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Initializes a projection head for TransTab models, used for mapping hidden dimensions to projection dimensions.

```python
class TransTabProjectionHead(nn.Module):
    def __init__(self, 
        hidden_dim=128,
        projection_dim=128):
        super().__init__()
```

--------------------------------

### Load and Preprocess Data from Local Files

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/dataset.html

Loads data from local files, identifies column types (numerical, binary, categorical), and applies preprocessing steps like filling missing values and scaling. It also handles updates from a dataset configuration.

```python
bnfile = os.path.join(dataname, 'binary_feature.txt')
        if os.path.exists(bnfile):
            with open(bnfile,'r') as f: bin_cols = [x.strip().lower() for x in f.readlines()]
        else:
            bin_cols = []
        cat_cols = [col for col in all_cols if col not in num_cols and col not in bin_cols]

        # update cols by loading dataset_config
        if dataset_config is not None:
            if 'columns' in dataset_config:
                new_cols = dataset_config['columns']
                X.columns = new_cols

            if 'bin' in dataset_config:
                bin_cols = dataset_config['bin']
            
            if 'cat' in dataset_config:
                cat_cols = dataset_config['cat']

            if 'num' in dataset_config:
                num_cols = dataset_config['num']
        
    else:
        dataset = openml.datasets.get_dataset(dataname)
        X,y,categorical_indicator, attribute_names = dataset.get_data(dataset_format='dataframe', target=dataset.default_target_attribute)
        
        if isinstance(dataname, int):
            openml_list = openml.datasets.list_datasets(output_format="dataframe")  # returns a dict
            dataname = openml_list.loc[openml_list.did == dataname].name.values[0]
        else:
            openml_list = openml.datasets.list_datasets(output_format="dataframe")  # returns a dict
            print(f'openml data index: {openml_list.loc[openml_list.name == dataname].index[0]}')
        
        print(f'load data from {dataname}')

        # drop cols which only have one unique value
        drop_cols = [col for col in attribute_names if X[col].nunique()<=1]

        all_cols = np.array(attribute_names)
        categorical_indicator = np.array(categorical_indicator)
        cat_cols = [col for col in all_cols[categorical_indicator] if col not in drop_cols]
        num_cols = [col for col in all_cols[~categorical_indicator] if col not in drop_cols]
        all_cols = [col for col in all_cols if col not in drop_cols]
        
        if dataset_config is not None:
            if 'bin' in dataset_config: bin_cols = [c for c in cat_cols if c in dataset_config['bin']]
        else: bin_cols = []
        cat_cols = [c for c in cat_cols if c not in bin_cols]

        # encode target label
        y = LabelEncoder().fit_transform(y.values)
        y = pd.Series(y,index=X.index)

    # start processing features
    # process num
    if len(num_cols) > 0:
        for col in num_cols: X[col].fillna(X[col].mode()[0], inplace=True)
        X[num_cols] = MinMaxScaler().fit_transform(X[num_cols])

    if len(cat_cols) > 0:
        for col in cat_cols: X[col].fillna(X[col].mode()[0], inplace=True)
        # process cate
        if encode_cat:
            X[cat_cols] = OrdinalEncoder().fit_transform(X[cat_cols])
        else:
            X[cat_cols] = X[cat_cols].astype(str)

    if len(bin_cols) > 0:
        for col in bin_cols: X[col].fillna(X[col].mode()[0], inplace=True)
        if 'binary_indicator' in dataset_config:
            X[bin_cols] = X[bin_cols].astype(str).applymap(lambda x: 1 if x.lower() in dataset_config['binary_indicator'] else 0).values
        else:
            X[bin_cols] = X[bin_cols].astype(str).applymap(lambda x: 1 if x.lower() in ['yes','true','1','t'] else 0).values        
        
        # if no dataset_config given, keep its original format
        # raise warning if there is not only 0/1 in the binary columns
        if (~X[bin_cols].isin([0,1])).any().any():
            raise ValueError(f'binary columns {bin_cols} contains values other than 0/1.')

    
    X = X[bin_cols + num_cols + cat_cols]

    # rename column names if is given
    if dataset_config is not None:
        data_config = dataset_config
        if 'columns' in data_config:
            new_cols = data_config['columns']
            X.columns = new_cols
            attribute_names = new_cols

        if 'bin' in data_config:
            bin_cols = data_config['bin']
        
        if 'cat' in data_config:
            cat_cols = data_config['cat']

        if 'num' in data_config:
            num_cols = data_config['num']


    # split train/val/test
    data_split_idx = None
    if dataset_config is not None:
        data_split_idx = dataset_config.get('data_split_idx', None)

    if data_split_idx is not None:
        train_idx = data_split_idx.get('train', None)
        val_idx = data_split_idx.get('val', None)
        test_idx = data_split_idx.get('test', None)

        if train_idx is None or test_idx is None:
            raise ValueError('train/test split indices must be provided together')
    
        else:
            train_dataset = X.iloc[train_idx]
            y_train = y[train_idx]
            test_dataset = X.iloc[test_idx]
            y_test = y[test_idx]
            if val_idx is not None:
                val_dataset = X.iloc[val_idx]
                y_val = y[val_idx]
            else:
                val_dataset = None

```

--------------------------------

### TransTabForCL Initialization

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Initializes the TransTabForCL model, setting up parameters for contrastive learning, including feature extraction, transformer layers, and projection heads. It includes assertions for valid input parameters like num_partition and overlap_ratio.

```python
def __init__(self,
        categorical_columns=None,
        numerical_columns=None,
        binary_columns=None,
        feature_extractor=None,
        hidden_dim=128,
        num_layer=2,
        num_attention_head=8,
        hidden_dropout_prob=0,
        ffn_dim=256,
        projection_dim=128,
        overlap_ratio=0.1,
        num_partition=2,
        supervised=True,
        temperature=10,
        base_temperature=10,
        activation='relu',
        device='cuda:0',
        **kwargs,
        ) -> None:
        super().__init__(
            categorical_columns=categorical_columns,
            numerical_columns=numerical_columns,
            binary_columns=binary_columns,
            feature_extractor=feature_extractor,
            hidden_dim=hidden_dim,
            num_layer=num_layer,
            num_attention_head=num_attention_head,
            hidden_dropout_prob=hidden_dropout_prob,
            ffn_dim=ffn_dim,
            activation=activation,
            device=device,
            **kwargs,
            )
        assert num_partition > 0, f'number of contrastive subsets must be greater than 0, got {num_partition}'
        assert isinstance(num_partition,int), f'number of constrative subsets must be int, got {type(num_partition)}'
        assert overlap_ratio >= 0 and overlap_ratio < 1, f'overlap_ratio must be in [0, 1), got {overlap_ratio}'
        self.projection_head = TransTabProjectionHead(hidden_dim, projection_dim)
        self.cross_entropy_loss = nn.CrossEntropyLoss()
        self.temperature = temperature
        self.base_temperature = base_temperature
        self.num_partition = num_partition
        self.overlap_ratio = overlap_ratio
        self.supervised = supervised
        self.device = device
        self.to(device)
```

--------------------------------

### Load Multiple Datasets from Local Directories

Source: https://transtab.readthedocs.io/en/latest/transtab.load_data.html

Loads multiple datasets from a list of local directory paths. Returns all dataset splits and column type lists.

```python
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \
    = transtab.load_data(['./data/credit-g','./data/credit-approval'])
```

--------------------------------

### transtab.build_contrastive_learner

Source: https://transtab.readthedocs.io/en/latest/transtab.build_contrastive_learner.html

Builds a contrastive learner for pretraining based on TransTab. It can handle categorical, numerical, and binary columns, with options for supervised or self-supervised learning and various transformer configurations.

```APIDOC
## transtab.build_contrastive_learner

### Description
Builds a contrastive learner for pretraining based on TransTab. If no categorical, numerical, or binary columns are specified, the model defaults to treating all columns as categorical, which may impact performance. Duplicate column assignments can be handled by setting `ignore_duplicate_cols=True`.

### Method
`build_contrastive_learner`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **categorical_columns** (list) – A list of categorical feature names.
- **numerical_columns** (list) – A list of numerical feature names.
- **binary_columns** (list) – A list of binary feature names (e.g., 'yes'/'no', 'true'/'false', 0/1).
- **feature_extractor** (TransTabFeatureExtractor) – A feature extractor to tokenize input tables. If not provided, the model builds its own.
- **hidden_dim** (int) – The dimension of hidden embeddings. Defaults to 128.
- **num_layer** (int) – The number of transformer layers in the encoder. Defaults to 2.
- **num_attention_head** (int) – The number of heads in the multi-head self-attention layer. Defaults to 8.
- **hidden_dropout_prob** (float) – The dropout ratio in the transformer encoder. Defaults to 0.
- **ffn_dim** (int) – The dimension of the feed-forward layer in the transformer layer. Defaults to 256.
- **projection_dim** (int) – The dimension of the projection head on top of the encoder. Defaults to 128.
- **overlap_ratio** (float) – The overlap ratio of columns across different partitions for vertical-partition contrastive learning. Defaults to 0.5.
- **num_partition** (int) – The number of partitions for vertical-partition contrastive learning. Defaults to 3.
- **supervised** (bool) – Whether to use supervised VPCL (True) or self-supervised VPCL (False). Defaults to True.
- **temperature** (float) – Temperature used for computing logits in contrastive learning.
- **base_temperature** (float) – Base temperature used for normalizing the temperature.
- **activation** (str) – The name of the activation function to use (e.g., "relu", "gelu", "selu", "leakyrelu"). Defaults to "relu".
- **device** (str) – The device to use ('cpu' or 'cuda:0'). Defaults to 'cuda:0'.
- **checkpoint** (str) – The directory of the pretrained transtab model.
- **ignore_duplicate_cols** (bool) – If True, ignores duplicate column assignments across feature types. Defaults to True.

### Request Example
```python
# Example usage (assuming necessary imports and data)
# model = transtab.build_contrastive_learner(
#     categorical_columns=['col1', 'col2'],
#     numerical_columns=['col3'],
#     supervised=False,
#     projection_dim=256
# )
```

### Response
#### Success Response
Returns a TransTabForCL model.

#### Response Example
```json
{
  "model": "TransTabForCL"
}
```
```

--------------------------------

### Load Dataset with Custom Configuration

Source: https://transtab.readthedocs.io/en/latest/transtab.load_data.html

Loads a dataset from OpenML and applies a custom configuration to manipulate columns and their types. This overrides default loading behavior.

```python
dataset_config = {
    'credit-g':{
        'columns':['a','b','c'], 
        'cat':['a'], 
        'bin':['b'], 
        'num':['c']}
        }

allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \
    = transtab.load_data('credit-g', dataset_config=dataset_config)
```

--------------------------------

### Load Dataset from Local Directory

Source: https://transtab.readthedocs.io/en/latest/transtab.load_data.html

Loads a dataset from a specified local directory path. Returns all dataset splits and column type lists.

```python
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \
    = transtab.load_data('./data/credit-g')
```

--------------------------------

### TransTabModel Constructor

Source: https://transtab.readthedocs.io/en/latest/transtab.basemodel.html

Initializes the TransTabModel. This is the base model for various downstream tasks. It accepts configurations for categorical, numerical, and binary columns, as well as parameters for the transformer encoder architecture and device.

```APIDOC
## TransTabModel Constructor

### Description
Initializes the TransTabModel, a base model for downstream tasks like contrastive learning and binary classification. It configures the model with specified column types and transformer architecture parameters.

### Parameters
* **categorical_columns** (list) - a list of categorical feature names.
* **numerical_columns** (list) - a list of numerical feature names.
* **binary_columns** (list) - a list of binary feature names (e.g., 'yes'/'no', 'true'/'false', 0/1).
* **feature_extractor** (TransTabFeatureExtractor) - An optional feature extractor to tokenize input tables. If not provided, the model builds its own.
* **hidden_dim** (int) - The dimension of hidden embeddings. Defaults to 128.
* **num_layer** (int) - The number of transformer layers in the encoder. Defaults to 2.
* **num_attention_head** (int) - The number of heads in the multi-head self-attention layer. Defaults to 8.
* **hidden_dropout_prob** (float) - The dropout ratio in the transformer encoder. Defaults to 0.1.
* **ffn_dim** (int) - The dimension of the feed-forward layer in the transformer layer. Defaults to 256.
* **activation** (str) - The activation function name. Supports 'relu', 'gelu', 'selu', 'leakyrelu'. Defaults to 'relu'.
* **device** (str) - The device to use ('cpu' or 'cuda:0'). Defaults to 'cuda:0'.
* **kwargs** - Additional keyword arguments.
```

--------------------------------

### TransTabInputEncoder Initialization

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Initializes the TransTabInputEncoder with feature extractors, processors, and device configuration. The model is automatically moved to the specified device.

```python
class TransTabInputEncoder(nn.Module):
    def __init__(self, 
        feature_extractor,
        feature_processor,
        device='cuda:0',
        ):
        super().__init__()
        self.feature_extractor = feature_extractor
        self.feature_processor = feature_processor
        self.device = device
        self.to(device)
```

--------------------------------

### Initialize TransTabFeatureProcessor

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Initializes the TransTabFeatureProcessor with configurations for vocabulary size, hidden dimensions, dropout, padding token ID, and device. It sets up word, numerical, and alignment embedding layers.

```python
class TransTabFeatureProcessor(nn.Module):
    r'''
    Process inputs from feature extractor to map them to embeddings.
    '''
    def __init__(self,
        vocab_size=None,
        hidden_dim=128,
        hidden_dropout_prob=0,
        pad_token_id=0,
        device='cuda:0',
        ) -> None:
        '''args:
categorical_columns: a list of categories feature names
numerical_columns: a list of numerical feature names
binary_columns: a list of yes or no feature names, accept binary indicators like
    (yes,no); (true,false); (0,1).
        '''
        super().__init__()
        self.word_embedding = TransTabWordEmbedding(
            vocab_size=vocab_size,
            hidden_dim=hidden_dim,
            hidden_dropout_prob=hidden_dropout_prob,
            padding_idx=pad_token_id
            )
        self.num_embedding = TransTabNumEmbedding(hidden_dim)
        self.align_layer = nn.Linear(hidden_dim, hidden_dim, bias=False)
        self.device = device
```

--------------------------------

### Load Pretrained Model and Finetune Classifier

Source: https://transtab.readthedocs.io/en/latest/example_pretrain.html

Loads a pretrained model from a specified checkpoint directory and finetunes it on a target dataset. The model's column dictionaries (categorical, numerical, binary) are updated after loading.

```python
# load the pretrained model and finetune on a target dataset
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols     = transtab.load_data('credit-approval')

# build transtab classifier model, and load from the pretrained dir
model = transtab.build_classifier(checkpoint='./checkpoint')

# update model's categorical/numerical/binary column dict
model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols})

```

--------------------------------

### Build TransTab Contrastive Learner

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/transtab.html

Builds a contrastive learner for pretraining TransTab models. Handles column type assignments and potential duplicate column issues. Supports supervised and unsupervised contrastive learning.

```python
def build_contrastive_learner(
    categorical_columns=None,
    numerical_columns=None,
    binary_columns=None,
    projection_dim=128,
    num_partition=3,
    overlap_ratio=0.5,
    supervised=True,
    hidden_dim=128,
    num_layer=2,
    num_attention_head=8,
    hidden_dropout_prob=0,
    ffn_dim=256,
    activation='relu',
    device='cuda:0',
    checkpoint=None,
    ignore_duplicate_cols=True,
    **kwargs,
    ):
    '''Build a contrastive learner for pretraining based on TransTab.
    If no cat/num/bin specified, the model takes ALL as categorical columns,
    which may undermine the performance significantly.

    If there is one column assigned to more than one type, e.g., the feature age is both nominated
    as categorical and binary columns, the model will raise errors. set ``ignore_duplicate_cols=True`` to avoid this error as 
    the model will ignore this duplicate feature.

    Parameters
    ----------
    categorical_columns: list 
        a list of categorical feature names.

    numerical_columns: list
        a list of numerical feature names.

    binary_columns: list
        a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1).
    
    feature_extractor: TransTabFeatureExtractor
        a feature extractor to tokenize the input tables. if not passed the model will build itself.

    hidden_dim: int
        the dimension of hidden embeddings.
    
    num_layer: int
        the number of transformer layers used in the encoder.
    
    num_attention_head: int
        the numebr of heads of multihead self-attention layer in the transformers.

    hidden_dropout_prob: float
        the dropout ratio in the transformer encoder.

    ffn_dim: int
        the dimension of feed-forward layer in the transformer layer.
    
    projection_dim: int
        the dimension of projection head on the top of encoder.
    
    overlap_ratio: float
        the overlap ratio of columns of different partitions when doing subsetting.
    
    num_partition: int
        the number of partitions made for vertical-partition contrastive learning.

    supervised: bool
    '''
    pass
```

--------------------------------

### TransTab Base Model Initialization

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Initializes the base TransTab model, setting up feature extraction, processing, input encoding, transformer encoder, and CLS token. It configures the model based on provided column types and dimensions.

```python
class TransTabModel(nn.Module):
    '''The base transtab model for downstream tasks like contrastive learning, binary classification, etc.
    All models subclass this basemodel and usually rewrite the ``forward`` function. Refer to the source code of
    :class:`transtab.modeling_transtab.TransTabClassifier` or :class:`transtab.modeling_transtab.TransTabForCL` for the implementation details.

    Parameters
    ----------
    categorical_columns: list
        a list of categorical feature names.

    numerical_columns: list
        a list of numerical feature names.

    binary_columns: list
        a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1).

    feature_extractor: TransTabFeatureExtractor
        a feature extractor to tokenize the input tables. if not passed the model will build itself.

    hidden_dim: int
        the dimension of hidden embeddings.

    num_layer: int
        the number of transformer layers used in the encoder.

    num_attention_head: int
        the numebr of heads of multihead self-attention layer in the transformers.

    hidden_dropout_prob: float
        the dropout ratio in the transformer encoder.

    ffn_dim: int
        the dimension of feed-forward layer in the transformer layer.

    activation: str
        the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``.

    device: str
        the device, ``"cpu"`` or ``"cuda:0"``.

    Returns
    -------
    A TransTabModel model.

    '''
    def __init__(self,
        categorical_columns=None,
        numerical_columns=None,
        binary_columns=None,
        feature_extractor=None,
        hidden_dim=128,
        num_layer=2,
        num_attention_head=8,
        hidden_dropout_prob=0.1,
        ffn_dim=256,
        activation='relu',
        device='cuda:0',
        **kwargs,
        ) -> None:

        super().__init__()
        self.categorical_columns=categorical_columns
        self.numerical_columns=numerical_columns
        self.binary_columns=binary_columns
        if categorical_columns is not None:
            self.categorical_columns = list(set(categorical_columns))
        if numerical_columns is not None:
            self.numerical_columns = list(set(numerical_columns))
        if binary_columns is not None:
            self.binary_columns = list(set(binary_columns))

        if feature_extractor is None:
            feature_extractor = TransTabFeatureExtractor(
                categorical_columns=self.categorical_columns,
                numerical_columns=self.numerical_columns,
                binary_columns=self.binary_columns,
                **kwargs,
            )

        feature_processor = TransTabFeatureProcessor(
            vocab_size=feature_extractor.vocab_size,
            pad_token_id=feature_extractor.pad_token_id,
            hidden_dim=hidden_dim,
            hidden_dropout_prob=hidden_dropout_prob,
            device=device,
            )
        
        self.input_encoder = TransTabInputEncoder(
            feature_extractor=feature_extractor,
            feature_processor=feature_processor,
            device=device,
            )

        self.encoder = TransTabEncoder(
            hidden_dim=hidden_dim,
            num_layer=num_layer,
            num_attention_head=num_attention_head,
            hidden_dropout_prob=hidden_dropout_prob,
            ffn_dim=ffn_dim,
            activation=activation,
            )

        self.cls_token = TransTabCLSToken(hidden_dim=hidden_dim)
        self.device = device
        self.to(device)
```

--------------------------------

### Load TransTab Feature Extractor Configuration

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Loads the tokenizer and column type configurations (categorical, binary, numerical) from a specified local directory. Initializes the tokenizer and sets the column type attributes.

```python
tokenizer_path = os.path.join(path, constants.TOKENIZER_DIR)
        coltype_path = os.path.join(path, constants.EXTRACTOR_STATE_NAME)

        self.tokenizer = BertTokenizerFast.from_pretrained(tokenizer_path)
        with open(coltype_path, 'r', encoding='utf-8') as f:
            col_type_dict = json.loads(f.read())

        self.categorical_columns = col_type_dict['categorical']
        self.numerical_columns = col_type_dict['numerical']
        self.binary_columns = col_type_dict['binary']
        logger.info(f'load feature extractor from {coltype_path}')
```

--------------------------------

### Load Pretrained Model and Update Columns

Source: https://transtab.readthedocs.io/en/latest/example_transfer.html

Load a pretrained model from a specified directory and update its column definitions. This is used for finetuning on a new dataset.

```python
# now let's load another data and try to leverage the pretrained model for finetuning
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols     = transtab.load_data('credit-approval')

# load the pretrained model
model.load('./checkpoint')

# update model's categorical/numerical/binary column dict
model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols})
```

--------------------------------

### TransTabClassifier Initialization

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Initializes the TransTabClassifier, setting up the feature extractor, transformer encoder, and a linear classifier. It configures the appropriate loss function based on the number of output classes.

```python
class TransTabClassifier(TransTabModel):
    '''The classifier model subclass from :class:`transtab.modeling_transtab.TransTabModel`.

    Parameters
    ----------
    categorical_columns: list
        a list of categorical feature names.

    numerical_columns: list
        a list of numerical feature names.

    binary_columns: list
        a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1).

    feature_extractor: TransTabFeatureExtractor
        a feature extractor to tokenize the input tables. if not passed the model will build itself.

    num_class: int
        number of output classes to be predicted.

    hidden_dim: int
        the dimension of hidden embeddings.

    num_layer: int
        the number of transformer layers used in the encoder.

    num_attention_head: int
        the numebr of heads of multihead self-attention layer in the transformers.

    hidden_dropout_prob: float
        the dropout ratio in the transformer encoder.

    ffn_dim: int
        the dimension of feed-forward layer in the transformer layer.

    activation: str
        the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``.

    device: str
        the device, ``"cpu"`` or ``"cuda:0"``.

    Returns
    -------
    A TransTabClassifier model.

    '''
    def __init__(
        self,
        categorical_columns=None,
        numerical_columns=None,
        binary_columns=None,
        feature_extractor=None,
        num_class=2,
        hidden_dim=128,
        num_layer=2,
        num_attention_head=8,
        hidden_dropout_prob=0,
        ffn_dim=256,
        activation='relu',
        device='cuda:0',
        **kwargs,
        ) -> None:
        super().__init__(
            categorical_columns=categorical_columns,
            numerical_columns=numerical_columns,
            binary_columns=binary_columns,
            feature_extractor=feature_extractor,
            hidden_dim=hidden_dim,
            num_layer=num_layer,
            num_attention_head=num_attention_head,
            hidden_dropout_prob=hidden_dropout_prob,
            ffn_dim=ffn_dim,
            activation=activation,
            device=device,
            **kwargs,
        )
        self.num_class = num_class
        self.clf = TransTabLinearClassifier(num_class=num_class, hidden_dim=hidden_dim)
        if self.num_class > 2:
            self.loss_fn = nn.CrossEntropyLoss(reduction='none')
        else:
            self.loss_fn = nn.BCEWithLogitsLoss(reduction='none')
        self.to(device)
```

--------------------------------

### Load Data and Build Classifier

Source: https://transtab.readthedocs.io/en/latest/fast_train.html

Loads multiple datasets and builds a TransTab classifier model. Ensure datasets are compatible in terms of label classes for supervised learning.

```python
import transtab

# load multiple datasets by passing a list of data names
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \
    = transtab.load_data(['credit-g','credit-approval'])

# build transtab classifier model
model = transtab.build_classifier(cat_cols, num_cols, bin_cols)

# specify training arguments, take validation loss for early stopping
training_arguments = {
    'num_epoch':5,
    'eval_metric':'val_loss',
    'eval_less_is_better':True,
    'output_dir':'./checkpoint'
    }
```

--------------------------------

### Build and Use TransTab Feature Extractor

Source: https://transtab.readthedocs.io/en/latest/transtab.build_extractor.html

This snippet demonstrates how to build a feature extractor by specifying categorical and numerical columns, create a sample DataFrame, and then use the extractor to process the DataFrame. It also shows the expected output format.

```python
# build the feature extractor
extractor = transtab.build_extractor(categorical_columns=['gender'], numerical_columns=['age'])

# build a table for inputs
df = pd.DataFrame({'age':[1,2], 'gender':['male','female']})

# extract the outputs
outputs = extractor(df)

print(outputs)

'''
    {
    'x_num': tensor([[1.],[2.]], dtype=torch.float64),
    'num_col_input_ids': tensor([[2287]]),
    'x_cat_input_ids': tensor([[5907, 3287], [5907, 2931]]),
    'x_bin_input_ids': None,
    'num_att_mask': tensor([[1]]),
    'cat_att_mask': tensor([[1, 1], [1, 1]])
    }
'''

```

--------------------------------

### TransTabModel.load

Source: https://transtab.readthedocs.io/en/latest/transtab.basemodel.html

Loads the model's state dictionary and feature extractor configuration from a specified directory.

```APIDOC
## TransTabModel.load

### Description
Loads the model's state dictionary and feature extractor configuration from a given checkpoint directory.

### Parameters
* **ckpt_dir** (str) - The directory path from which to load the model checkpoint.

### Returns
None
```

--------------------------------

### Load Best Checkpoint and Predict

Source: https://transtab.readthedocs.io/en/latest/fast_train.html

Loads the best model checkpoint based on validation loss and makes predictions on test data. Ensure the output directory is correctly specified.

```python
model.load('./checkpoint')

x_test, y_test = testset[0]

ypred = transtab.predict(x_test)
```

--------------------------------

### Load Model State and Feature Extractor Configuration

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Loads the model's state dictionary and feature extractor configuration from a specified directory. It logs any missing or unexpected keys during the state dict loading and then loads the feature extractor's state.

```python
    def load(self, ckpt_dir):
        '''Load the model state_dict and feature_extractor configuration
        from the ``ckpt_dir``.

        Parameters
        ----------
        ckpt_dir: str
            the directory path to load.

        Returns
        -------
        None

        '''
        # load model weight state dict
        model_name = os.path.join(ckpt_dir, constants.WEIGHTS_NAME)
        state_dict = torch.load(model_name, map_location='cpu')
        missing_keys, unexpected_keys = self.load_state_dict(state_dict, strict=False)
        logger.info(f'missing keys: {missing_keys}')
        logger.info(f'unexpected keys: {unexpected_keys}')
        logger.info(f'load model from {ckpt_dir}')

        # load feature extractor
        self.input_encoder.feature_extractor.load(os.path.join(ckpt_dir, constants.EXTRACTOR_STATE_DIR))
        self.binary_columns = self.input_encoder.feature_extractor.binary_columns
        self.categorical_columns = self.input_encoder.feature_extractor.categorical_columns
        self.numerical_columns = self.input_encoder.feature_extractor.numerical_columns

```

--------------------------------

### Initialize TransTabForCL Model and Collator

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/transtab.html

Creates a TransTabForCL model and its corresponding collator function for contrastive learning. Optionally loads a pre-trained checkpoint for both the model and the feature extractor.

```python
model = TransTabForCL(
        categorical_columns = categorical_columns,
        numerical_columns = numerical_columns,
        binary_columns = binary_columns,
        num_partition= num_partition,
        hidden_dim=hidden_dim,
        num_layer=num_layer,
        num_attention_head=num_attention_head,
        hidden_dropout_prob=hidden_dropout_prob,
        supervised=supervised,
        ffn_dim=ffn_dim,
        projection_dim=projection_dim,
        overlap_ratio=overlap_ratio,
        activation=activation,
        device=device,
    )
if checkpoint is not None:
    model.load(checkpoint)

# build collate function for contrastive learning
collate_fn = TransTabCollatorForCL(
    categorical_columns=categorical_columns,
    numerical_columns=numerical_columns,
    binary_columns=binary_columns,
    overlap_ratio=overlap_ratio,
    num_partition=num_partition,
    ignore_duplicate_cols=ignore_duplicate_cols
)
if checkpoint is not None:
    collate_fn.feature_extractor.load(os.path.join(checkpoint, constants.EXTRACTOR_STATE_DIR))

return model, collate_fn
```

--------------------------------

### TransTabForCL Model Creation and Loading

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/transtab.html

This function creates and optionally loads a TransTabForCL model. It configures the model architecture based on specified column types and dimensions, and can load pre-trained weights from a given checkpoint directory. It also prepares a corresponding collate function for contrastive learning.

```APIDOC
## TransTabForCL Model Creation and Loading

### Description
Creates and optionally loads a TransTabForCL model. Configures the model architecture and prepares a collate function for contrastive learning.

### Parameters

#### Categorical Columns
- **categorical_columns** (list) - List of categorical column names.

#### Numerical Columns
- **numerical_columns** (list) - List of numerical column names.

#### Binary Columns
- **binary_columns** (list) - List of binary column names.

#### Model Architecture Parameters
- **num_partition** (int) - Number of partitions for the model.
- **hidden_dim** (int) - Dimension of the hidden layers.
- **num_layer** (int) - Number of layers in the model.
- **num_attention_head** (int) - Number of attention heads.
- **hidden_dropout_prob** (float) - Dropout probability for hidden layers.
- **ffn_dim** (int) - Dimension of the feed-forward network.
- **projection_dim** (int) - Dimension of the projection layer.
- **overlap_ratio** (float) - Overlap ratio for partitions.
- **activation** (str) - Activation function to use (e.g., "relu", "gelu").
- **device** (str) - Device to run the model on (e.g., "cpu", "cuda:0").

#### Training/Learning Parameters
- **supervised** (bool) - Whether to use supervised learning.
- **temperature** (float) - Temperature for contrastive learning logits.
- **base_temperature** (float) - Base temperature for normalization.

#### Model Loading and Configuration
- **checkpoint** (str) - Path to the directory containing the pre-trained model checkpoint.
- **ignore_duplicate_cols** (bool) - If True, ignore duplicate column assignments.

### Returns
- **model**: A TransTabForCL model instance.
- **collate_fn**: A TransTabCollatorForCL instance.
```

--------------------------------

### load

Source: https://transtab.readthedocs.io/en/latest/_modules/transtab/modeling_transtab.html

Loads the model state dictionary and feature extractor configuration from a specified directory. This method is crucial for resuming training or deploying a pre-trained model.

```APIDOC
## load

### Description
Loads the model state dictionary and feature extractor configuration from the ``ckpt_dir``.

### Parameters
#### Path Parameters
- **ckpt_dir** (str) - Required - The directory path to load the model from.

### Returns
None
```

--------------------------------

### Load Dataset by Index from OpenML

Source: https://transtab.readthedocs.io/en/latest/transtab.load_data.html

Loads a dataset from OpenML using its numerical index. Returns all dataset splits and column type lists.

```python
allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \
    = transtab.load_data(31)
```