### Install Tab Transformer PyTorch

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Use pip to install the library.

```bash
pip install tab-transformer-pytorch
```

--------------------------------

### TabTransformer Training Example for Binary Classification

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Demonstrates a complete training loop for binary classification using TabTransformer. Includes data generation, DataLoader setup, model initialization, loss calculation, optimization, and inference. Requires PyTorch and tab_transformer_pytorch.

```python
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from tab_transformer_pytorch import TabTransformer

# Configuration
batch_size = 32
num_epochs = 10
learning_rate = 1e-4

# Generate synthetic tabular dataset
num_samples = 1000
num_categorical_features = 5
num_continuous_features = 10
categories = (10, 5, 6, 5, 8)  # Unique values per categorical feature

# Create synthetic data
x_categ = torch.stack([
    torch.randint(0, cat, (num_samples,)) 
    for cat in categories
], dim=1)
x_cont = torch.randn(num_samples, num_continuous_features)
y = torch.randint(0, 2, (num_samples, 1)).float()

# Create DataLoader
dataset = TensorDataset(x_categ, x_cont, y)
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

# Initialize model
model = TabTransformer(
    categories = categories,
    num_continuous = num_continuous_features,
    dim = 32,
    dim_out = 1,
    depth = 6,
    heads = 8,
    attn_dropout = 0.1,
    ff_dropout = 0.1,
    mlp_hidden_mults = (4, 2)
)

# Loss and optimizer
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

# Training loop
model.train()
for epoch in range(num_epochs):
    total_loss = 0
    for batch_categ, batch_cont, batch_y in dataloader:
        optimizer.zero_grad()

        # Forward pass
        logits = model(batch_categ, batch_cont)
        loss = criterion(logits, batch_y)

        # Backward pass
        loss.backward()
        optimizer.step()

        total_loss += loss.item()

    avg_loss = total_loss / len(dataloader)
    print(f"Epoch {epoch+1}/{num_epochs}, Loss: {avg_loss:.4f}")

# Inference
model.eval()
with torch.no_grad():
    test_categ = torch.randint(0, 5, (1, 5))
    test_cont = torch.randn(1, 10)
    prediction = torch.sigmoid(model(test_categ, test_cont))
    print(f"Prediction probability: {prediction.item():.4f}")
```

--------------------------------

### Install Tab Transformer

Source: https://github.com/lucidrains/tab-transformer-pytorch/blob/main/README.md

Install the package via pip.

```bash
$ pip install tab-transformer-pytorch
```

--------------------------------

### Get Predictions and Attention Weights

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Perform a forward pass requesting attention weights by setting `return_attn=True`. The attention tensor shape is (depth, batch, heads, num_categories, num_categories).

```python
# Prepare input
x_categ = torch.randint(0, 5, (1, 5))
x_cont = torch.randn(1, 10)

# Get predictions with attention weights
model.eval()
with torch.no_grad():
    pred, attns = model(x_categ, x_cont, return_attn=True)

print(f"Prediction: {pred.item():.4f}")
print(f"Attention tensor shape: {attns.shape}")
# Shape: (depth, batch, heads, num_categories, num_categories)

# Extract attention from last layer, first head
last_layer_attn = attns[-1, 0, 0]  # Shape: (5, 5)
print(f"Last layer attention matrix:\n{last_layer_attn}")

# Average attention across all heads and layers
avg_attn = attns.mean(dim=(0, 2))  # Average over depth and heads
print(f"Average attention per sample: {avg_attn.shape}")  # (batch, seq, seq)

# Feature importance based on attention (which features attend to which)
feature_importance = avg_attn[0].sum(dim=0)  # Sum attention received by each feature
print(f"Relative feature importance: {feature_importance}")
```

--------------------------------

### FTTransformer Forward Pass and Attention

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Performs a forward pass with the FTTransformer model to get predictions and optionally return attention weights for interpretability. Ensure input data matches model's expected feature types.

```python
# Prepare input data
# Categorical values: integers from 0 to (num_categories - 1) for each feature
x_categ = torch.randint(0, 5, (4, 5))    # Batch of 4, 5 categorical features
# Numerical values: float values (will be embedded)
x_numer = torch.randn(4, 10)             # Batch of 4, 10 continuous features

# Forward pass
pred = model(x_categ, x_numer)
print(f"Predictions shape: {pred.shape}")  # Output: torch.Size([4, 1])

# Forward pass with attention weights for interpretability
pred, attns = model(x_categ, x_numer, return_attn=True)
print(f"Attention shape: {attns.shape}")   # Output: torch.Size([6, 4, 8, 16, 16])
# Attention dimensions: (depth, batch, heads, seq_len+1, seq_len+1)
# seq_len+1 includes the CLS token
```

--------------------------------

### Initialize and Use Tab Transformer

Source: https://github.com/lucidrains/tab-transformer-pytorch/blob/main/README.md

Configure the TabTransformer model with categorical and continuous features, then perform a forward pass.

```python
import torch
import torch.nn as nn
from tab_transformer_pytorch import TabTransformer

cont_mean_std = torch.randn(10, 2)

model = TabTransformer(
    categories = (10, 5, 6, 5, 8),      # tuple containing the number of unique values within each category
    num_continuous = 10,                # number of continuous values
    dim = 32,                           # dimension, paper set at 32
    dim_out = 1,                        # binary prediction, but could be anything
    depth = 6,                          # depth, paper recommended 6
    heads = 8,                          # heads, paper recommends 8
    attn_dropout = 0.1,                 # post-attention dropout
    ff_dropout = 0.1,                   # feed forward dropout
    mlp_hidden_mults = (4, 2),          # relative multiples of each hidden dimension of the last mlp to logits
    mlp_act = nn.ReLU(),                # activation for final mlp, defaults to relu, but could be anything else (selu etc)
    continuous_mean_std = cont_mean_std # (optional) - normalize the continuous values before layer norm
)

x_categ = torch.randint(0, 5, (1, 5))     # category values, from 0 - max number of categories, in the order as passed into the constructor above
x_cont = torch.randn(1, 10)               # assume continuous values are already normalized individually

pred = model(x_categ, x_cont) # (1, 1)
```

--------------------------------

### Initialize TabTransformer for Regression

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Use this to initialize the TabTransformer model for regression tasks. Set `dim_out` to 1 for a single continuous output.

```python
import torch
import torch.nn as nn
from tab_transformer_pytorch import TabTransformer

# Initialize TabTransformer for regression
model = TabTransformer(
    categories = (10, 5, 6, 5, 8),
    num_continuous = 10,
    dim = 32,
    dim_out = 1,                         # Single continuous output
    depth = 6,
    heads = 8,
    attn_dropout = 0.1,
    ff_dropout = 0.1,
    mlp_hidden_mults = (4, 2),
    mlp_act = nn.ReLU()
)
```

--------------------------------

### Initialize and Use FT Transformer

Source: https://github.com/lucidrains/tab-transformer-pytorch/blob/main/README.md

Configure the FTTransformer model for tabular data comparison.

```python
import torch
from tab_transformer_pytorch import FTTransformer

model = FTTransformer(
    categories = (10, 5, 6, 5, 8),      # tuple containing the number of unique values within each category
    num_continuous = 10,                # number of continuous values
    dim = 32,                           # dimension, paper set at 32
    dim_out = 1,                        # binary prediction, but could be anything
    depth = 6,                          # depth, paper recommended 6
    heads = 8,                          # heads, paper recommends 8
    attn_dropout = 0.1,                 # post-attention dropout
    ff_dropout = 0.1                    # feed forward dropout
)

x_categ = torch.randint(0, 5, (1, 5))     # category values, from 0 - max number of categories, in the order as passed into the constructor above
x_numer = torch.randn(1, 10)              # numerical value

pred = model(x_categ, x_numer) # (1, 1)
```

--------------------------------

### Citations

Source: https://github.com/lucidrains/tab-transformer-pytorch/blob/main/README.md

BibTeX citations for the implemented architectures and related research.

```bibtex
@misc{huang2020tabtransformer,
    title   = {TabTransformer: Tabular Data Modeling Using Contextual Embeddings},
    author  = {Xin Huang and Ashish Khetan and Milan Cvitkovic and Zohar Karnin},
    year    = {2020},
    eprint  = {2012.06678},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}
```

```bibtex
@article{Gorishniy2021RevisitingDL,
    title   = {Revisiting Deep Learning Models for Tabular Data},
    author  = {Yu. V. Gorishniy and Ivan Rubachev and Valentin Khrulkov and Artem Babenko},
    journal = {ArXiv},
    year    = {2021},
    volume  = {abs/2106.11959}
}
```

```bibtex
@article{Zhu2024HyperConnections,
    title   = {Hyper-Connections},
    author  = {Defa Zhu and Hongzhi Huang and Zihao Huang and Yutao Zeng and Yunyao Mao and Banggu Wu and Qiyang Min and Xun Zhou},
    journal = {ArXiv},
    year    = {2024},
    volume  = {abs/2409.19606},
    url     = {https://api.semanticscholar.org/CorpusID:272987528}
}
```

--------------------------------

### Initialize TabTransformer for Attention Visualization

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Initialize TabTransformer with attention dropout set to 0.0 to ensure consistent attention weights for visualization. This is useful for interpretability.

```python
import torch
from tab_transformer_pytorch import TabTransformer

model = TabTransformer(
    categories = (10, 5, 6, 5, 8),
    num_continuous = 10,
    dim = 32,
    dim_out = 1,
    depth = 6,
    heads = 8,
    attn_dropout = 0.0,                  # Disable dropout for consistent attention
    ff_dropout = 0.0
)
```

--------------------------------

### Initialize FTTransformer Model

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Initializes the FTTransformer model with specified parameters for categorical and continuous features. Use this for setting up the model architecture.

```python
model = FTTransformer(
    categories = (10, 5, 6, 5, 8),      # Number of unique values per categorical feature
    num_continuous = 10,                 # Number of continuous features
    dim = 32,                            # Embedding dimension (paper recommends 32)
    dim_out = 1,                         # Output dimension (1 for binary classification)
    depth = 6,                           # Number of transformer layers (paper recommends 6)
    heads = 8,                           # Number of attention heads (paper recommends 8)
    dim_head = 16,                       # Dimension per attention head
    attn_dropout = 0.1,                  # Dropout after attention
    ff_dropout = 0.1,                    # Dropout in feed-forward layers
    num_special_tokens = 2,              # Number of special tokens (for missing values)
    num_residual_streams = 4             # Number of hyper-connection residual streams
)
```

--------------------------------

### Import FTTransformer

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Import the FTTransformer class from the library.

```python
import torch
from tab_transformer_pytorch import FTTransformer
```

--------------------------------

### Regression Task Forward Pass and Loss Calculation

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Demonstrates a forward pass for regression and calculates Mean Squared Error (MSE) and Mean Absolute Error (MAE) losses.

```python
# Prepare input data
x_categ = torch.randint(0, 5, (4, 5))
x_cont = torch.randn(4, 10)
targets = torch.randn(4, 1)              # Continuous target values

# Forward pass
predictions = model(x_categ, x_cont)
print(f"Predictions shape: {predictions.shape}")  # Output: torch.Size([4, 1])

# Calculate MSE loss for regression
criterion = nn.MSELoss()
loss = criterion(predictions, targets)
print(f"MSE Loss: {loss.item():.4f}")

# Calculate MAE
mae = torch.mean(torch.abs(predictions - targets))
print(f"MAE: {mae.item():.4f}")
```

--------------------------------

### Initialize FTTransformer for Multi-class Classification

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Configures and initializes the FTTransformer model for multi-class classification tasks. Adjust `dim_out` to match the number of classes.

```python
import torch
import torch.nn as nn
from tab_transformer_pytorch import FTTransformer

num_classes = 10

# Initialize FTTransformer for multi-class classification
model = FTTransformer(
    categories = (10, 5, 6, 5, 8),
    num_continuous = 10,
    dim = 64,                            # Larger dimension for complex task
    dim_out = num_classes,               # Output one logit per class
    depth = 6,
    heads = 8,
    attn_dropout = 0.1,
    ff_dropout = 0.1
)
```

--------------------------------

### Initialize TabTransformer with Special Tokens for Missing Values

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Initialize TabTransformer to handle missing categorical values using special tokens. Set `num_special_tokens` to the count of special tokens you intend to use (e.g., -1, -2).

```python
import torch
from tab_transformer_pytorch import TabTransformer

model = TabTransformer(
    categories = (10, 5, 6, 5, 8),
    num_continuous = 10,
    dim = 32,
    dim_out = 1,
    depth = 6,
    heads = 8,
    num_special_tokens = 2,              # Support for 2 special tokens (-1, -2)
    attn_dropout = 0.1,
    ff_dropout = 0.1
)
```

--------------------------------

### Forward Pass with Missing Values

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Input data with missing values represented by negative integers (-1 or -2). The model automatically handles these special tokens during the forward pass.

```python
# Create input with missing values (represented as -1)
x_categ = torch.tensor([
    [3, 2, -1, 4, 1],    # Third category is missing (-1)
    [5, -1, 3, 2, 0],    # Second category is missing (-1)
    [1, 4, 2, -2, 3],    # Fourth category uses special token -2
    [0, 1, 2, 3, 4]      # No missing values
])
x_cont = torch.randn(4, 10)

# Forward pass handles special tokens automatically
pred = model(x_categ, x_cont)
print(f"Predictions with missing values: {pred.shape}")  # Output: torch.Size([4, 1])
```

--------------------------------

### FTTransformer Forward Pass for Multi-class Classification

Source: https://context7.com/lucidrains/tab-transformer-pytorch/llms.txt

Performs a forward pass with the FTTransformer model for multi-class classification and calculates the cross-entropy loss. Ensure `targets` are class indices.

```python
# Prepare input data
x_categ = torch.randint(0, 5, (4, 5))
x_numer = torch.randn(4, 10)
targets = torch.randint(0, num_classes, (4,))

# Forward pass
logits = model(x_categ, x_numer)
print(f"Logits shape: {logits.shape}")     # Output: torch.Size([4, 10])

# Calculate loss with CrossEntropyLoss
criterion = nn.CrossEntropyLoss()
loss = criterion(logits, targets)
print(f"Loss: {loss.item():.4f}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.