### Install torchtuples

Source: https://github.com/havakv/torchtuples/blob/master/README.md

Installation commands for the torchtuples package using various package managers.

```bash
pip install torchtuples
```

```bash
conda install -c conda-forge torchtuples
```

```bash
pip install git+git://github.com/havakv/torchtuples.git
```

```bash
git clone https://github.com/havakv/torchtuples.git
cd torchtuples
python setup.py install
```

--------------------------------

### Setup for larger dataset experiments

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Initializes a larger dataset and sets training parameters including batch size and number of workers for subsequent performance tests.

```python
np.random.seed(123)
inp, tar, inp_tensor, tar_tensor = make_dataset(100000)
epochs = 10
batch_size = 256
num_workers = 0
```

--------------------------------

### Import necessary libraries

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Imports core libraries including numpy, torch, and torchtuples. Ensure these are installed before running.

```python
import numpy as np
import torch
import torchtuples
from torchtuples import Model, tuplefy
from torchtuples.practical import MLPVanilla
```

--------------------------------

### Training Output Log

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Example output log from the model training process, showing epoch number, time, and training/validation loss and accuracy.

```text
0: 	[0s / 0s],		train_loss: 0.3064, 	train_acc: 0.8833, 	val_loss: 0.2519, 	val_acc: 0.9087
1: 	[0s / 0s],		train_loss: 0.2406, 	train_acc: 0.9116, 	val_loss: 0.1735, 	val_acc: 0.9425
2: 	[0s / 0s],		train_loss: 0.2017, 	train_acc: 0.9316, 	val_loss: 0.1573, 	val_acc: 0.9488
3: 	[0s / 0s],		train_loss: 0.1991, 	train_acc: 0.9284, 	val_loss: 0.1535, 	val_acc: 0.9413
4: 	[0s / 0s],		train_loss: 0.1781, 	train_acc: 0.9361, 	val_loss: 0.1367, 	val_acc: 0.9513
5: 	[0s / 1s],		train_loss: 0.1628, 	train_acc: 0.9419, 	val_loss: 0.1354, 	val_acc: 0.9506
6: 	[0s / 1s],		train_loss: 0.1545, 	train_acc: 0.9452, 	val_loss: 0.1216, 	val_acc: 0.9531
7: 	[0s / 1s],		train_loss: 0.1701, 	train_acc: 0.9341, 	val_loss: 0.1371, 	val_acc: 0.9513
8: 	[0s / 1s],		train_loss: 0.1678, 	train_acc: 0.9394, 	val_loss: 0.1402, 	val_acc: 0.9469
9: 	[0s / 1s],		train_loss: 0.1562, 	train_acc: 0.9414, 	val_loss: 0.1234, 	val_acc: 0.9575
10: 	[0s / 1s],		train_loss: 0.1473, 	train_acc: 0.9447, 	val_loss: 0.1166, 	val_acc: 0.9556
11: 	[0s / 1s],		train_loss: 0.1426, 	train_acc: 0.9487, 	val_loss: 0.1145, 	val_acc: 0.9550
12: 	[0s / 1s],		train_loss: 0.1427, 	train_acc: 0.9505, 	val_loss: 0.1107, 	val_acc: 0.9575
13: 	[0s / 2s],		train_loss: 0.1306, 	train_acc: 0.9525, 	val_loss: 0.1101, 	val_acc: 0.9606
14: 	[0s / 2s],		train_loss: 0.1322, 	train_acc: 0.9503, 	val_loss: 0.1099, 	val_acc: 0.9600
15: 	[0s / 2s],		train_loss: 0.1396, 	train_acc: 0.9481, 	val_loss: 0.1197, 	val_acc: 0.9606
16: 	[0s / 2s],		train_loss: 0.1466, 	train_acc: 0.9470, 	val_loss: 0.1255, 	val_acc: 0.9556
17: 	[0s / 2s],		train_loss: 0.1501, 	train_acc: 0.9478, 	val_loss: 0.1287, 	val_acc: 0.9594
18: 	[0s / 3s],		train_loss: 0.1440, 	train_acc: 0.9463, 	val_loss: 0.1214, 	val_acc: 0.9563
19: 	[0s / 3s],		train_loss: 0.1408, 	train_acc: 0.9445, 	val_loss: 0.1263, 	val_acc: 0.9569
20: 	[0s / 3s],		train_loss: 0.1391, 	train_acc: 0.9497, 	val_loss: 0.1260, 	val_acc: 0.9513
21: 	[0s / 3s],		train_loss: 0.1395, 	train_acc: 0.9475, 	val_loss: 0.1417, 	val_acc: 0.9506
22: 	[0s / 3s],		train_loss: 0.1281, 	train_acc: 0.9547, 	val_loss: 0.1136, 	val_acc: 0.9581
23: 	[0s / 3s],		train_loss: 0.1245, 	train_acc: 0.9530, 	val_loss: 0.1233, 	val_acc: 0.9575
24: 	[0s / 3s],		train_loss: 0.1211, 	train_acc: 0.9544, 	val_loss: 0.1194, 	val_acc: 0.9594
25: 	[0s / 3s],		train_loss: 0.1212, 	train_acc: 0.9555, 	val_loss: 0.1154, 	val_acc: 0.9613
26: 	[0s / 4s],		train_loss: 0.1171, 	train_acc: 0.9544, 	val_loss: 0.1156, 	val_acc: 0.9600
27: 	[0s / 4s],		train_loss: 0.1155, 	train_acc: 0.9570, 	val_loss: 0.1168, 	val_acc: 0.9600
28: 	[0s / 4s],		train_loss: 0.1098, 	train_acc: 0.9592, 	val_loss: 0.1170, 	val_acc: 0.9606
29: 	[0s / 4s],		train_loss: 0.1107, 	train_acc: 0.9578, 	val_loss: 0.1180, 	val_acc: 0.9606
30: 	[0s / 4s],		train_loss: 0.1100, 	train_acc: 0.9584, 	val_loss: 0.1168, 	val_acc: 0.9600
31: 	[0s / 4s],		train_loss: 0.1260, 	train_acc: 0.9559, 	val_loss: 0.1296, 	val_acc: 0.9556
32: 	[0s / 4s],		train_loss: 0.1361, 	train_acc: 0.9473, 	val_loss: 0.1250, 	val_acc: 0.9581
33: 	[0s / 4s],		train_loss: 0.1291, 	train_acc: 0.9527, 	val_loss: 0.1234, 	val_acc: 0.9563
34: 	[0s / 4s],		train_loss: 0.1206, 	train_acc: 0.9530, 	val_loss: 0.1264, 	val_acc: 0.9563
35: 	[0s / 5s],		train_loss: 0.1276, 	train_acc: 0.9520, 	val_loss: 0.1237, 	val_acc: 0.9581
36: 	[0s / 5s],		train_loss: 0.1293, 	train_acc: 0.9481, 	val_loss: 0.1291, 	val_acc: 0.9575
37: 	[0s / 5s],		train_loss: 0.1227, 	train_acc: 0.9555, 	val_loss: 0.1234, 	val_acc: 0.9575
38: 	[0s / 5s],		train_loss: 0.1166, 	train_acc: 0.9572, 	val_loss: 0.1245, 	val_acc: 0.9581
39: 	[0s / 5s],		train_loss: 0.1211, 	train_acc: 0.9559, 	val_loss: 0.1246, 	val_acc: 0.9600
40: 	[0s / 5s],		train_loss: 0.1166, 	train_acc: 0.9566, 	val_loss: 0.1241, 	val_acc: 0.9613
```

--------------------------------

### Check torchtuples version

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Verifies the installed version of torchtuples.

```python
torchtuples.__version__
```

--------------------------------

### Check PyTorch version

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Verifies the installed version of PyTorch.

```python
torch.__version__
```

--------------------------------

### Create MLPVanilla with Batch Norm and Dropout

Source: https://context7.com/havakv/torchtuples/llms.txt

Constructs a multi-layer perceptron using MLPVanilla, which supports batch normalization, dropout, and customizable activation functions. This example shows creating an MLP with multiple hidden layers and then another with varied dropout rates per layer.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim
from torchtuples.practical import MLPVanilla, accuracy_binary

# Create MLP with multiple hidden layers
net = MLPVanilla(
    in_features=20,           # Input dimension
    num_nodes=[64, 32, 16],   # Hidden layer sizes
    out_features=1,           # Output dimension
    batch_norm=True,          # Use batch normalization
    dropout=0.1,              # Dropout rate (same for all layers)
    activation=nn.ReLU,       # Activation function
    output_activation=None,   # No activation on output (for logits)
    output_bias=True
)

print(net)
# MLPVanilla(
#   (net): Sequential(
#     (0): DenseVanillaBlock(...)
#     (1): DenseVanillaBlock(...)
#     (2): DenseVanillaBlock(...)
#     (3): Linear(in_features=16, out_features=1, bias=True)
#   )
# )

# Create model with custom metrics
model = Model(net, nn.BCEWithLogitsLoss(), optim.Adam(lr=0.001))

# Training data
x = np.random.randn(1000, 20).astype('float32')
y = np.random.randint(0, 2, (1000, 1)).astype('float32')

# Train with additional metrics
log = model.fit(
    x, y,
    batch_size=64,
    epochs=10,
    metrics={'accuracy': accuracy_binary}  # Track accuracy during training
)

# Different dropout per layer
net_varied = MLPVanilla(
    in_features=20,
    num_nodes=[64, 32],
    out_features=1,
    dropout=[0.2, 0.1]  # Different dropout for each layer
)
```

--------------------------------

### Import necessary libraries for torchtuples

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Imports core libraries including numpy, pandas, matplotlib, torch, and specific modules from torchtuples for model building and callbacks. Ensure these are installed before running.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import torch
from torchtuples import Model, optim
import torchtuples.callbacks as cb
from torchtuples.practical import MLPVanilla, accuracy_binary
```

--------------------------------

### Get Minimum Validation Loss

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Retrieves the minimum validation loss recorded during training from the log data.

```python
log.to_pandas().val_loss.min()
```

--------------------------------

### Initialize Model and Optimizer

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Instantiates the defined neural network, specifies the loss function (CrossEntropyLoss), and sets up the optimizer (SGD) with the defined learning rate and momentum.

```python
net = Net()
model = Model(net, nn.CrossEntropyLoss(), optim.SGD(lr, momentum))
```

--------------------------------

### Efficient Data Loading with DatasetTuple and DataLoaderBatch

Source: https://context7.com/havakv/torchtuples/llms.txt

Demonstrates creating optimized dataloaders for batch-based processing and handling input-only data.

```python
import torch
import numpy as np
from torchtuples import tuplefy, make_dataloader
from torchtuples.data import DatasetTuple, DataLoaderBatch, dataloader_input_only

# Create dataset from tuple data
x = torch.randn(1000, 10)
y = torch.randn(1000, 1)

dataset = DatasetTuple(x, y)
print(len(dataset))  # 1000

# Get batch by slicing
batch = dataset[:32]
print(batch[0].shape, batch[1].shape)  # (32, 10), (32, 1)

# Create optimized dataloader
dataloader = DataLoaderBatch(
    dataset,
    batch_size=64,
    shuffle=True,
    num_workers=0
)

# Iterate through batches
for batch_x, batch_y in dataloader:
    print(batch_x.shape, batch_y.shape)
    break  # (64, 10), (64, 1)

# Using make_dataloader helper
data = (np.random.randn(1000, 10).astype('float32'),
        np.random.randn(1000, 1).astype('float32'))

dl = make_dataloader(
    data,
    batch_size=64,
    shuffle=True,
    num_workers=0,
    to_tensor=True  # Converts numpy to torch tensors
)

# Create input-only dataloader for predictions
# (removes targets from batches)
dl_input = dataloader_input_only(dataloader)
for batch_x in dl_input:
    print(batch_x.shape)  # (64, 10) - no targets
    break
```

--------------------------------

### Training the model

Source: https://github.com/havakv/torchtuples/blob/master/examples/01_introduction.ipynb

Configuring the loss function and optimizer, then fitting the model to the training data.

```python
loss = torch.nn.BCEWithLogitsLoss()
optimizer = torchtuples.optim.Adam(lr=0.001)
model = Model(net, loss, optimizer)
```

```python
log = model.fit(x_train, y_train, batch_size=256, epochs=5, val_data=(x_val, y_val))
```

```python
log = model.fit(x_train, y_train, batch_size=256, epochs=5, val_data=(x_val, y_val))
```

--------------------------------

### Define Training Metrics

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Set up a dictionary of metrics to monitor during training. `accuracy_binary` is imported from `torchtuples.practical`.

```python
metrics = {'acc': accuracy_binary}
```

--------------------------------

### Train a PyTorch model with torchtuples

Source: https://context7.com/havakv/torchtuples/llms.txt

Initializes a model with a neural network, loss function, and optimizer, then executes the training loop using the fit method.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim

# Create training data
n_samples = 1000
x_train = np.random.randn(n_samples, 10).astype('float32')
y_train = (x_train.sum(axis=1, keepdims=True) > 0).astype('float32')

# Define a simple neural network
net = nn.Sequential(
    nn.Linear(10, 32),
    nn.ReLU(),
    nn.Linear(32, 1)
)

# Create model with loss function and optimizer
loss_fn = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(lr=0.001)
model = Model(net, loss_fn, optimizer)

# Train the model
log = model.fit(
    x_train,
    y_train,
    batch_size=64,
    epochs=10,
    verbose=True
)

# Plot training progress
log.to_pandas()  # Returns DataFrame with training metrics
# log.plot()     # Plot training curves
```

--------------------------------

### Set training parameters and generate data

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Configures the number of epochs, batch size, and workers for data loading, then generates the dataset using the `make_dataset` function.

```python
np.random.seed(123)
inp, tar, inp_tensor, tar_tensor = make_dataset(2000)
epochs = 50
batch_size = 256
num_workers = 0
```

--------------------------------

### Import dataset creation utility

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Imports the `make_classification` function from scikit-learn to generate synthetic datasets.

```python
from sklearn.datasets import make_classification # to create a data set
```

--------------------------------

### Import torchtuples components

Source: https://github.com/havakv/torchtuples/blob/master/README.md

Initial imports required to use the Model and optimizer classes.

```python
import torch
from torch import nn
from torchtuples import Model, optim
```

--------------------------------

### Initialize torchtuples Model with BCEWithLogitsLoss and AdamWR optimizer

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Creates a torchtuples Model instance using the defined neural network, binary cross-entropy loss, and the AdamWR optimizer. The optimizer is configured with decoupled weight decay and learning rate/cycle multipliers for cyclical learning rates.

```python
optimizer = optim.AdamWR(decoupled_weight_decay=0.01, cycle_multiplier=2, cycle_eta_multiplier=0.8)
model = Model(net, torch.nn.BCEWithLogitsLoss(), optimizer)
```

--------------------------------

### Configure Early Stopping Callback

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Initialize the `EarlyStoppingCycle` callback for cyclical optimizers. Set `patience` to 1 cycle and `min_cycles` to 6. For regular optimizers, use `EarlyStopping`.

```python
es = cb.EarlyStoppingCycle(patience=1, min_cycles=6)
```

--------------------------------

### Fit Model with Callbacks and Metrics

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Train the model using `model.fit`, specifying training and validation data, batch size, number of epochs, and providing the defined callbacks and metrics.

```python
log = model.fit(x_train, y_train, batch_size, epochs=200, val_data=val, callbacks=[es], metrics=metrics)
```

--------------------------------

### Train Model with Callbacks

Source: https://context7.com/havakv/torchtuples/llms.txt

Demonstrates training a model with early stopping and gradient clipping callbacks. Early stopping monitors a metric and stops training if no improvement is seen for a specified patience. Gradient clipping prevents exploding gradients by limiting the norm of the gradients.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim
from torchtuples.callbacks import EarlyStopping, ClipGradNorm

# Prepare data
x_train = np.random.randn(1000, 10).astype('float32')
y_train = np.random.randn(1000, 1).astype('float32')
x_val = np.random.randn(200, 10).astype('float32')
y_val = np.random.randn(200, 1).astype('float32')

net = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))
model = Model(net, nn.MSELoss(), optim.Adam(lr=0.001))

# Early stopping callback
early_stopping = EarlyStopping(
    metric='loss',           # Metric to monitor
    dataset='val',           # Monitor validation data
    minimize=True,           # Lower is better
    patience=5,              # Stop after 5 epochs without improvement
    min_delta=0.001,         # Minimum improvement threshold
    checkpoint_model=True,   # Save best weights
    load_best=True           # Load best weights after training
)

# Gradient clipping callback
grad_clip = ClipGradNorm(net, max_norm=1.0, norm_type=2)

# Train with callbacks
log = model.fit(
    x_train, y_train,
    batch_size=64,
    epochs=100,  # Will stop early
    val_data=(x_val, y_val),
    callbacks=[early_stopping, grad_clip]
)

print(f"Training stopped at epoch {len(log.epochs)}")

# Best weights callback (without early stopping)
best_weights = BestWeights(
    metric='loss',
    dataset='val',
    minimize=True,
    load_best=True,
    rm_file=True  # Remove checkpoint file after training
)
```

--------------------------------

### Visualize Sample Predictions

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Selects a random sample of 8 images from the test set, displays them using Matplotlib's `make_grid`, and shows the corresponding predicted class labels.

```python
images = next(iter(test_input))
idx = np.random.choice(test_input.batch_sampler.batch_size, 8)
sub = images[idx]
plt.figure(figsize=(10, 6))
_ = plt.imshow(torchvision.utils.make_grid(sub)[0].numpy(), cmap='gray')
preds[idx]
```

--------------------------------

### Save and Load Models

Source: https://context7.com/havakv/torchtuples/llms.txt

Methods for persisting model weights or entire network architectures.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim

# Create and train model
net = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))
model = Model(net, nn.MSELoss(), optim.Adam())

x = np.random.randn(500, 10).astype('float32')
y = np.random.randn(500, 1).astype('float32')
model.fit(x, y, batch_size=64, epochs=5)

# Save model weights only (recommended)
model.save_model_weights('model_weights.pt')

# Load weights into existing model
model.load_model_weights('model_weights.pt')

# Save entire network (architecture + weights)
model.save_net('full_model.pt')

# Load entire network
new_model = Model('full_model.pt', nn.MSELoss(), optim.Adam())
# Or load into existing model
model.load_net('full_model.pt')

# Make predictions with loaded model
preds = model.predict_net(x)
```

--------------------------------

### Import data splitting and generation tools

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Imports functions for creating synthetic datasets and splitting them into training, validation, and testing sets. These are essential for model evaluation.

```python
from sklearn.datasets import make_classification # to create a data set
from sklearn.model_selection import train_test_split
```

--------------------------------

### Importing dependencies

Source: https://github.com/havakv/torchtuples/blob/master/examples/01_introduction.ipynb

Initial imports required for data handling, plotting, and torchtuples functionality.

```python
import numpy as np
import matplotlib.pyplot as plt
import torch
import torchtuples
from torchtuples import Model
from torchtuples.practical import MLPVanilla
```

```python
from sklearn.datasets import make_classification # to create a data set
from sklearn.model_selection import train_test_split
```

```python
# to reproduce results
np.random.seed(1234)
_ = torch.manual_seed(4321)
```

--------------------------------

### Perform model predictions

Source: https://context7.com/havakv/torchtuples/llms.txt

Shows how to use predict and predict_net to generate outputs, including handling custom network methods and returning torch tensors.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim

# Create model with custom predict method
class CustomNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))

    def forward(self, x):
        return self.fc(x)

    def predict(self, x):
        # Custom predict returns probabilities
        return torch.sigmoid(self.forward(x))

net = CustomNet()
model = Model(net, nn.BCEWithLogitsLoss(), optim.Adam())

# Train (assuming trained model)
x_test = np.random.randn(100, 10).astype('float32')

# predict() uses net.predict() - returns probabilities
probs = model.predict(x_test)  # Returns numpy array by default
print(probs.shape)  # (100, 1)

# predict_net() uses net.forward() - returns logits
logits = model.predict_net(x_test)
print(logits.shape)  # (100, 1)

# Return torch tensor instead of numpy
logits_tensor = model.predict_net(x_test, numpy=False)
probs_from_logits = logits_tensor.sigmoid()

# Use custom function after prediction
probs_custom = model.predict_net(x_test, numpy=False, func=torch.sigmoid)

# Large dataset prediction with CPU offload
large_x = np.random.randn(100000, 10).astype('float32')
preds = model.predict(large_x, batch_size=8224, to_cpu=True, num_workers=4)
```

--------------------------------

### Create torchtuples dataloader

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Creates a dataloader using torchtuples with specified batch size, shuffle option, and number of workers.

```python
dl_tt = tuplefy(inp_tensor, tar_tensor).make_dataloader(batch_size, True, num_workers)
```

--------------------------------

### Plot Actual Learning Rates

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Visualize the actual learning rates used during training by multiplying the LR multiplier with the initial learning rate.

```python
lrs = lr_multiplier * model.optimizer.param_groups[0]['initial_lr']
lrs.plot()
plt.xlabel('batch')
_ = plt.ylabel("lr")
```

--------------------------------

### Import PyTorch Libraries

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Imports necessary libraries from PyTorch for building and training neural networks.

```python
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
from torchvision import datasets, transforms
```

--------------------------------

### Callbacks: EarlyStopping, BestWeights, and ClipGradNorm

Source: https://context7.com/havakv/torchtuples/llms.txt

Callbacks customize the training loop. EarlyStopping halts training when validation loss plateaus, and BestWeights saves/restores the optimal model weights.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim
from torchtuples.callbacks import EarlyStopping, BestWeights, ClipGradNorm
```

--------------------------------

### Prepare nested tuple dataset

Source: https://github.com/havakv/torchtuples/blob/master/README.md

Creating sample covariates structured as a nested tuple for model input.

```python
n = 500
x0, x1, x2 = [torch.randn(n, 3) for _ in range(3)]
y = torch.randn(n, 1)
x = (x0, (x0, x1, x2))
```

--------------------------------

### Visualizing training progress

Source: https://github.com/havakv/torchtuples/blob/master/examples/01_introduction.ipynb

Accessing training logs to export to pandas or plot performance metrics.

```python
log.to_pandas().head()
```

```python
log.plot()
```

--------------------------------

### Define Training Parameters

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Sets hyperparameters for the training process, including batch sizes, number of epochs, learning rate, and momentum.

```python
num_workers = 0
batch_size = 64
batch_size_test = 1000
epochs = 10
lr = 0.01
momentum=0.5
```

--------------------------------

### Create and Train MixedInputMLP

Source: https://context7.com/havakv/torchtuples/llms.txt

Initializes a network for mixed numeric and categorical data and trains it using tuple-based inputs.

```python
net = MixedInputMLP(
    in_features=5,              # Number of numeric features
    num_embeddings=[10, 5],     # Vocabulary size for each categorical
    embedding_dims=[4, 3],      # Embedding dimensions for each categorical
    num_nodes=[32, 16],         # Hidden layer sizes
    out_features=1,
    batch_norm=True,
    dropout=0.1,
    dropout_embedding=0.1       # Dropout after embeddings
)

model = Model(net, nn.MSELoss(), optim.Adam(lr=0.001))

# Train with tuple input (numeric, categorical)
x = (x_numeric, x_categoric)
log = model.fit(x, y, batch_size=64, epochs=10)

# Predict
x_test = (
    np.random.randn(50, 5).astype('float32'),
    np.column_stack([
        np.random.randint(0, 10, 50),
        np.random.randint(0, 5, 50)
    ]).astype('int64')
)
preds = model.predict_net(x_test)
```

--------------------------------

### Manually set learning rate for the model

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Allows manual setting of the learning rate for the model's optimizer. This is often preferred over using the lr_finder's output directly, as the finder might suggest a learning rate that is too high.

```python
model.optimizer.set_lr(0.1)
model.optimizer.param_groups[0]['lr']
```

--------------------------------

### Preparing the dataset

Source: https://github.com/havakv/torchtuples/blob/master/examples/01_introduction.ipynb

Generating a synthetic classification dataset and splitting it into training, validation, and test sets. Data must be cast to float32 and reshaped for compatibility with PyTorch loss functions.

```python
n = 10000
x_train, y_train = make_classification(n)
x_train = x_train.astype('float32')
y_train = y_train.astype('float32').reshape(-1, 1)
x_train, x_test, y_train, y_test = train_test_split(x_train, y_train, test_size=0.2)
x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.2)
```

--------------------------------

### Verify model weights consistency

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Trains models using both `torchtuples` and standard PyTorch, then asserts that their learned weights are identical to ensure functional equivalence.

```python
net_tt = fit_torchtuples(inp, tar, epochs, batch_size, num_workers)

net_t = fit_torch(inp_tensor, tar_tensor, epochs, batch_size, num_workers)

assert all([(w_tt == w_t).all() for w_tt, w_t in zip(net_tt.parameters(), net_t.parameters())]), 'Not equal weights'
```

--------------------------------

### Defining the neural network

Source: https://github.com/havakv/torchtuples/blob/master/examples/01_introduction.ipynb

Creating a multi-layer perceptron using the MLPVanilla utility.

```python
in_features = x_train.shape[1]
num_nodes = [32, 32]
out_features = 1
batch_norm = True
dropout = 0.1

net = MLPVanilla(in_features, num_nodes, out_features, batch_norm, dropout)
```

```python
net
```

--------------------------------

### Set computation device

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Defines the device for computation, defaulting to CPU. Change to 'cuda' for GPU acceleration if available.

```python
device = 'cpu' # change to run on gpu
```

--------------------------------

### Train with validation data

Source: https://context7.com/havakv/torchtuples/llms.txt

Demonstrates training a model while providing validation data via the val_data parameter to track performance metrics during training.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim
from sklearn.model_selection import train_test_split

# Prepare data
n = 5000
x = np.random.randn(n, 20).astype('float32')
y = np.random.randint(0, 2, (n, 1)).astype('float32')

x_train, x_val, y_train, y_val = train_test_split(x, y, test_size=0.2)

# Create model
net = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1))
model = Model(net, nn.BCEWithLogitsLoss(), optim.Adam(lr=0.001))

# Fit with validation data
log = model.fit(
    x_train,
    y_train,
    batch_size=256,
    epochs=20,
    val_data=(x_val, y_val),  # Validation data as tuple
    val_batch_size=512,       # Separate batch size for validation
    shuffle=True,
    num_workers=0,
    verbose=True
)

# Access training history
print(log.to_pandas().tail())
# Output:
#     train_loss  val_loss
# 15    0.5432    0.5521
# 16    0.5398    0.5489
# ...
```

--------------------------------

### Optimizers: SGD, Adam, AdamW, AdamWR, and RMSprop

Source: https://context7.com/havakv/torchtuples/llms.txt

torchtuples provides wrapped optimizers with features like reinitialization and decoupled weight decay. AdamW offers proper decoupled weight decay, and AdamWR adds cosine annealing with warm restarts.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim

net = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))

# Standard SGD with momentum
optimizer_sgd = optim.SGD(lr=0.01, momentum=0.9, weight_decay=0.0001)

# Adam optimizer
optimizer_adam = optim.Adam(lr=0.001, betas=(0.9, 0.999))

# AdamW with decoupled weight decay (recommended)
optimizer_adamw = optim.AdamW(
    lr=0.001,
    betas=(0.9, 0.99),
    decoupled_weight_decay=0.01,  # True decoupled weight decay
    dwd_normalize=True            # Normalize weight decay
)

# AdamWR with cosine annealing warm restarts
optimizer_adamwr = optim.AdamWR(
    lr=0.001,
    betas=(0.9, 0.99),
    decoupled_weight_decay=0.01,
    cycle_len=1,           # Epochs per cycle
    cycle_multiplier=2,    # Multiply cycle length after each restart
    cycle_eta_multiplier=1.0,
    eta_min=0
)

# RMSprop
optimizer_rmsprop = optim.RMSprop(lr=0.01, alpha=0.99)

# Create model with chosen optimizer
model = Model(net, nn.MSELoss(), optimizer_adamw)

# Modify learning rate after creation
model.optimizer.set_lr(0.0001)
```

--------------------------------

### Create PyTorch DataLoader

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Creates a standard PyTorch DataLoader with a TensorDataset, specifying batch size, shuffle option, and number of workers.

```python
ds = torch.utils.data.TensorDataset(inp_tensor, tar_tensor)
dl_t = torch.utils.data.DataLoader(ds, batch_size, True, num_workers=num_workers)
```

--------------------------------

### Create dataset function

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Generates a dataset using `make_classification` and converts it to both NumPy arrays and PyTorch tensors. This function is used for both implementations.

```python
def make_dataset(n):
    inp, tar = make_classification(n)
    inp = inp.astype('float32')
    tar = tar.reshape(-1, 1).astype('float32')
    inp_tensor = torch.from_numpy(inp)
    tar_tensor = torch.from_numpy(tar)
    return inp, tar, inp_tensor, tar_tensor
```

--------------------------------

### Fit model using torchtuples.Model

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Trains a simple MLP using `torchtuples.Model`. This approach is optimized for performance with small datasets.

```python
def fit_torchtuples(inp, tar, epochs, batch_size, num_workers):
    torch.manual_seed(0)
    net = MLPVanilla(inp.shape[1], [64, 64], 1)
    optimizer = torchtuples.optim.SGD(0.01)
    model = Model(net, loss_func, optimizer, device=device)
    model.fit(inp, tar, batch_size, epochs, verbose=False, num_workers=num_workers)
    return net
```

--------------------------------

### Prepare MixedInputMLP Data

Source: https://context7.com/havakv/torchtuples/llms.txt

Sets up sample data for MixedInputMLP, which handles both numeric and categorical features. Categorical features are represented as integer indices, suitable for entity embeddings.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim
from torchtuples.practical import MixedInputMLP

# Data with numeric and categorical features
n_samples = 1000

# Numeric features
x_numeric = np.random.randn(n_samples, 5).astype('float32')

# Categorical features (as integer indices)
# e.g., category1 has 10 unique values, category2 has 5
x_categoric = np.column_stack([
    np.random.randint(0, 10, n_samples),  # Category 1: 10 classes
    np.random.randint(0, 5, n_samples)    # Category 2: 5 classes
]).astype('int64')

y = np.random.randn(n_samples, 1).astype('float32')
```

--------------------------------

### Define a PyTorch MLP model architecture

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Instantiates a simple Multi-Layer Perceptron (MLP) using MLPVanilla with two hidden layers, ReLU activation, batch normalization, and dropout. Adjust `in_features`, `num_nodes`, `out_features`, `batch_norm`, and `dropout` as needed.

```python
in_features = x_train.shape[1]
num_nodes = [32, 32]
out_features = 1
batch_norm = True
dropout = 0.1

net = MLPVanilla(in_features, num_nodes, out_features, batch_norm, dropout)
```

--------------------------------

### Create and split a binary classification dataset

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Generates a synthetic binary classification dataset using make_classification and splits it into training, validation, and testing sets. Ensures data is float32 and y is a column vector for compatibility with PyTorch loss functions.

```python
# for reproducability
np.random.seed(1234)
_ = torch.manual_seed(4321)
```

```python
n = 10000
x_train, y_train = make_classification(n)
x_train = x_train.astype('float32')
y_train = y_train.astype('float32').reshape(-1, 1)
x_train, x_test, y_train, y_test = train_test_split(x_train, y_train, test_size=0.2)
x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.2)
```

```python
train = (x_train, y_train)
val = (x_val, y_val)
test = (x_test, y_test)
```

--------------------------------

### Train and predict with Model

Source: https://github.com/havakv/torchtuples/blob/master/README.md

Fitting the model and generating predictions using the torchtuples Model interface.

```python
model = Model(Net(), nn.MSELoss(), optim.SGD(0.01))
log = model.fit(x, y, batch_size=64, epochs=5)
```

```python
preds = model.predict(x)
```

```python
preds = model.predict_net(x)
```

--------------------------------

### Find Optimal Learning Rate with LR Finder

Source: https://context7.com/havakv/torchtuples/llms.txt

Utilizes the LR Finder to determine an optimal learning rate for training. It gradually increases the learning rate and tracks the loss, suggesting a suitable value. The model is automatically reset with the suggested LR after the finder completes.

```python
import torch
from torch import nn
import numpy as np
from torchtuples import Model, optim

# Create model
net = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
model = Model(net, nn.MSELoss(), optim.Adam())

# Generate data
x = np.random.randn(5000, 10).astype('float32')
y = np.random.randn(5000, 1).astype('float32')

# Run LR finder
lr_finder = model.lr_finder(
    x, y,
    batch_size=256,
    lr_min=1e-4,      # Acceptable lower bound for suggested LR
    lr_max=1.0,       # Acceptable upper bound for suggested LR
    lr_range=(1e-7, 10.0),  # Actual range to test
    n_steps=100,
    tolerance=np.inf
)

# Get suggested learning rate
best_lr = lr_finder.get_best_lr()
print(f"Suggested LR: {best_lr}")

# Plot LR vs loss curve
# lr_finder.plot()

# View results as DataFrame
print(lr_finder.to_pandas().head())

# Model is automatically reset with the suggested LR
# Now train with optimal LR
log = model.fit(x, y, batch_size=256, epochs=20)
```

--------------------------------

### Plot Training and Validation Loss

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Use this to visualize the training and validation loss over epochs. Requires the 'log' object and matplotlib.

```python
log.to_pandas()[['train_loss', 'val_loss']].plot()
plt.xlabel('epoch')
_ = plt.ylabel('loss')
```

--------------------------------

### Fit model using standard PyTorch

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Trains a similar MLP using standard PyTorch components like `TensorDataset` and `DataLoader`. This implementation is common but can be slower for small datasets.

```python
def fit_torch(inp, tar, epochs, batch_size, num_workers):
    """ We have used (more or less) the same setup for the regular 
    pytorch implementation as they do in scorch: 
    https://skorch.readthedocs.io/en/latest/user/tutorials.html
    """
    torch.manual_seed(0)
    net = MLPVanilla(inp.shape[1], [64, 64], 1)
    optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
    dataset = torch.utils.data.TensorDataset(inp, tar)
    dataloader = torch.utils.data.DataLoader(dataset, batch_size, True, num_workers=num_workers)
    for _ in range(epochs):
        for x, y in dataloader:
            optimizer.zero_grad()
            x, y = x.to(device), y.to(device)
            y_pred = net(x)
            loss = loss_func(y_pred, y)
            loss.backward()
            optimizer.step()
    return net
```

--------------------------------

### Find optimal learning rate using lr_finder

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Utilizes the model's lr_finder to determine a suitable learning rate for training. The finder iterates through different learning rates and records the corresponding loss. A tolerance can be set to stop early if the loss exceeds a threshold.

```python
batch_size = 256
lrfinder = model.lr_finder(x_train, y_train, batch_size, tolerance=10)
_ = lrfinder.plot()
```

--------------------------------

### Plot Training and Validation Accuracy

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Use this to visualize the training and validation accuracy over epochs. Requires the 'log' object and matplotlib.

```python
log.to_pandas()[['train_acc', 'val_acc']].plot()
plt.xlabel('epoch')
_ = plt.ylabel('accuracy')
```

--------------------------------

### Set Random Seeds for Reproducibility

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Ensures that the results of the training process are reproducible by setting manual seeds for NumPy and PyTorch.

```python
# for reproducability
np.random.seed(123456)
_ = torch.manual_seed(654321)
```

--------------------------------

### Load MNIST Training DataLoader

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Creates a DataLoader for the MNIST training dataset using torchvision. It applies transformations including converting images to tensors and normalizing them.

```python
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([transforms.ToTensor(),
                                                 transforms.Normalize((0.1307,), (0.3081,))])),
    batch_size=batch_size, shuffle=True, num_workers=num_workers)
```

--------------------------------

### Import Torchtuples and Matplotlib

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Imports specific modules from torchtuples for model definition, optimization, and practical utilities, along with Matplotlib for plotting.

```python
import matplotlib.pyplot as plt
from torchtuples import Model, optim, practical, data
```

--------------------------------

### Create Input-Only DataLoader for Predictions

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Generates a DataLoader that only yields input data (images) without labels, which is required for the `predict` method.

```python
test_input = data.dataloader_input_only(test_loader)
```

--------------------------------

### Retrieve the best learning rate found

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Extracts the recommended learning rate identified by the lr_finder. This value can be used to manually set the learning rate for the model's optimizer.

```python
lrfinder.get_best_lr()
```

--------------------------------

### Load MNIST Test DataLoader

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Creates a DataLoader for the MNIST test dataset. Similar to the training loader, it applies tensor conversion and normalization.

```python
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False,
                   transform=transforms.Compose([transforms.ToTensor(),
                                                 transforms.Normalize((0.1307,), (0.3081,))])),
    batch_size=batch_size_test, shuffle=False, num_workers=num_workers)
```

--------------------------------

### Score Batches with Model

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Evaluate the model's performance on a given dataset (e.g., validation set). Returns a dictionary containing loss and accuracy.

```python
model.score_in_batches(val)
```

--------------------------------

### Train the Model

Source: https://github.com/havakv/torchtuples/blob/master/examples/mnist_dataloader.ipynb

Fits the model to the training data using the `fit_dataloader` method. It includes validation on the test set and calculates accuracy as a metric.

```python
log = model.fit_dataloader(train_loader, epochs=epochs, val_dataloader=test_loader, metrics={'acc': practical.accuracy_argmax})
```

--------------------------------

### Time iteration of torchtuples dataloader

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Measures the time taken to iterate through one batch of the torchtuples dataloader using %timeit.

```python
%timeit next(iter(dl_tt))
```

--------------------------------

### Plot Learning Rate Multiplier

Source: https://github.com/havakv/torchtuples/blob/master/examples/02_general_usage.ipynb

Visualize the learning rate multiplier over batches. This shows how the base learning rate is adjusted by the scheduler.

```python
lr_multiplier = pd.Series(model.optimizer.lr_scheduler.get_etas())
lr_multiplier.plot()
plt.xlabel('batch')
_ = plt.ylabel("lr multiplier")
```

--------------------------------

### Define loss function

Source: https://github.com/havakv/torchtuples/blob/master/examples/benchmark.ipynb

Initializes the Binary Cross-Entropy with Logits loss function, commonly used for binary classification tasks.

```python
loss_func = torch.nn.BCEWithLogitsLoss()
```