### Install PyTorch Forecasting (CPU)

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Install the pytorch-forecasting library with CPU support. This command uses an extra index URL for PyTorch wheels.

```bash
pip install pytorch-forecasting --extra-index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Install PyTorch Lightning

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Install the PyTorch Lightning library, a dependency for pytorch-forecasting. This command installs the core library.

```bash
pip install lightning
```

--------------------------------

### Install PyTorch on Windows

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Use this command to install PyTorch on Windows systems. Ensure you are using pip.

```bash
pip install torch -f https://download.pytorch.org/whl/torch_stable.html
```

--------------------------------

### Install PyTorch Forecasting

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/index.md

Install the PyTorch Forecasting package using pip. This is the standard installation method for most users.

```bash
pip install pytorch-forecasting
```

--------------------------------

### Install PyTorch Forecasting (Conda)

Source: https://github.com/sktime/pytorch-forecasting/blob/main/README.md

Install using conda, specifying channels for pytorch-forecasting and PyTorch.

```bash
conda install pytorch-forecasting pytorch -c pytorch>=1.7 -c conda-forge
```

--------------------------------

### Install PyTorch Forecasting with MQF2 Loss

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/index.md

Install PyTorch Forecasting with support for the MQF2 loss (multivariate quantile loss). This is required for specific advanced loss functionalities.

```bash
pip install pytorch-forecasting[mqf2]
```

--------------------------------

### Setup Test Dataloader

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ptf_V2_example.ipynb

Prepares the test dataset and creates a dataloader for model evaluation. Ensure the data module is set up with the 'test' stage.

```python
data_module.setup(stage="test")
test_dataloader = data_module.test_dataloader()
```

--------------------------------

### Temporal Fusion Transformer Package v2 Example

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/pkg_v2.md

Demonstrates how to initialize and use the TFT_pkg_v2 with configuration dictionaries for datamodule, model, and trainer, followed by fitting and predicting on a dataset.

```python
from pytorch_forecasting.models.temporal_fusion_transformer._tft_pkg_v2 import TFT_pkg_v2
from pytorch_forecasting.metrics import MAE, SMAPE
from pytorch_forecasting.data.encoders import NaNLabelEncoder, TorchNormalizer
from sklearn.preprocessing import StandardScaler

# Define Configurations
datamodule_cfg = dict(
    max_encoder_length=30,
    max_prediction_length=1,
    batch_size=32,
)

model_cfg = dict(
    loss=MAE(),
    logging_metrics=[MAE(), SMAPE()],
    optimizer="adam",
    optimizer_params={"lr": 1e-3},
    hidden_size=64,
    num_layers=2,
)

trainer_cfg = dict(
    max_epochs=5,
    accelerator="auto",
    devices=1,
)

# Initialize the Package
model_pkg = TFT_pkg_v2(
    model_cfg=model_cfg,
    trainer_cfg=trainer_cfg,
    datamodule_cfg=datamodule_cfg,
)

# Train the model
# (Assuming `dataset` is a previously defined D1 TimeSeries object)
model_pkg.fit(dataset)

# Generate predictions
preds = model_pkg.predict(dataset, return_info=["index", "x", "y"])

```

--------------------------------

### Train Time Series Model with PyTorch Lightning

Source: https://github.com/sktime/pytorch-forecasting/blob/main/README.md

This example demonstrates the complete workflow for training a time series forecasting model. It includes necessary imports, data preparation using TimeSeriesDataSet, dataloader creation, and PyTorch Lightning Trainer setup with callbacks for early stopping and learning rate monitoring. Ensure your pandas DataFrame contains columns for target, timeseries ID, and observation time.

```python
import lightning.pytorch as pl
from lightning.pytorch.loggers import TensorBoardLogger
from lightning.pytorch.callbacks import EarlyStopping, LearningRateMonitor
from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer, QuantileLoss
from lightning.pytorch.tuner import Tuner

data = ...

max_encoder_length = 36
max_prediction_length = 6
training_cutoff = "YYYY-MM-DD"

training = TimeSeriesDataSet(
    data[lambda x: x.date <= training_cutoff],
    time_idx= ...,  # column name of time of observation
    target= ...,  # column name of target to predict
    group_ids=[ ... ],  # column name(s) for timeseries IDs
    max_encoder_length=max_encoder_length,  # how much history to use
    max_prediction_length=max_prediction_length,  # how far to predict into future
    static_categoricals=[ ... ],
    static_reals=[ ... ],
    time_varying_known_categoricals=[ ... ],
    time_varying_known_reals=[ ... ],
    time_varying_unknown_categoricals=[ ... ],
    time_varying_unknown_reals=[ ... ],
)

validation = TimeSeriesDataSet.from_dataset(training, data, min_prediction_idx=training.index.time.max() + 1, stop_randomization=True)

batch_size = 128
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=2)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size, num_workers=2)

early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-4, patience=1, verbose=False, mode="min")
lr_logger = LearningRateMonitor()
trainer = pl.Trainer(
    max_epochs=100,
    accelerator="auto",  # run on CPU, if on multiple GPUs, use strategy="ddp"
    gradient_clip_val=0.1,
    limit_train_batches=30,  # 30 batches per epoch
    callbacks=[lr_logger, early_stop_callback],
    logger=TensorBoardLogger("lightning_logs")
)

```

--------------------------------

### Install PyTorch Forecasting from a Specific Branch

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

To install a version from a specific branch of the repository, append the branch name to the repository URL using the '@' symbol. Replace `<branch_name>` with the desired branch.

```bash
pip install git+https://github.com/sktime/pytorch-forecasting.git@<branch_name>
```

--------------------------------

### Install Latest PyTorch Forecasting Version

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Use this command to install the most recent static snapshot of the repository, which includes features not yet published in a release. This is useful for developers testing or building with the latest updates.

```bash
pip install git+https://github.com/sktime/pytorch-forecasting.git
```

--------------------------------

### Install Editable PyTorch Forecasting with All Extras and Dev Dependencies

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Install PyTorch Forecasting in editable mode, including all optional dependencies and development dependencies. Use this if you need all functionalities for development.

```bash
pip install -e ".[all_extras,dev]"
```

--------------------------------

### Initialize TimeXer with Quantile Loss

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/tslib_v2_example.ipynb

Initialize the TimeXer model using QuantileLoss for probabilistic forecasting. This setup allows for predicting specific quantiles, such as the 10th, 50th (median), and 90th percentiles.

```python
model2 = TimeXer(
    loss=QuantileLoss(quantiles=[0.1, 0.5, 0.9]),  # quantiles of 0.1, 0.5 and 0.9 used.
    hidden_size=64,
    nhead=4,
    e_layers=2,
    d_ff=256,
    dropout=0.1,
    patch_length=4,
    optimizer="adam",
    optimizer_params={"lr": 1e-3},
    lr_scheduler="reduce_lr_on_plateau",
    lr_scheduler_params={
        "mode": "min",
        "factor": 0.5,
        "patience": 5,
    },
    metadata=data_module.metadata,
)
```

--------------------------------

### Install Editable PyTorch Forecasting with Dev Dependencies

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Install PyTorch Forecasting in editable mode, including only the development dependencies. This allows changes in your local code to be reflected immediately without reinstallation.

```bash
pip install -e ".[dev]"
```

--------------------------------

### Basic Time Series Forecasting Model Training

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/getting-started.md

This snippet demonstrates the end-to-end process of preparing data, defining a TimeSeriesDataSet, instantiating a TemporalFusionTransformer model, and training it using PyTorch Lightning. It includes steps for data loading, dataset creation, model initialization, trainer setup with callbacks, learning rate finding, and model fitting.

```python
import lightning.pytorch as pl
from lightning.pytorch.callbacks import EarlyStopping, LearningRateMonitor
from lightning.pytorch.tuner import Tuner
from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer

# load data
data = ...

# define dataset
max_encoder_length = 36
max_prediction_length = 6
training_cutoff = "YYYY-MM-DD"  # day for cutoff

training = TimeSeriesDataSet(
    data[lambda x: x.date < training_cutoff],
    time_idx= ..., 
    target= ..., 
    # weight="weight",
    group_ids=[ ... ],
    max_encoder_length=max_encoder_length,
    max_prediction_length=max_prediction_length,
    static_categoricals=[ ... ],
    static_reals=[ ... ],
    time_varying_known_categoricals=[ ... ],
    time_varying_known_reals=[ ... ],
    time_varying_unknown_categoricals=[ ... ],
    time_varying_unknown_reals=[ ... ],
)

# create validation and training dataset
validation = TimeSeriesDataSet.from_dataset(training, data, min_prediction_idx=training.index.time.max() + 1, stop_randomization=True)
batch_size = 128
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=2)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size, num_workers=2)

# define trainer with early stopping
early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-4, patience=1, verbose=False, mode="min")
lr_logger = LearningRateMonitor()
trainer = pl.Trainer(
    max_epochs=100,
    accelerator="auto",
    gradient_clip_val=0.1,
    limit_train_batches=30,
    callbacks=[lr_logger, early_stop_callback],
)

# create the model
tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.03,
    hidden_size=32,
    attention_head_size=1,
    dropout=0.1,
    hidden_continuous_size=16,
    output_size=7,
    loss=QuantileLoss(),
    log_interval=2,
    reduce_on_plateau_patience=4
)
print(f"Number of parameters in network: {tft.size()/1e3:.1f}k")

# find optimal learning rate (set limit_train_batches to 1.0 and log_interval = -1)
res = Tuner(trainer).lr_find(
    tft, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader, early_stop_threshold=1000.0, max_lr=0.3,
)

print(f"suggested learning rate: {res.suggestion()}")
fig = res.plot(show=True, suggest=True)
fig.show()

# fit the model
trainer.fit(
    tft, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader,
)

```

--------------------------------

### Activate Conda Development Environment

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Activate the newly created conda environment to start working on PyTorch Forecasting.

```bash
conda activate pytorch-forecasting-dev
```

--------------------------------

### Plot First 10 Predictions

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Plots the predictions for the first 10 examples using the raw prediction data.

```python
for idx in range(10):  # plot 10 examples
    best_tft.plot_prediction(
        raw_predictions.x, raw_predictions.output, idx=idx, add_loss_to_title=True
    )
```

--------------------------------

### Instantiate and Summarize Model

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Create an instance of the FullyConnectedMultiTargetModel using the dataset and specified hyperparameters, then print a summary of the model architecture.

```python
model = FullyConnectedMultiTargetModel.from_dataset(
    multi_target_dataset,
    hidden_size=10,
    n_hidden_layers=2,
    loss=MultiLoss(metrics=[MAE(), SMAPE()], weights=[2.0, 1.0]),
)
print(ModelSummary(model, max_depth=-1))
```

--------------------------------

### Get TimeSeriesDataSet Parameters

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Retrieve the configuration parameters of a TimeSeriesDataSet object. This is useful for verifying the dataset setup.

```python
dataset.get_parameters()
```

--------------------------------

### Initialize TFT_pkg_v2 Model

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ptf_V2_example.ipynb

Instantiate the TFT_pkg_v2 model with specified configurations for the model, trainer, and datamodule.

```python
model_pkg = TFT_pkg_v2(
    model_cfg=model_cfg,
    trainer_cfg=trainer_cfg,
    datamodule_cfg=datamodule_cfg,
)
```

--------------------------------

### Import Necessary Packages

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/tslib_v2_example.ipynb

Import the required metrics and loss functions for initializing the TimeXer model.

```python
from pytorch_forecasting.metrics import MAE, SMAPE, QuantileLoss
```

--------------------------------

### Configure Trainer and Model

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/deepar.ipynb

Set up the PyTorch Lightning Trainer with early stopping and gradient clipping. Then, initialize the DeepAR model from a dataset, specifying optimizer, loss function, and network architecture.

```python
early_stop_callback = EarlyStopping(
    monitor="val_loss", min_delta=1e-4, patience=10, verbose=False, mode="min"
)
trainer = pl.Trainer(
    max_epochs=30,
    accelerator="cpu",
    enable_model_summary=True,
    gradient_clip_val=0.1,
    callbacks=[early_stop_callback],
    limit_train_batches=50,
    enable_checkpointing=True,
)


net = DeepAR.from_dataset(
    training,
    learning_rate=1e-2,
    log_interval=10,
    log_val_interval=1,
    hidden_size=30,
    rnn_layers=2,
    optimizer="Adam",
    loss=MultivariateNormalDistributionLoss(rank=30),
)
```

--------------------------------

### Configure Network and Trainer for Learning Rate Finder

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Sets up the PyTorch Lightning trainer and the Temporal Fusion Transformer model for the learning rate finder. Ensure PyTorch Lightning and the model are imported.

```python
pl.seed_everything(42)
trainer = pl.Trainer(
    accelerator="cpu",
    # clipping gradients is a hyperparameter and important to prevent divergance
    # of the gradient for recurrent neural networks
    gradient_clip_val=0.1,
)

tft = TemporalFusionTransformer.from_dataset(
    training,
    # not meaningful for finding the learning rate but otherwise very important
    learning_rate=0.03,
    hidden_size=8,  # most important hyperparameter apart from learning rate
    # number of attention heads. Set to up to 4 for large datasets
    attention_head_size=1,
    dropout=0.1,  # between 0.1 and 0.3 are good values
    hidden_continuous_size=8,  # set to <= hidden_size
    loss=QuantileLoss(),
    optimizer="ranger",
    # reduce learning rate if no improvement in validation loss after x epochs
    # reduce_on_plateau_patience=1000,
)
print(f"Number of parameters in network: {tft.size() / 1e3:.1f}k")
```

--------------------------------

### Initialize Model and Print Summary

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Initializes the FullyConnectedModel from a dataset and prints a summary of the model's architecture and parameters using ModelSummary. This is useful for verifying the model structure.

```python
from lightning.pytorch.utilities.model_summary import ModelSummary

model = FullyConnectedModel.from_dataset(dataset, hidden_size=10, n_hidden_layers=2)
print(ModelSummary(model, max_depth=-1))
model.hparams
```

--------------------------------

### Initialize Trainer for Model 1

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/tslib_v2_example.ipynb

Set up a PyTorch Lightning Trainer for model1 with automatic device selection, enabling progress bar and model summary. This trainer is configured for a maximum of 5 epochs.

```python
trainer1 = Trainer(
    max_epochs=5,
    accelerator="auto",
    devices=1,
    enable_progress_bar=True,
    enable_model_summary=True,
)
```

--------------------------------

### Generate and Plot Raw Predictions

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Uses the trained model to predict on new data in raw mode and plots the predictions for the first 10 examples. Ensure the trainer is configured appropriately, for example, by specifying the accelerator.

```python
new_raw_predictions = best_tft.predict(
    new_prediction_data,
    mode="raw",
    return_x=True,
    trainer_kwargs=dict(accelerator="cpu"),
)

for idx in range(10):  # plot 10 examples
    best_tft.plot_prediction(
        new_raw_predictions.x,
        new_raw_predictions.output,
        idx=idx,
        show_future_observed=False,
    )
```

--------------------------------

### Get Raw Predictions

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Obtains raw predictions from the best model, including quantiles, for further analysis.

```python
# raw predictions are a dictionary from which all kind of information including quantiles can be extracted
raw_predictions = best_tft.predict(
    val_dataloader, mode="raw", return_x=True, trainer_kwargs=dict(accelerator="cpu")
)
```

--------------------------------

### Initialize Trainer for Model 2

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/tslib_v2_example.ipynb

Set up a PyTorch Lightning Trainer for model2 with automatic device selection, enabling progress bar and model summary. This trainer is configured for a maximum of 4 epochs.

```python
trainer2 = Trainer(
    max_epochs=4,
    accelerator="auto",
    devices=1,
    enable_progress_bar=True,
    enable_model_summary=True,
)
```

--------------------------------

### Quantile Prediction Shape

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Get the shape of quantile predictions from the model using the predict method.

```python
model.predict(dataloader, mode="quantiles", mode_kwargs=dict(n_samples=100)).shape
```

--------------------------------

### BaseModel Initialization Parameters

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

This snippet displays the documentation for the `__init__` method of the `BaseModel` class, detailing its arguments for model configuration.

```python
print(BaseModel.__init__.__doc__)
```

--------------------------------

### Get Prediction Shape

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/tslib_v2_example.ipynb

Prints the shape of the prediction tensor. This is useful for verifying that the output dimensions match the expected format.

```python
y_pred["prediction"].shape

```

--------------------------------

### Plotting N-HiTS Interpretation

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/nhits.ipynb

Visualizes the interpretation of N-HiTS model predictions for a given number of examples, showing how the model arrives at its forecasts.

```python
for idx in range(2):  # plot 10 examples
    best_model.plot_interpretation(raw_predictions.x, raw_predictions.output, idx=idx)
```

--------------------------------

### Train TFT Model using Package Class V2

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/models_v2.md

Use this workflow to orchestrate model training with the high-level TFT_pkg_v2 class. It requires defining dataset, model, trainer, and datamodule configurations as dictionaries.

```python
from pytorch_forecasting.models.temporal_fusion_transformer._tft_pkg_v2 import TFT_pkg_v2
from pytorch_forecasting.data.timeseries._timeseries_v2 import TimeSeries
from pytorch_forecasting.data.encoders import NaNLabelEncoder, TorchNormalizer
from sklearn.preprocessing import StandardScaler
from pytorch_forecasting.metrics import MAE, SMAPE

# 1. D1 Layer: Create the TimeSeries dataset
# This takes the raw pandas DataFrame and prepares the tensor extraction
dataset = TimeSeries(
    data=data_df,
    time="time_idx",
    target="y",
    group=["series_id"],
    num=["x", "future_known_feature", "static_feature"],
    cat=["category", "static_feature_cat"],
    known=["future_known_feature"],
    unknown=["x", "category"],
    static=["static_feature", "static_feature_cat"],
)

# 2. D2 Layer Configuration
datamodule_cfg = dict(
    max_encoder_length=30,
    max_prediction_length=1,
    batch_size=32,
)

# 3. Model Configuration
model_cfg = dict(
    loss=MAE(),
    logging_metrics=[MAE(), SMAPE()],
    optimizer="adam",
    optimizer_params={"lr": 1e-3},
    lr_scheduler="reduce_lr_on_plateau",
    lr_scheduler_params={"mode": "min", "factor": 0.1, "patience": 10},
    hidden_size=64,
    num_layers=2,
    attention_head_size=4,
    dropout=0.1,
)

# 4. Trainer Configuration
trainer_cfg = dict(
    max_epochs=5,
    accelerator="auto",
    devices=1,
    enable_progress_bar=True,
    log_every_n_steps=10,
)

# 5. Package Layer: Orchestration
model_pkg = TFT_pkg_v2(
    model_cfg=model_cfg,
    trainer_cfg=trainer_cfg,
    datamodule_cfg=datamodule_cfg,
)

# Fit the model (You can pass the D1 dataset or a D2 DataModule here)
model_pkg.fit(dataset)

# Generate predictions (You can also pass a DataModule or Dataloader here)
preds = model_pkg.predict(dataset, return_info=["index", "x", "y"])

```

--------------------------------

### Python Code Snippet Placeholder

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

This is a placeholder for Python code examples. It indicates where Python code would typically be found or inserted.

```python
# Placeholder for Python code

```

--------------------------------

### Train TFT Model with PyTorch Lightning Trainer

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ptf_V2_example.ipynb

Sets up and runs the training process for the initialized TFT model using PyTorch Lightning's Trainer. Configures training epochs, accelerator, devices, progress bar, and logging frequency.

```python
from lightning.pytorch import Trainer
```

```python
# Train the model
print("\nTraining model...")
trainer = Trainer(
    max_epochs=5,
    accelerator="auto",
    devices=1,
    enable_progress_bar=True,
    log_every_n_steps=10,
)

trainer.fit(model, data_module)
```

--------------------------------

### Model Training Output and Summary

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ptf_V2_example.ipynb

This output shows the progress and summary of the model training process, including GPU utilization, model parameters, and training status.

```text
INFO: GPU available: True (cuda), used: True
INFO:lightning.pytorch.utilities.rank_zero:GPU available: True (cuda), used: True
INFO: TPU available: False, using: 0 TPU cores
INFO:lightning.pytorch.utilities.rank_zero:TPU available: False, using: 0 TPU cores
INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
INFO: 
  | Name                  | Type               | Params | Mode 
---------------------------------------------------------------------
0 | loss                  | MAE                | 0      | train
1 | encoder_var_selection | Sequential         | 709    | train
2 | decoder_var_selection | Sequential         | 193    | train
3 | static_context_linear | Linear             | 192    | train
4 | lstm_encoder          | LSTM               | 51.5 K | train
5 | lstm_decoder          | LSTM               | 50.4 K | train
6 | self_attention        | MultiheadAttention | 16.6 K | train
7 | pre_output            | Linear             | 4.2 K  | train
8 | output_layer          | Linear             | 65     | train
---------------------------------------------------------------------
123 K     Trainable params
0         Non-trainable params
123 K     Total params
0.495     Total estimated model params size (MB)
18        Modules in train mode
0         Modules in eval mode
INFO:lightning.pytorch.callbacks.model_summary:
  | Name                  | Type               | Params | Mode 
---------------------------------------------------------------------
0 | loss                  | MAE                | 0      | train
1 | encoder_var_selection | Sequential         | 709    | train
2 | decoder_var_selection | Sequential         | 193    | train
3 | static_context_linear | Linear             | 192    | train
4 | lstm_encoder          | LSTM               | 51.5 K | train
5 | lstm_decoder          | LSTM               | 50.4 K | train
6 | self_attention        | MultiheadAttention | 16.6 K | train
7 | pre_output            | Linear             | 4.2 K  | train
8 | output_layer          | Linear             | 65     | train
---------------------------------------------------------------------
123 K     Trainable params
0         Non-trainable params
123 K     Total params
0.495     Total estimated model params size (MB)
18        Modules in train mode
0         Modules in eval mode
```

--------------------------------

### Plot N-Beats Interpretations

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ar.ipynb

Visualizes the interpretability of the N-Beats model for a specified number of examples. This helps in understanding which parts of the input influenced the forecast.

```python
for idx in range(10):  # plot 10 examples
    best_model.plot_interpretation(raw_predictions.x, raw_predictions.output, idx=idx)
```

--------------------------------

### Configure Network and Trainer for Training

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Sets up the PyTorch Lightning trainer with callbacks for early stopping and learning rate monitoring, and configures the Temporal Fusion Transformer model for training. Adjust `limit_train_batches` for faster debugging.

```python
early_stop_callback = EarlyStopping(
    monitor="val_loss", min_delta=1e-4, patience=10, verbose=False, mode="min"
)
lr_logger = LearningRateMonitor()  # log the learning rate
logger = TensorBoardLogger("lightning_logs")  # logging results to a tensorboard

trainer = pl.Trainer(
    max_epochs=50,
    accelerator="cpu",
    enable_model_summary=True,
    gradient_clip_val=0.1,
    limit_train_batches=50,  # comment in for training, running validation every 30 batches
    # fast_dev_run=True,  # comment in to check that networkor dataset has no serious bugs
    callbacks=[lr_logger, early_stop_callback],
    logger=logger,
)

tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.03,
    hidden_size=16,
    attention_head_size=2,
    dropout=0.1,
    hidden_continuous_size=8,
    loss=QuantileLoss(),
    log_interval=10,  # uncomment for learning rate finder and otherwise, e.g. to 10 for logging every 10 batches
    optimizer="ranger",
    reduce_on_plateau_patience=4,
)
print(f"Number of parameters in network: {tft.size() / 1e3:.1f}k")
```

--------------------------------

### Get Prediction Shape

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/tslib_v2_example.ipynb

Retrieves and prints the shape of the prediction tensor generated by the model. This is useful for verifying the output dimensions match expectations.

```python
y_pred["prediction"].shape
```

--------------------------------

### Instantiate and Summarize Model

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Create an instance of the FullyConnectedClassificationModel from the dataset and print its summary. Access hyperparameters using .hparams.

```python
model = FullyConnectedClassificationModel.from_dataset(
    classification_dataset, hidden_size=10, n_hidden_layers=2
)
print(ModelSummary(model, max_depth=-1))
model.hparams
```

--------------------------------

### Clone PyTorch Forecasting Repository

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Clone your forked repository to your local disk to begin development. Remember to replace `<username>` with your actual GitHub username.

```bash
git clone git@github.com:<username>/sktime/pytorch-forecasting.git
cd pytorch-forecasting
```

--------------------------------

### Get Raw Predictions for Visualization

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ar.ipynb

Retrieves raw predictions along with the input data from the N-Beats model. This is necessary for plotting individual predictions and interpretations.

```python
raw_predictions = best_model.predict(val_dataloader, mode="raw", return_x=True)
```

--------------------------------

### Plot N-Beats Predictions

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ar.ipynb

Visualizes the forecasted predictions against the actual values for a specified number of examples from the validation set. The loss is optionally added to the plot title.

```python
for idx in range(10):  # plot 10 examples
    best_model.plot_prediction(
        raw_predictions.x, raw_predictions.output, idx=idx, add_loss_to_title=True
    )
```

--------------------------------

### Predict Quantiles with Stallion

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Use this snippet to get quantile predictions from the Stallion model for a filtered subset of training data. Specify `mode='quantiles'` and optionally configure the trainer with `trainer_kwargs`.

```python
best_tft.predict(
    training.filter(
        lambda x: (x.agency == "Agency_01")
        & (x.sku == "SKU_01")
        & (x.time_idx_first_prediction == 15)
    ),
    mode="quantiles",
    trainer_kwargs=dict(accelerator="cpu"),
)
```

--------------------------------

### Initialize N-Beats Model and Trainer

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ar.ipynb

Sets the random seed, initializes the PyTorch Lightning Trainer, and configures the N-Beats model with specified learning rate, weight decay, and network widths.

```python
pl.seed_everything(42)
trainer = pl.Trainer(accelerator="auto", gradient_clip_val=0.1)
net = NBeats.from_dataset(
    training,
    learning_rate=3e-2,
    weight_decay=1e-2,
    widths=[32, 512],
    backcast_loss_ratio=0.1,
)
```

--------------------------------

### Initialize and Use FullyConnectedModel

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Initializes a FullyConnectedModel using from_dataset and performs a forward pass with sample data.

```python
model = FullyConnectedModel.from_dataset(
    dataset, input_size=5, output_size=2, hidden_size=10, n_hidden_layers=2
)
x, y = next(iter(dataloader))
model(x)
```

--------------------------------

### Display DataFrame Description

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Use the `describe()` method to get a statistical summary of the DataFrame, including count, mean, standard deviation, min, max, and quartiles. This is useful for understanding the distribution of numerical features.

```python
data.describe()
```

--------------------------------

### Predicting Multiple Targets Simultaneously

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Demonstrates how to create a pandas DataFrame with multiple target variables, including mixed types, for multi-target forecasting. This setup is essential for training models that predict several outcomes at once.

```python
import pandas as pd
import numpy as np

multi_target_test_data = pd.DataFrame(
    dict(
        target1=np.random.rand(30),
        target2=np.random.rand(30),
        group=np.repeat(np.arange(3), 10),
        time_idx=np.tile(np.arange(10), 3),
    )
)
multi_target_test_data
```

--------------------------------

### Implement get_base_test_params() for Model Testing

Source: https://github.com/sktime/pytorch-forecasting/blob/main/extension_templates/v1/network/README.md

This method must return at least two parameter settings for valid model instantiation, exercising different configurations, and running quickly in CI. The first element must be an empty dictionary to test default initialization.

```python
return [
    {},
    {"hidden_size": 8, "use_exogenous": True},
]
```

--------------------------------

### LSTMModel Forward Pass

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

The forward method orchestrates the model's execution by first calling the encode method to get the hidden state and then the decode method to generate predictions. It returns the network output in the expected format.

```python
    def forward(self, x: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:
        hidden_state = self.encode(x)  # encode to hidden state
        output = self.decode(x, hidden_state)  # decode leveraging hidden state

        return self.to_network_output(prediction=output)
```

--------------------------------

### Import TFT Model and Metrics

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ptf_V2_example.ipynb

Imports the Temporal Fusion Transformer (TFT) model and relevant metrics (MAE, SMAPE) for model initialization and evaluation.

```python
from pytorch_forecasting.metrics import MAE, SMAPE
from pytorch_forecasting.models.temporal_fusion_transformer._tft_v2 import TFT
```

--------------------------------

### Predict Raw Data with Stallion and Plot

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

This snippet shows how to obtain raw predictions and the corresponding input data from the Stallion model. It then uses `plot_prediction` to visualize the results. Ensure `return_x=True` is set to get the input data for plotting.

```python
raw_prediction = best_tft.predict(
    training.filter(
        lambda x: (x.agency == "Agency_01")
        & (x.sku == "SKU_01")
        & (x.time_idx_first_prediction == 15)
    ),
    mode="raw",
    return_x=True,
    trainer_kwargs=dict(accelerator="cpu"),
)
best_tft.plot_prediction(raw_prediction.x, raw_prediction.output, idx=0)
```

--------------------------------

### Generate DataLoader and Inspect Output

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Create a DataLoader from the dataset and retrieve a batch to inspect the target values.

```python
x, y = next(iter(multi_target_dataset.to_dataloader(batch_size=4)))
y[0]  # target values are a list of targets
```

--------------------------------

### Test Model with Trainer and Data Module

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/tslib_v2_example.ipynb

This snippet shows how to initiate testing of a trained model using a PyTorch Lightning trainer and a data module. It's used to evaluate model performance on a test dataset.

```python
test_metrics = trainer1.test(model1, data_module)
```

--------------------------------

### Initialize Model with Custom Metric (MAE)

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Pass a custom metric instance, such as MAE, to the model's initialization, preferably via the `from_dataset()` method. Ensure the metric is imported from `pytorch_forecasting.metrics`.

```python
from pytorch_forecasting.metrics import MAE

model = FullyConnectedModel.from_dataset(
    dataset, hidden_size=10, n_hidden_layers=2, loss=MAE()
)
model.hparams
```

--------------------------------

### Configure Upstream Remote Repository

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

After cloning, set up the 'upstream' remote to point to the main repository. This allows you to fetch changes from the original project.

```bash
git remote add upstream https://github.com/sktime/pytorch-forecasting.git
```

--------------------------------

### Verify Git Remotes

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/installation.md

Verify that both 'origin' (your fork) and 'upstream' (the main repository) remotes are correctly configured.

```bash
git remote -v
```

--------------------------------

### Train TFT Model with EncoderDecoderDataModule

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/models_v2.md

This workflow demonstrates training a TFT model without the Package class, using EncoderDecoderTimeSeriesDataModule and Lightning's Trainer. It requires creating a TimeSeries dataset, a data module, and the model instance.

```python
from pytorch_forecasting.data.timeseries import TimeSeries
from pytorch_forecasting.data.data_module import EncoderDecoderTimeSeriesDataModule
from pytorch_forecasting.metrics import MAE, SMAPE
from pytorch_forecasting.models.temporal_fusion_transformer._tft_v2 import TFT
from lightning.pytorch import Trainer

# create `TimeSeries` dataset that returns the raw data in terms of tensors
dataset = TimeSeries(
    data=data_df,
    time="time_idx",
    target="y",
    group=["series_id"],
    num=["x", "future_known_feature", "static_feature"],
    cat=["category", "static_feature_cat"],
    known=["future_known_feature"],
    unknown=["x", "category"],
    static=["static_feature", "static_feature_cat"],
)

# create the `data_module` that handles the dataloaders and preprocessing
data_module = EncoderDecoderTimeSeriesDataModule(
    time_series_dataset=dataset,
    max_encoder_length=30,
    max_prediction_length=1,
    batch_size=32,
)

# Initialise the Model
model = TFT(
    loss=MAE(),
    logging_metrics=[MAE(), SMAPE()],
    optimizer="adam",
    optimizer_params={"lr": 1e-3},
    lr_scheduler="reduce_lr_on_plateau",
    lr_scheduler_params={"mode": "min", "factor": 0.1, "patience": 10},
    hidden_size=64,
    num_layers=2,
    attention_head_size=4,
    dropout=0.1,
    metadata=data_module.metadata,  # pass the metadata from the datamodule to the model
    # to initialise important params like `encoder_cont` etc
)

# Train the model
trainer = Trainer(
    max_epochs=5,
    accelerator="auto",
    devices=1,
    enable_progress_bar=True,
    log_every_n_steps=10,
)

trainer.fit(model, data_module)

```

--------------------------------

### Instantiate LSTMModel

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Instantiate the LSTMModel using the from_dataset class method, providing the dataset and model-specific hyperparameters like n_layers and hidden_size. This is a convenient way to create a model ready for training.

```python
model = LSTMModel.from_dataset(dataset, n_layers=2, hidden_size=10)
```

--------------------------------

### Find Optimal Learning Rate

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/deepar.ipynb

Use the Tuner to find the optimal learning rate for your model. This helps in faster convergence and better performance. Requires PyTorch Lightning.

```python
# find optimal learning rate
from pytorch_forecasting.tuning import Tuner

res = Tuner(trainer).lr_find(
    net,
    train_dataloaders=train_dataloader,
    val_dataloaders=val_dataloader,
    min_lr=1e-5,
    max_lr=1e0,
    early_stop_threshold=100,
)
print(f"suggested learning rate: {res.suggestion()}")
fig = res.plot(show=True, suggest=True)
net.hparams.learning_rate = res.suggestion()
```

--------------------------------

### Import Libraries

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/stallion.ipynb

Import necessary libraries for data manipulation, modeling, and forecasting. This includes pandas, numpy, PyTorch Lightning, and components from pytorch-forecasting.

```python
import copy
from pathlib import Path
import warnings

import lightning.pytorch as pl
from lightning.pytorch.callbacks import EarlyStopping, LearningRateMonitor
from lightning.pytorch.loggers import TensorBoardLogger
import numpy as np
import pandas as pd
import torch

from pytorch_forecasting import Baseline, TemporalFusionTransformer, TimeSeriesDataSet
from pytorch_forecasting.data import GroupNormalizer
from pytorch_forecasting.metrics import MAE, SMAPE, PoissonLoss, QuantileLoss
from pytorch_forecasting.models.temporal_fusion_transformer.tuning import (
    optimize_hyperparameters,
)
```

--------------------------------

### Configuring Loss Quantiles

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Instantiate a NormalDistributionLoss with custom quantiles.

```python
NormalDistributionLoss(quantiles=[0.2, 0.8]).quantiles
```

--------------------------------

### Generate and Pass Batch to Model

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Creates a sample batch from the dataset's dataloader and passes it through the model for a forward pass. This is a common step for testing model input/output.

```python
x, y = next(iter(dataset_with_covariates.to_dataloader(batch_size=4)))  # generate batch
model(x)  # pass batch through model
```

--------------------------------

### Initialize N-HiTS Model

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/nhits.ipynb

Initializes the N-HiTS model with specified hyperparameters, including loss function and optimizer. Sets up the PyTorch Lightning Trainer.

```python
pl.seed_everything(42)
trainer = pl.Trainer(accelerator="cpu", gradient_clip_val=0.1)
net = NHiTS.from_dataset(
    training,
    learning_rate=3e-2,
    weight_decay=1e-2,
    loss=MQF2DistributionLoss(prediction_length=max_prediction_length),
    backcast_loss_ratio=0.0,
    hidden_size=64,
    optimizer="AdamW",
)
```

--------------------------------

### Initialize Temporal Fusion Transformer (TFT) Model

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ptf_V2_example.ipynb

Initializes the Temporal Fusion Transformer (TFT) model, passing the loss function, logging metrics, and crucially, the metadata from the data module for internal configuration.

```python
model = TFT(
    loss=nn.MSELoss(),
    logging_metrics=[MAE(), SMAPE()],
    metadata=data_module.metadata,  # <-- crucial for model setup
    ...
)
```

--------------------------------

### Initialize NaNLabelEncoder

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/building.ipynb

Instantiate the NaNLabelEncoder for handling missing values in labels. This is often a prerequisite for preparing data for pytorch-forecasting models.

```python
from pytorch_forecasting.data.encoders import NaNLabelEncoder
```

--------------------------------

### Configure Trainer

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/ptf_V2_example.ipynb

Defines a configuration dictionary for the PyTorch Lightning Trainer. This includes settings for maximum epochs, accelerator, devices, progress bar enablement, and logging frequency.

```python
trainer_cfg = dict(
    max_epochs=5,
    accelerator="auto",
    devices=1,
    enable_progress_bar=True,
    log_every_n_steps=10,
)
```

--------------------------------

### Import Libraries

Source: https://github.com/sktime/pytorch-forecasting/blob/main/docs/source/tutorials/nhits.ipynb

Imports necessary libraries for PyTorch Lightning, N-HiTS, data handling, and plotting.

```python
import lightning.pytorch as pl
from lightning.pytorch.callbacks import EarlyStopping
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch

from pytorch_forecasting import Baseline, NHiTS, TimeSeriesDataSet
from pytorch_forecasting.data import NaNLabelEncoder
from pytorch_forecasting.data.examples import generate_ar_data
from pytorch_forecasting.metrics import MAE, SMAPE, MQF2DistributionLoss, QuantileLoss
```