### Install Dependencies and Setup Git Hooks

Source: https://github.com/awslabs/gluonts/blob/dev/docs/community/devsetup.md

Run this script to install all required packages and set up Git hooks for automated type and style checks on commit. Use quotes for package names if using zsh.

```bash
./dev_setup.sh
pip install -e .[dev]

# if you use zsh you might need to escape `[` and `]`
pip install -e ".[dev]"
```

--------------------------------

### Setup EC2 Instance Script

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/README.md

Use this script to set up dependencies on an Ubuntu 20.04 EC2 instance. Ensure Python, CUDA, and Poetry are installed.

```bash
bash bin/setup-ec2.sh
```

--------------------------------

### Install Package with Extras

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs optional dependencies for a package. Used here to show the syntax for installing extras.

```sh
pip install "some-package[extra-1,extra-2]==version"
```

--------------------------------

### Install Poetry and Configure Virtual Environment

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/few_shot_prediction/README.md

Installs Poetry, configures it to use project-local virtual environments, and sets up a new project with Python 3.8.9.

```bash
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
source $HOME/.poetry/env
poetry config virtualenvs.in-project true
cd to_your_project
pyenv local 3.8.9
poetry config virtualenvs.in-project true
poetry install
```

--------------------------------

### Install Dependencies with Poetry

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/README.md

Install all project dependencies using Poetry. This command should be run from the root of the TSBench repository.

```bash
poetry install
```

--------------------------------

### Install GluonTS

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs the latest version of GluonTS from PyPi. Use this for general usage.

```sh
pip install gluonts
```

--------------------------------

### Install pyenv and Python 3.8.9

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/few_shot_prediction/README.md

Installs pyenv, configures environment variables, and installs Python version 3.8.9. Ensure to source the profile after configuration.

```bash
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
cd ~/.pyenv && src/configure && make -C src
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.profile
echo 'eval "$(pyenv init --path)"' >> ~/.profile
source ~/.profile
pyenv install 3.8.9
```

--------------------------------

### Install GluonTS with Shell Extra

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs GluonTS with shell integration, useful for environments like Amazon SageMaker.

```sh
pip install "gluonts[shell]"
```

--------------------------------

### Install PytorchTS

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/SCott/README.md

Installs the PytorchTS framework, a dependency for the SCott implementation. This command uses conda for package management.

```bash
$ conda install pytorchts==0.2.0
```

--------------------------------

### Clone Repository and Navigate

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/README.md

Clone the GluonTS repository and navigate to the TSBench directory. This is a prerequisite for installing dependencies.

```bash
git clone git@github.com:awslabs/gluonts.git
cd gluonts/src/gluonts/nursery/tsbench
```

--------------------------------

### Run Sphinx Autobuild for Documentation

Source: https://github.com/awslabs/gluonts/blob/dev/docs/community/devsetup.md

Start a local web server with a watchdog to automatically rebuild documentation when .rst files change. Ensure no syntax errors or warnings.

```bash
cd docs                          # go to the docs folder
make livehtml                    # run the autobuild watchdog, ensure that
                                 # there are no syntax errors and warnings
open http://127.0.0.1:8000       # open the autobuild preview
```

--------------------------------

### Install GluonTS with Version Pinning

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs a specific minor version of GluonTS. Recommended for production usage to ensure stability.

```sh
pip install gluonts==0.12.*
```

--------------------------------

### Install GluonTS with Prophet Extra

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs GluonTS with support for the Prophet forecasting library. This includes the 'prophet' package.

```sh
pip install "gluonts[prophet]"
```

--------------------------------

### Preprocess Dataset

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/SCott/README.md

Generates the necessary dataset files by running the preprocessing script. This script should be executed after installing the required packages.

```bash
$ python3 preprocess_data.py
```

--------------------------------

### Display TSBench CLI Help

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/README.md

Run the help command for the tsbench CLI to get an overview of available subcommands and options. This requires Poetry shell to be activated.

```bash
tsbench --help
```

--------------------------------

### Install GluonTS with PyTorch Extra

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs GluonTS with PyTorch support, including PyTorch and PyTorch Lightning. Required for using PyTorch-based models.

```sh
pip install "gluonts[torch]"
```

--------------------------------

### Instantiate a local Prophet model

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/concepts.md

Example of directly creating a local Prophet Predictor for immediate use without a separate training step.

```python
# local Prophet model
predictor = ProphetPredictor(prediction_length=24)
```

--------------------------------

### Install GluonTS with Arrow Extra

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs GluonTS with support for reading and writing Arrow and Parquet datasets. This includes the necessary Apache Arrow packages.

```sh
pip install "gluonts[arrow]"
```

--------------------------------

### Install GluonTS with MXNet Extra

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs GluonTS with MXNet support, including a CPU-only version of MXNet. Refer to MXNet documentation for GPU versions.

```sh
pip install "gluonts[mxnet]"
```

--------------------------------

### Library Imports and Device Setup

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/spliced_binned_pareto/run_nursery_model_example.ipynb

Imports necessary libraries for plotting, data manipulation, and deep learning. It also configures the device (GPU or CPU) and sets random seeds for reproducibility.

```python
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import pandas as pd
import numpy as np
import os
import random
from scipy import stats
import time
from tqdm import tqdm, trange

import torch
from torch import optim

from spliced_binned_pareto import SplicedBinnedPareto, Binned
from distr_tcn import DistributionalTCN
from gaussian_model import GaussianModel
from training_functions import (
    train_step_from_batch,
    eval_on_series,
    plot_prediction,
    highlight_min,
)
from data_functions import create_ds, create_ds_asymmetric

font = {"family": "serif", "weight": "normal", "size": 12}
matplotlib.rc("font", **font)


###########################
# Get device information
###########################
cuda_id = "0"
if torch.cuda.is_available():
    dev = f"cuda:{cuda_id}"
else:
    dev = "cpu"
device = torch.device(dev)
print("Device is", device)


# Reproducibility
seed = 42
os.environ["PYTHONHASHSEED"] = str(seed)
# Torch RNG
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
# Python RNG
np.random.seed(seed)
random.seed(seed)
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/awslabs/gluonts/blob/dev/examples/GluonTS_SageMaker_SDK_Tutorial.ipynb

Imports essential libraries for AWS interaction, SageMaker, GluonTS, and data handling. Ensure these are installed before running.

```python
# Third-party requirements
import boto3
import sagemaker
from pathlib import Path
import tempfile

# First-party requirements
from gluonts.nursery.sagemaker_sdk.estimator import GluonTSFramework
from gluonts.mx import SimpleFeedForwardEstimator
from gluonts.dataset.repository.datasets import get_dataset, dataset_recipes
from gluonts.mx.trainer import Trainer
```

--------------------------------

### Train DeepAR Model and Make Predictions

Source: https://github.com/awslabs/gluonts/blob/dev/README.md

This example demonstrates loading air passenger data, splitting it for training and testing, training a DeepAR model, and generating probabilistic forecasts. It requires pandas, matplotlib, and gluonts. The forecasts include 50% and 90% prediction intervals.

```python
import pandas as pd
import matplotlib.pyplot as plt

from gluonts.dataset.pandas import PandasDataset
from gluonts.dataset.split import split
from gluonts.torch import DeepAREstimator

# Load data from a CSV file into a PandasDataset
df = pd.read_csv(
    "https://raw.githubusercontent.com/AileenNielsen/"\
    "TimeSeriesAnalysisWithPython/master/data/AirPassengers.csv",
    index_col=0,
    parse_dates=True,
)
dataset = PandasDataset(df, target="#Passengers")

# Split the data for training and testing
training_data, test_gen = split(dataset, offset=-36)
test_data = test_gen.generate_instances(prediction_length=12, windows=3)

# Train the model and make predictions
model = DeepAREstimator(
    prediction_length=12, freq="M", trainer_kwargs={\"max_epochs\": 5}
).train(training_data)

forecasts = list(model.predict(test_data.input))

# Plot predictions
plt.plot(df["1954":], color="black")
for forecast in forecasts:
  forecast.plot()
plt.legend(["True values"], loc="upper left", fontsize="xx-large")
plt.show()
```

--------------------------------

### Build Project Documentation

Source: https://github.com/awslabs/gluonts/blob/dev/docs/community/devsetup.md

Generate the project's documentation. The output will be in docs/_build/html.

```bash
python setup.py docs
```

--------------------------------

### Generate and Preview Documentation Locally

Source: https://github.com/awslabs/gluonts/blob/dev/docs/community/devsetup.md

Commands to generate the documentation using make and open the generated HTML files in a browser. Ensure no syntax errors or warnings before committing.

```bash
make -C docs html                # generate the docs
open docs/_build/html/index.html # open the generated docs in a browser
```

--------------------------------

### Install GluonTS with R Extra

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs GluonTS with support for R-based models. This requires R and the 'forecast' package to be installed separately.

```sh
pip install "gluonts[R]"
```

--------------------------------

### Initialize GluonTS Framework

Source: https://github.com/awslabs/gluonts/blob/dev/examples/GluonTS_SageMaker_SDK_Tutorial.ipynb

Sets up the GluonTSFramework for SageMaker training. This requires the SageMaker session, IAM role, Docker image URI, base job name, instance type, and dependency paths.

```python
my_experiment = GluonTSFramework(
                    sagemaker_session=sagemaker_session,
                    role=role,
                    image_uri=docker_image,  
                    base_job_name=base_job_description,
                    instance_type=instance_type,
                    dependencies=[my_requirements_txt_file_path],
                    output_path=experiment_parent_dir, # optional, but recommended
                    code_location=experiment_parent_dir, # optional, but recommended
                )
```

--------------------------------

### Build Documentation with Just

Source: https://github.com/awslabs/gluonts/blob/dev/docs/community/devsetup.md

Use the 'just' command to build the entire documentation or compile notebooks to Markdown files.

```bash
just docs
```

```bash
just compile_notebooks skip
```

--------------------------------

### Global Variables and Initialization

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/3. predict/1-2. recursive_store_cat_PREDICT.ipynb

Sets up global variables including version number, random seed, and the number of CPU cores to be used. It also applies the seed to the LightGBM parameters.

```python
########################### Vars
#################################################################################
VER = 1
SEED = 42
seed_everything(SEED)
lgb_params["seed"] = SEED
N_CORES = psutil.cpu_count()



```

--------------------------------

### Work with SampleForecast and QuantileForecast Objects

Source: https://context7.com/awslabs/gluonts/llms.txt

Demonstrates creating and manipulating forecast objects. SampleForecast generates quantiles from samples, while QuantileForecast stores pre-computed quantiles.

```python
import numpy as np
import pandas as pd
from gluonts.model.forecast import SampleForecast, QuantileForecast

start = pd.Period("2024-01-01", freq="D")

# SampleForecast: 200 samples over a 7-day horizon
samples = np.random.randn(200, 7).cumsum(axis=1)
sf = SampleForecast(samples=samples, start_date=start, item_id="ts_1")

print(sf.mean.shape)           # (7,)
print(sf.quantile(0.1).shape)  # (7,)
print(sf.quantile("p90"))      # 90th percentile array
print(sf.median)               # same as sf.quantile(0.5)
print(sf.num_samples)          # 200
print(sf.prediction_length)    # 7

# Convert to QuantileForecast
qf = sf.to_quantile_forecast(quantiles=["0.1", "0.5", "0.9", "mean"])
print(qf.quantile(0.5).shape)  # (7,)
print(qf.mean.shape)           # (7,)
```

--------------------------------

### Global Variables and Initialization

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/3. predict/1-3. recursive_store_dept_PREDICT.ipynb

Sets up global variables including the version number, random seed, and the number of CPU cores to be used. Initializes the random seed for reproducibility.

```python
########################### Vars
#################################################################################
VER = 1
SEED = 42
seed_everything(SEED)
lgb_params["seed"] = SEED
N_CORES = psutil.cpu_count()
```

--------------------------------

### Install R Forecast Package

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs the R 'forecast' and 'nnfor' packages. Required for using R-based forecasting models within GluonTS.

```sh
R -e 'install.packages(c("forecast", "nnfor"), repos="https://cloud.r-project.org")'
```

--------------------------------

### Initialize Ensemble Analyzer

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/examples/evaluate-ensemble-performance.ipynb

Sets up the EnsembleAnalyzer with the initialized ModelTracker for performance analysis.

```python
from tsbench.config import DATASET_REGISTRY, Config
from tsbench.config.model.models import (
    TemporalFusionTransformerModelConfig,
    DeepARModelConfig,
    ARIMAModelConfig,
)
from tsbench.analysis import EnsembleAnalyzer
```

```python
analyzer = EnsembleAnalyzer(tracker)
```

--------------------------------

### Global Variables and Constants

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/2. train/1-1. recursive_store_TRAIN.ipynb

Sets up global variables including version, seed, number of CPU cores, target variable name, training data limits, prediction horizon, and lists of features to remove or use.

```python
## Vars
VER = 1
SEED = 42
seed_everything(SEED)
lgb_params["seed"] = SEED
N_CORES = psutil.cpu_count()


# LIMITS and const
TARGET = "sales"
START_TRAIN = 0
END_TRAIN = 1941 - 28 * KKK
P_HORIZON = 28
USE_AUX = False

remove_features = [
    "id",
    "state_id",
    "store_id",
    "date",
    "wm_yr_wk",
    "d",
    TARGET,
]
mean_features = [
    "enc_cat_id_mean",
    "enc_cat_id_std",
    "enc_dept_id_mean",
    "enc_dept_id_std",
    "enc_item_id_mean",
    "enc_item_id_std",
]

ORIGINAL = raw_data_dir
BASE = processed_data_dir + "grid_part_1.pkl"
PRICE = processed_data_dir + "grid_part_2.pkl"
CALENDAR = processed_data_dir + "grid_part_3.pkl"
LAGS = processed_data_dir + "lags_df_28.pkl"
MEAN_ENC = processed_data_dir + "mean_encoding_df.pkl"



```

--------------------------------

### Install orjson for Faster JSON Parsing

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/install.md

Installs the 'orjson' package for faster JSON parsing. GluonTS recommends this for performance, especially with large datasets.

```sh
pip install orjson
```

--------------------------------

### Download TSBench Datasets

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/README.md

Download all bundled datasets for TSBench using the CLI. Requires a Kaggle account and API token configured.

```bash
bash bin/download-kaggle.sh
```

```bash
# Download and preprocess all datasets
tsbench datasets download
```

```bash
# Upload locally available datasets to your S3 bucket
tsbench datasets upload --bucket <your_bucket_name>
```

--------------------------------

### Global Variables and Initialization

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/3. predict/1-1. recursive_store_PREDICT.ipynb

Sets up global variables including the model version, random seed, and the number of CPU cores to be used for parallel processing. It also initializes the LightGBM seed.

```python
VER = 1
SEED = 42
seed_everything(SEED)
lgb_params["seed"] = SEED
N_CORES = psutil.cpu_count()


```

--------------------------------

### Install GluonTS with Dev Dependencies

Source: https://github.com/awslabs/gluonts/blob/dev/examples/GluonTS_SageMaker_SDK_Tutorial.ipynb

Installs the latest version of GluonTS from the dev branch along with required dependencies. Use this to ensure you have the latest features for SageMaker integration.

```python
!pip install --upgrade mxnet==1.6  git+https://github.com/awslabs/gluonts.git#egg=gluonts[dev]
```

--------------------------------

### Build and Upload Docker Container

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/README.md

Build a Docker container for AWS SageMaker training jobs and upload it to ECR. Ensure an ECR repository named 'tsbench' exists first.

```bash
bash bin/build-container.sh
```

--------------------------------

### Define Data and Log Directories

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/2. train/1-2. recursive_store_cat_TRAIN.ipynb

Set up paths for raw data, processed data, logs, and model storage.

```python
raw_data_dir = dir_ + "2. data/"
processed_data_dir = dir_ + "2. data/processed/"
log_dir = dir_ + "4. logs/"
model_dir = dir_ + "5. models/"
```

--------------------------------

### Train an Estimator to get a Predictor

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/concepts.md

An Estimator is trained using training data to produce a Predictor object, which can then be used for forecasting.

```python
predictor = estimator.train(train_data)
forecasts = predictor.predict(data)
```

--------------------------------

### Initializing Method Dictionary

Source: https://github.com/awslabs/gluonts/blob/dev/examples/demo_spliced_binned_pareto.ipynb

Sets up a dictionary to store results for different forecasting methods. Keys are derived from method names by lowercasing and removing spaces/hyphens.

```python
title_methods = ["Spliced Binned-Pareto", "Gaussian"]

dict_storage = dict(
    zip(
        list(
            map(
                lambda x: x.lower().replace(" ", "").replace("-", ""),
                title_methods
            )
        ),
        list(
            map(
                lambda title_method: dict({}), 
                title_methods
            )
        ),
    )
)
```

--------------------------------

### List Available Datasets

Source: https://github.com/awslabs/gluonts/blob/dev/examples/GluonTS_SageMaker_SDK_Tutorial.ipynb

Prints the keys of available dataset recipes. Use these keys to select a dataset for training.

```python
print(dataset_recipes.keys())
```

--------------------------------

### Import Libraries for GluonTS Forecasting

Source: https://github.com/awslabs/gluonts/blob/dev/examples/COV19-forecast.ipynb

Imports necessary libraries for data manipulation, modeling, and visualization. Ensure these libraries are installed before running.

```python
%matplotlib inline
import mxnet as mx
from mxnet import gluon
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import json
import os
from tqdm.autonotebook import tqdm
from pathlib import Path
```

--------------------------------

### Create a complete configuration object

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/examples/browse-offline-evaluations.ipynb

Combine a model configuration with a dataset configuration from the registry to create a complete Config object. This object is used to fetch specific performance metrics and forecasts.

```python
config = Config(model_config, DATASET_REGISTRY["m4_monthly"]())
```

--------------------------------

### Instantiate and train a global DeepAR model

Source: https://github.com/awslabs/gluonts/blob/dev/docs/getting_started/concepts.md

Example of creating a global DeepAR Estimator and training it with provided data to obtain a Predictor.

```python
# global DeepAR model
estimator = DeepAREstimator(prediction_length=24, freq="H")
predictor = estimator.train(train_data)
```

--------------------------------

### Visualize Results with TensorBoard

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/SCott/README.md

Launches TensorBoard to visualize experiment results stored in the './runs' directory. This command requires specifying a port for access.

```bash
tensorboard --logdir ./runs --port <port>
```

--------------------------------

### Create ListDataset from Python Dictionaries

Source: https://context7.com/awslabs/gluonts/llms.txt

Converts an iterable of Python dictionaries into a GluonTS Dataset. Each dictionary must contain 'start' and 'target' keys.

```python
import pandas as pd
from gluonts.dataset.common import ListDataset

raw_data = [
    {"start": "2021-01-01", "target": [1.0, 2.5, 3.1, 4.7, 5.0, 6.3]},
    {"start": "2021-01-01", "target": [10.0, 12.0, 11.5, 13.0, 14.2, 15.0],
     "feat_static_cat": [0]},
]

dataset = ListDataset(raw_data, freq="D")
for entry in dataset:
    print(entry["start"], entry["target"].shape, entry["target"].dtype)
# 2021-01-01  (6,)  float32
# 2021-01-01  (6,)  float32
```

--------------------------------

### Get all available evaluations

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/examples/browse-offline-evaluations.ipynb

Retrieve all available offline evaluations managed by the tracker. This provides access to the performance data across different datasets and models.

```python
evaluations = tracker.get_evaluations()
```

--------------------------------

### Define Forecaster via Hyper-parameter

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/shell/README.md

Example of passing the forecaster class as a hyper-parameter within a configuration object. The entire class path must be specified.

```JSON
{
    ...
    "forecaster_name": "gluonts.model.deepar.DeepAREstimator",
    ...

}
```

--------------------------------

### Define Feature Columns

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/3. predict/2-3. nonrecursive_store_dept_PREDICT.ipynb

Specifies the starting day for analysis and lists columns to be removed from the dataset. It also defines categorical variables for feature engineering.

```python
FIRST_DAY = 710
remove_feature = [
    "id",
    "state_id",
    "store_id",
    #                   'item_id',
    "dept_id",
    "cat_id",
    "date",
    "wm_yr_wk",
    "d",
    "sales",
]

cat_var = ["item_id", "dept_id", "store_id", "cat_id", "state_id"] + [
    "event_name_1",
    "event_name_2",
    "event_type_1",
    "event_type_2",
]
cat_var = list(set(cat_var) - set(remove_feature))
```

--------------------------------

### Run Project Tests

Source: https://github.com/awslabs/gluonts/blob/dev/docs/community/devsetup.md

Execute the project's tests using pytest or the setup.py script.

```bash
pytest
```

```bash
python setup.py tests
```

--------------------------------

### Training Phase Marker

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/2. train/2-1. nonrecursive_store_TRAIN.ipynb

This marker indicates the start of the 'TX_2' phase in the training process. It is used to delineate different stages of the model's development.

```text
TX_2 start
```

--------------------------------

### Import DeepARModelConfig

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/examples/browse-offline-evaluations.ipynb

Import the DeepARModelConfig class to define specific configurations for the DeepAR model. This is used when retrieving detailed metrics for a particular model setup.

```python
from tsbench.config.model.models import DeepARModelConfig
```

--------------------------------

### Download Evaluations with Forecasts

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/examples/evaluate-ensemble-performance.ipynb

Use this command to download publicly available evaluations along with their stored forecasts. Ensure you have sufficient local storage (approx. 600 GiB).

```bash
tsbench evaluations download --include_forecasts
```

--------------------------------

### Initialize Global Variables and Seed

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/2. train/1-2. recursive_store_cat_TRAIN.ipynb

Set the version, seed, and number of CPU cores for the training process.

```python
########################### Vars
VER = 1
SEED = 42
seed_everything(SEED)
lgb_params["seed"] = SEED
N_CORES = psutil.cpu_count()
```

--------------------------------

### Memory Profiling Functions

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/1. preprocessing/1. preprocessing.ipynb

Provides utility functions to get current memory usage and format byte sizes. These are helpful for monitoring and optimizing memory consumption.

```python
## Simple "Memory profilers" to see memory usage
def get_memory_usage():
    return np.round(
        psutil.Process(os.getpid()).memory_info()[0] / 2.0**30, 2
    )


def sizeof_fmt(num, suffix="B"):
    for unit in ["", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi"]:
        if abs(num) < 1024.0:
            return "%3.1f%s%s" % (num, unit, suffix)
        num /= 1024.0
    return "%.1f%s%s" % (num, "Yi", suffix)
```

--------------------------------

### Predicting Sales Data

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/3. predict/2-3. nonrecursive_store_dept_PREDICT.ipynb

This code block indicates the start and end of the prediction process for a specific store and department. It's used to track the progress of generating forecasts.

```python
CA_4 FOODS_3 start
starting to predict
done predicting
TX_1 HOBBIES_1 start
starting to predict
done predicting
TX_1 HOBBIES_2 start
done predicting
TX_1 FOODS_1 start
starting to predict
done predicting
TX_1 FOODS_2 start
starting to predict
done predicting
TX_1 FOODS_3 start
starting to predict
done predicting
TX_2 FOODS_1 start
starting to predict
done predicting
TX_2 FOODS_2 start
starting to predict
done predicting
TX_2 FOODS_3 start
starting to predict
```

--------------------------------

### Load Sample Submission and Extract IDs

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/3. predict/3-1. Final ensemble.ipynb

Reads the sample submission CSV to extract relevant IDs for filtering. It specifically selects IDs starting from the 30490th row.

```python
submission = pd.read_csv(raw_data_dir + "sample_submission.csv")
ids = pd.DataFrame({"id": submission.iloc[30490:]["id"]})
```

--------------------------------

### Set up computation device

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/SCott/examples/Multivariate-Flow-Solar.ipynb

Determines whether to use a CUDA-enabled GPU or the CPU for computations.

```python
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

--------------------------------

### Define entry point script path

Source: https://github.com/awslabs/gluonts/blob/dev/examples/GluonTS_SageMaker_SDK_Tutorial.ipynb

Locates the default entry point script provided by GluonTS for SageMaker.

```python
run_entry_point_path = (
    Path(os.path.dirname(gluonts.__file__)) 
    / "nursery" 
    / "sagemaker_sdk" 
    / "entry_point_scripts" 
    / "run_entry_point.py"
)
```

--------------------------------

### Import configuration and dataset registry

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/examples/browse-offline-evaluations.ipynb

Import the Config class and DATASET_REGISTRY from tsbench.config. These are needed to create a complete configuration object that includes both model and dataset details.

```python
from tsbench.config import DATASET_REGISTRY, Config
```

--------------------------------

### Initialize TransformerTempFlowEstimator

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/SCott/examples/Multivariate-Flow-Solar.ipynb

Initializes the TransformerTempFlowEstimator with specified hyperparameters for multivariate time series forecasting. Ensure the device, epochs, learning rate, and batch size are configured appropriately for your training setup.

```python
estimator = TransformerTempFlowEstimator(
    d_model=16,
    num_heads=4,
    input_size=552,
    target_dim=int(dataset.metadata.feat_static_cat[0].cardinality),
    prediction_length=dataset.metadata.prediction_length,
    context_length=dataset.metadata.prediction_length * 4,
    flow_type="MAF",
    dequantize=True,
    freq=dataset.metadata.freq,
    trainer=Trainer(
        device=device,
        epochs=14,
        learning_rate=1e-3,
        num_batches_per_epoch=100,
        batch_size=64,
    ),
)
```

--------------------------------

### Model Configuration Constants

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/QRX-Wrapped-M5-Accuracy-Solution/3. code/2. train/1-1. recursive_store_TRAIN.ipynb

Sets up versioning and a list of store IDs to be processed. The `KKK` variable likely controls the training data split or horizon.

```python
ver, KKK = "priv", 0
STORES_IDS = [
    "CA_1",
    "CA_2",
    "CA_3",
    "CA_4",
    "TX_1",
    "TX_2",
    "TX_3",
    "WI_1",
    "WI_2",
    "WI_3",
]
```

--------------------------------

### Initialize MLPSurrogate

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/nursery/tsbench/examples/train-a-recommender.ipynb

Trains a surrogate model (MLP) using the ModelTracker to predict performance metrics like nCRPS and latency. This model learns from all available evaluations to guide future recommendations.

```python
from tsbench.surrogate import MLPSurrogate
```

```python
surrogate = MLPSurrogate(
    tracker,
    objective="ranking",
    discount="linear",
    hidden_layer_sizes=[32, 32],
    predict=["ncrps_mean", "latency_mean"],
)
```

--------------------------------

### Build a Docker Container with GluonTS Shell

Source: https://github.com/awslabs/gluonts/blob/dev/src/gluonts/shell/README.md

A minimal Dockerfile to build a container with GluonTS and its shell dependencies installed. Ensures the shell module is set as the entry point for training and serving commands.

```Dockerfile
FROM python:3.7

# "[shell]" ensures that shell-dependencies are installed
RUN pip install gluonts[shell]

# This line is crucial. We need to set `gluonts.shell` as entry-point to
# dispatch the `train` and `serve` commands.
ENTRYPOINT ["python", "-m", "gluonts.shell"]
```