### Initialize Optimizer and Scheduler

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Configures model parameters for optimization and sets up learning rate schedulers. Supports linear, cosine, and cosine with hard restarts scheduling strategies.

```python
def get_optimizer_params(model, encoder_lr, decoder_lr, layerwise_learning_rate_decay, weight_decay=0.0):
    param_optimizer = list(model[0].named_parameters()) + list(model[1].named_parameters())
    no_decay = ["bias", "LayerNorm.bias", "LayerNorm.weight"]
    optimizer_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'lr': encoder_lr, 'weight_decay': weight_decay},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'lr': encoder_lr, 'weight_decay': 0.0},
    ]
    return optimizer_parameters

def get_scheduler(cfg, optimizer, len_train_folds):
    num_train_steps = int(len_train_folds / cfg.batch_size * cfg.epochs)
    if cfg.scheduler == 'linear':
        scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=cfg.num_warmup_steps, num_training_steps=num_train_steps)
    elif cfg.scheduler == 'cosine':
        scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=cfg.num_warmup_steps, num_training_steps=num_train_steps, num_cycles=cfg.num_cycles)
    elif cfg.scheduler == 'cosine_hard':
        scheduler = get_cosine_with_hard_restarts_schedule_with_warmup(optimizer, num_warmup_steps=cfg.num_warmup_steps, num_training_steps=num_train_steps, num_cycles=cfg.num_cycles)
    return scheduler
```

--------------------------------

### Install Project Dependencies via Pip

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Executes shell commands to install required machine learning frameworks and local package wheels. This ensures the environment matches the project's specific dependency requirements.

```bash
!pip install -q torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 timm==0.9.12
!pip install "/kaggle/input/pytorch-2-1-3/pytorch_lightning-2.1.3-py3-none-any.whl"
!pip install "/kaggle/input/albumentation-package/albumentations-1.4.0-py3-none-any.whl"
```

--------------------------------

### Configure Optimizers and Schedulers

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Logic to configure model optimizers, including support for 8-bit Adam via bitsandbytes, and dynamic learning rate scheduling based on training folds.

```python
def configure_optimizers(self):
    optimizer_parameters = get_optimizer_params([self.backbone1, self.backbone2], 
                                                encoder_lr=CFG.encoder_lr, 
                                                decoder_lr=CFG.decoder_lr)
    if self.cfg.use_8bit_optimizer:
        import bitsandbytes as bnb
        optimizer = bnb.optim.Adam8bit(optimizer_parameters, lr=CFG.encoder_lr)
    else:
        optimizer = AdamW(optimizer_parameters, lr=CFG.encoder_lr)
    
    scheduler = get_scheduler(CFG, optimizer, len(self.train_folds))
    return {'optimizer': optimizer, 'lr_scheduler': {"scheduler": scheduler, "interval": CFG.sch_interval}}
```

--------------------------------

### Install Specific Transformers and Tokenizers Versions

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/12- Feedback Prize - English Language Learning/Training Notebook.ipynb

Uninstalls existing versions of 'transformers' and 'tokenizers' and then installs specific versions from local wheel files. This ensures compatibility and reproducibility in the Kaggle environment.

```python
import os
os.system('pip uninstall -y transformers')
os.system('pip uninstall -y tokenizers')
os.system('python -m pip install --no-index --find-links=../input/fb3-pip-wheels transformers')
os.system('python -m pip install --no-index --find-links=../input/fb3-pip-wheels tokenizers')

import tokenizers
import transformers
print(f"tokenizers.__version__: {tokenizers.__version__}")
print(f"transformers.__version__: {transformers.__version__}")
from transformers import AutoTokenizer, AutoModel, AutoConfig
```

--------------------------------

### Environment Setup and Dependency Installation

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/10- RSNA 2024 Lumbar Spine Degenerative Classification/Inference Notebook - Ensemble.ipynb

Installs the Ultralytics library in an offline Kaggle environment and imports necessary data processing and machine learning libraries.

```python
!pip install -q --no-index --find-links /kaggle/input/ultralytics ultralytics

import os
import pydicom
from PIL import Image
import numpy as np
from multiprocessing import Pool, cpu_count

import sklearn.metrics
import torch
import cv2
import pandas as pd 
from tqdm.auto import tqdm
```

--------------------------------

### Install Albumentations via pip

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Installs the Albumentations library from a local wheel file within the Kaggle environment, ensuring all dependencies like OpenCV and Scikit-Image are satisfied.

```bash
pip install /kaggle/input/albumentations-package/albumentations-1.4.0-py3-none-any.whl
```

--------------------------------

### Initialize Model and Data Loaders for TTA

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Sets up the PyTorch model and multiple DataLoader instances configured with different augmentation transforms for Test-Time Augmentation (TTA).

```python
model = CustomModel_4(cfg=CFG, criterion=None, train_folds=None, valid_folds=None, fold=None, pretrained=CFG.pretrained)

valid_dataset_original = TrainDataset(CFG, data=test_df, train=False, transform=get_transforms('valid', CFG))
valid_loader_original = DataLoader(valid_dataset_original, batch_size=CFG.valid_batch_size, shuffle=False, num_workers=CFG.num_workers, drop_last=False)

valid_dataset_vflip = TrainDataset(CFG, data=test_df, train=False, transform=get_transforms_TTA(aug_vflip))
valid_loader_vflip = DataLoader(valid_dataset_vflip, batch_size=CFG.valid_batch_size, shuffle=False, num_workers=CFG.num_workers, drop_last=False)

valid_dataset_time_shift = TrainDataset(CFG, data=test_df, train=False, transform=get_transforms_TTA(aug_time_shift))
valid_loader_time_shift = DataLoader(valid_dataset_time_shift, batch_size=CFG.valid_batch_size, shuffle=False, num_workers=CFG.num_workers, drop_last=False)
```

--------------------------------

### Import Computer Vision and Custom Metric Libraries

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Imports specialized computer vision libraries including OpenCV and Timm, and configures the system path to include custom evaluation scripts for Kaggle competitions.

```python
import cv2
import albumentations as A
import timm
sys.path.append('/kaggle/input/kaggle-kl-div')
from kaggle_kl_div import score
```

--------------------------------

### Initialize WaveNet Model (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Initializes the WaveNet model with specified input channels and kernel size. It sets up two sequential models (model1, model2) composed of Wave_Block layers and a list of GRU layers for recurrent processing.

```python
def __init__(self, input_channels: int = 1, kernel_size: int = 3):
        super(WaveNet, self).__init__()
        self.model1 = nn.Sequential(
                Wave_Block(input_channels, 8, 12, kernel_size),
                Wave_Block(8, 16, 8, kernel_size),
                Wave_Block(16, 32, 4, kernel_size),
                Wave_Block(32, 64, 1, kernel_size) 
        )
        self.model2 = nn.Sequential(
                Wave_Block(input_channels, 8, 12, kernel_size),
                Wave_Block(8, 16, 8, kernel_size),
                Wave_Block(16, 32, 4, kernel_size),
                Wave_Block(32, 64, 1, kernel_size) 
        )
        self.rnn = nn.ModuleList([
                nn.GRU(64, 64, bidirectional=True),
                nn.GRU(128, 64, bidirectional=True),
                nn.GRU(128, 64, bidirectional=True),
                nn.GRU(128, 64, bidirectional=True),
        ])
```

--------------------------------

### Initialize Model and Data Loaders for TTA

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Initializes a custom PyTorch model and sets up multiple DataLoader instances with specific transformations for Test-Time Augmentation. This prepares the environment for the inference loop.

```python
model = CustomModel_3(cfg=CFG,
                    criterion=None,
                    train_folds=None,
                    valid_folds=None,
                    fold=None,
                    pretrained=CFG.pretrained)

valid_dataset_original = TrainDataset(CFG, data=test_df, train=False, 
                             transform=get_transforms('valid', CFG))
valid_loader_original = DataLoader(valid_dataset_original,
                          batch_size=CFG.valid_batch_size,
                          shuffle=False,
                          num_workers=CFG.num_workers,
                          drop_last=False)

valid_dataset_vflip = TrainDataset(CFG, data=test_df, train=False, 
                             transform=get_transforms_TTA(aug_vflip))
valid_loader_vflip = DataLoader(valid_dataset_vflip,
                          batch_size=CFG.valid_batch_size,
                          shuffle=False,
                          num_workers=CFG.num_workers,
                          drop_last=False)

valid_dataset_time_shift = TrainDataset(CFG, data=test_df, train=False, 
                             transform=get_transforms_TTA(aug_time_shift))
valid_loader_time_shift = DataLoader(valid_dataset_time_shift,
                          batch_size=CFG.valid_batch_size,
                          shuffle=False,
                          num_workers=CFG.num_workers,
                          drop_last=False)
```

--------------------------------

### Configure Weights & Biases Integration

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/12- Feedback Prize - English Language Learning/Training Notebook.ipynb

Initializes W&B for experiment tracking using Kaggle secrets for authentication. It converts the configuration class to a dictionary and starts a new run.

```python
if CFG.wandb:
    import wandb
    from kaggle_secrets import UserSecretsClient
    
    user_secrets = UserSecretsClient()
    secret_value_0 = user_secrets.get_secret("wandb_api")
    wandb.login(key=secret_value_0)

    def class2dict(f):
        return dict((name, getattr(f, name)) for name in dir(f) if not name.startswith('__'))

    run = wandb.init(project='FB3-Public', name=CFG.model, config=class2dict(CFG))
```

--------------------------------

### Install Offline Dependencies and Configure Environment

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/09- RSNA 2022 Cervical Spine Fracture Detection/Training Notebook.ipynb

This snippet installs necessary Python packages from local wheel files to ensure compatibility in an offline Kaggle environment. It also sets up the directory for pre-trained model checkpoints.

```python
try:
    import pylibjpeg
except:
    !mkdir -p /root/.cache/torch/hub/checkpoints/
    !cp ../input/rsna-2022-whl/efficientnet_v2_s-dd5fe13b.pth  /root/.cache/torch/hub/checkpoints/
    !pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}
    !pip install /kaggle/input/rsna-2022-whl/{torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl,torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl}
```

--------------------------------

### Initialize Environment and Load OOF Predictions

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Sets up the local output directory and reads multiple CSV files containing model predictions. Each CSV is cleaned by renaming string-based column headers to integers and extracting the underlying data as a NumPy array.

```python
import os
import pandas as pd

OUTPUT_DIR = './'
if not os.path.exists(OUTPUT_DIR):
    os.makedirs(OUTPUT_DIR)
    
# Example loading pattern for model predictions
oof_5 = pd.read_csv('/kaggle/input/hms-ckpts-3/mobilenet/mobilenet_Reverted2Stages.csv').rename({f'{i}':i for i in range(6)},axis=1).values
oof_20 = pd.read_csv('/kaggle/input/hms-ckpts-2/wavenet/wavenet_PLs/wavenet_PLs_oof.csv').rename({f'{i}':i for i in range(6)},axis=1).values[:,1:]
```

--------------------------------

### Configure PyTorch DataLoader for Validation

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Initializes a PyTorch DataLoader for validation datasets. It utilizes configuration parameters for batch size, worker count, and shuffling to ensure consistent evaluation.

```python
valid_loader_xy_mask = DataLoader(valid_dataset_xy_mask,
                          batch_size=CFG.valid_batch_size,
                          shuffle=False,
                          num_workers=CFG.num_workers,
                          drop_last=False)
```

--------------------------------

### Load and Prepare EEG Data for Kaggle Competition

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Loads EEG data from CSV files, renames columns for consistency, and prepares it for further processing. It handles both ground truth labels and predictions.

```python
oof_27 = pd.read_csv('/kaggle/input/temp-oof/wavenet_Reverted2Stages.csv').rename({f'{i}':i for i in range(6)},axis=1).values   ## wavenet Reverted 2 Stages

y = pd.read_csv('/kaggle/input/hms-ckpts-2/y_true.csv').rename({f'{i}':i for i in range(6)},axis=1).values

subs = [oof_5,oof_6,oof_8,oof_9,oof_10,oof_12,oof_14,oof_15,oof_16,oof_18,oof_19,oof_20]
```

--------------------------------

### Initialize Data Science and Deep Learning Environment

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Imports standard data processing, visualization, and machine learning libraries. It configures global settings for Pandas and Matplotlib, and initializes PyTorch and PyTorch Lightning modules for model training.

```python
import os
import gc
import re
import ast
import sys
import copy
import json
import time
import math
import string
import pickle
import random
import joblib
import itertools
import warnings
warnings.filterwarnings("ignore")

import scipy as sp
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
plt.rcParams["figure.figsize"] = (12, 8)
from glob import glob
from tqdm.auto import tqdm
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

import torch
import torch.nn as nn
from torch.nn import Parameter
from torch.optim import Optimizer
import torch.nn.init as init
import torch.nn.functional as F
from torch.autograd.function import InplaceFunction
from torch.optim import Adam, SGD, AdamW
from torch.utils.data import DataLoader, Dataset
import torchvision.models as models

import pytorch_lightning as pl
from pytorch_lightning import LightningModule, Trainer, seed_everything
from pytorch_lightning.tuner.tuning import Tuner
from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping 
from pytorch_lightning.loggers import CSVLogger

from transformers import get_linear_schedule_with_warmup, get_cosine_schedule_with_warmup

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

--------------------------------

### Define Wavelet Transform Parameters

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Sets up parameters for wavelet feature extraction. It defines the wavelet to be used (currently set to None), names for the resulting features, and lists of EEG channels to be used for each feature.

```python
import pywt, librosa

USE_WAVELET = None 

NAMES = ['LL','LP','RP','RR']

FEATS = [['Fp1','F7','T3','T5','O1'],
         ['Fp1','F3','C3','P3','O1'],
         ['Fp2','F8','T4','T6','O2'],
         ['Fp2','F4','C4','P4','O2']]
```

--------------------------------

### Perform Dual-Backbone Feature Fusion in PyTorch

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

This snippet demonstrates the forward pass of a model utilizing two backbones. It extracts features, applies head processing, performs element-wise multiplication for fusion, and outputs the final classification.

```python
features1 = self.backbone1.forward_features(x1)
features2 = self.backbone2.forward_features(x2)

features1 = self.backbone1.forward_head(features1, pre_logits = True)
features2 = self.backbone2.forward_head(features2, pre_logits = True)

# Concatenate the features from both backbones
combined_features = features1 * features2

# Apply the final classification layer
output = self.fc(combined_features)

return output
```

--------------------------------

### Load Test Data and Define Targets

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Loads the test dataset from a CSV file and defines the target column names for the classification task. It prints the shape of the test data and displays the first few rows.

```python
# ====================================================
# Data Loading
# ====================================================
def = pd.read_csv(TEST_PATH + 'test.csv')
TARGETS = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']

print(f"test.shape: {df.shape}")
display(df.head())
```

--------------------------------

### Machine Learning Configuration Class (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Defines a configuration class `CFG` for a machine learning project, likely for a Kaggle competition. It encapsulates parameters related to the competition, data loading, training, optimization, scheduling, cross-validation, and augmentation. This class helps in organizing and managing hyperparameters.

```python
class CFG:
    competition = 'HMS'   # Competition Name
    seed = 42
########################################################################################################
    # Data
    batch_size = 32        
    valid_batch_size = 2 * batch_size
    num_workers = os.cpu_count()       # Threads in Data Loader
    target_cols = TARGETS
########################################################################################################
    # Training
    pretrained = False
    epochs = 4
########################################################################################################
    # Optimizer
    encoder_lr = 0.0013         # Pretrained Model lr  
    decoder_lr = 1.5e-5         # Custom Model lr  
    
    layerwise_learning_rate_decay = 1.0  
    eps = 1e-6                # Adam Parameters 
    betas=(0.9, 0.999)        # Adam Parameters
    weight_decay = 0.02
    lr_finder=False
########################################################################################################
    # Scheduler
    use_scheduler = True    # Use Scheduler
    scheduler = 'cosine'      # 'cosine' or 'linear' or 'cosine_hard'
    num_cycles = 0.5
    num_warmup_steps = 0
    sch_interval = 'step'    # 'step' or 'epoch'
########################################################################################################
    # CV
    n_fold=5
    trn_fold=list(range(n_fold))
########################################################################################################
    # Augmentation
    train_aug_list = [
    ]
    
    valid_aug_list = [
    ]

```

--------------------------------

### Training and Validation Steps with PyTorch Lightning (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Defines the training_step and validation_step methods for the CustomModel. These methods handle a single batch of data, compute predictions, calculate loss using a specified criterion, and log metrics. Dependencies include PyTorch, PyTorch Lightning, and F.log_softmax/F.softmax.

```python
def training_step(self, batch, batch_idx):
    inputs, labels = batch[0], batch[1]
    
    y_preds = self(inputs)
    y_preds = F.log_softmax(y_preds, dim=1)
    loss = self.criterion(y_preds, labels)

    self.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True, logger=True)

    for param_group in self.trainer.optimizers[0].param_groups:
        lr = param_group["lr"]
    self.log("lr", lr, on_step=True, on_epoch=False, prog_bar=True)

    return loss


def validation_step(self, batch, batch_idx):
    inputs, labels = batch[0], batch[1]
    
    y_preds = self(inputs)
    y_preds = F.softmax(y_preds, dim=1)
    loss = self.criterion(y_preds, labels)

    self.log('val_loss', loss, on_step=False, on_epoch=True, prog_bar=True, logger=True)
    self.val_step_outputs.append(y_preds)
    self.val_step_labels.append(labels)

    return loss
```

--------------------------------

### Load Test Dataset and DataLoader for TTA

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

This snippet demonstrates how to load a test dataset using a custom `TrainDataset` class and create a `DataLoader` for test-time augmentation (TTA). It specifies batch size, shuffling, number of workers, and whether to drop the last batch.

```python
valid_dataset_xy_mask = TrainDataset(CFG, data=test_df, train=False, 
                             transform=get_transforms_TTA(aug_xy_mask))
valid_loader_xy_mask = DataLoader(valid_dataset_xy_mask,
                          batch_size=CFG.valid_batch_size,
                          shuffle=False,
                          num_workers=CFG.num_workers,
                          drop_last=False)
```

--------------------------------

### Execute Training Pipeline (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/15- Novozymes Enzyme Stability Prediction/Training Notebook - ThermoNet.ipynb

This snippet demonstrates how to initiate the training process by setting up parameters and calling the run_train function. It utilizes predefined best parameters and specifies the project name for logging. Dependencies include the copy module for parameter copying.

```python
if TRAIN:
    params = copy.copy(BEST_PARAMS)
    thermonet_models = run_train(WANDB_TRAIN_NAME, params, project=WANDB_TRAIN_PROJECT)[0]
```

--------------------------------

### Install Iterative Stratification with Pip

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/12- Feedback Prize - English Language Learning/Training Notebook.ipynb

Installs the 'iterative-stratification' package with a specific version (0.1.7) using pip. This is a common step in machine learning projects requiring advanced stratification techniques.

```python
import os
os.system('pip install iterative-stratification==0.1.7')
from iterstrat.ml_stratifiers import MultilabelStratifiedKFold
```

--------------------------------

### Import Dependencies and Configure Environment

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/15- Novozymes Enzyme Stability Prediction/Training Notebook - ProtBert.ipynb

Imports essential data science and deep learning libraries, sets pandas display options, and initializes the device (GPU/CPU) for PyTorch operations.

```python
import os, gc, re, ast, sys, copy, json, time, math, string, pickle, random, joblib, itertools, warnings
import scipy as sp, numpy as np, pandas as pd
import torch, torch.nn as nn, torch.nn.functional as F
from torch.utils.data import DataLoader, Dataset
from transformers import AutoTokenizer, AutoModel, AutoConfig

%env TOKENIZERS_PARALLELISM=true
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

--------------------------------

### Configure Output Paths

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/15- Novozymes Enzyme Stability Prediction/Training Notebook - ProtBert.ipynb

Sets the output directory and configuration file path for the model training or inference process.

```python
CFG.path = OUTPUT_DIR
CFG.config_path = CFG.path+'config.pth'
```

--------------------------------

### PyTorch Lightning Module for Kaggle Competitions

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Defines a custom PyTorch Lightning Module for Kaggle competitions. It initializes two efficientnet backbones, sets up data loaders for training and validation, and defines the training, validation, and forward passes. Dependencies include PyTorch, PyTorch Lightning, timm, and custom utility functions for datasets, transforms, optimizers, and scoring.

```python
class CustomModel_4(LightningModule):
    def __init__(self, cfg, criterion, train_folds, valid_folds, fold, pretrained=False):
        super().__init__()
        self.cfg = cfg
        self.criterion = criterion
        self.train_folds = train_folds
        self.valid_folds = valid_folds
        self.fold = fold
        self.pretrained = pretrained
        self.val_step_outputs = []
        self.val_step_labels = []
        if self.cfg.lr_finder:
            self.lr = CFG.encoder_lr

        self.backbone1 = timm.create_model('efficientnet_b1', pretrained=False, in_chans=1)
        self.backbone2 = timm.create_model('efficientnet_b1', pretrained=False, in_chans=4)
    
        in_features = self.backbone1.num_features 
        self.fc = nn.Linear(in_features, len(CFG.target_cols))
                    
    def train_dataloader(self):
        train_dataset = TrainDataset(CFG, self.train_folds, transform=get_transforms('train', CFG))
        train_loader = DataLoader(train_dataset,
                                  batch_size=CFG.batch_size,
                                  shuffle=True,
                                  num_workers=CFG.num_workers,
                                  drop_last=True,
                                  pin_memory=True,
                                 )
        return train_loader
    
    
    def val_dataloader(self):
        valid_dataset = TrainDataset(CFG, self.valid_folds, transform=get_transforms('valid', CFG))
        valid_loader = DataLoader(valid_dataset,
                                  batch_size=CFG.valid_batch_size,
                                  shuffle=False,
                                  num_workers=CFG.num_workers,
                                  drop_last=False,
                                 )
        return valid_loader
    
    def training_step(self, batch, batch_idx):
        inputs, labels = batch[0], batch[1]
        
        y_preds = self(inputs)
        y_preds = F.log_softmax(y_preds, dim=1)
        loss = self.criterion(y_preds, labels)

        self.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True, logger=True)

        for param_group in self.trainer.optimizers[0].param_groups:
            lr = param_group["lr"]
        self.log("lr", lr, on_step=True, on_epoch=False, prog_bar=True)

        return loss

    
    def validation_step(self, batch, batch_idx):
        inputs, labels = batch[0], batch[1]
        
        y_preds = self(inputs)
        y_preds = F.softmax(y_preds, dim=1)
        loss = self.criterion(y_preds, labels)

        self.log('val_loss', loss, on_step=False, on_epoch=True, prog_bar=True, logger=True)
        self.val_step_outputs.append(y_preds)
        self.val_step_labels.append(labels)

        return loss

        
    def configure_optimizers(self):
        optimizer_parameters = get_optimizer_params([self.backbone1,self.backbone2],
                                                    encoder_lr=CFG.encoder_lr,
                                                    decoder_lr=CFG.decoder_lr,
                                                    layerwise_learning_rate_decay=CFG.layerwise_learning_rate_decay,
                                                    weight_decay=CFG.weight_decay)
        
        if self.cfg.use_8bit_optimizer:
            import bitsandbytes as bnb
            optimizer = bnb.optim.Adam8bit(optimizer_parameters,
                  lr=CFG.encoder_lr,
                  eps=CFG.eps,
                  betas=CFG.betas)
        else:
            optimizer = AdamW(optimizer_parameters,
                              lr=CFG.encoder_lr,
                              eps=CFG.eps,
                              betas=CFG.betas)
        
        scheduler = get_scheduler(CFG, optimizer, len(self.train_folds))
        lr_scheduler_dict = {"scheduler": scheduler, "interval": CFG.sch_interval}
        if self.cfg.use_scheduler:
            return {'optimizer': optimizer, 'lr_scheduler': lr_scheduler_dict}
        else:
            return {'optimizer': optimizer}
    
    def on_validation_epoch_end(self):
        all_preds = torch.cat(self.val_step_outputs)
        all_labels = torch.cat(self.val_step_labels)
        self.val_step_outputs.clear()
        self.val_step_labels.clear()
        
        val_KLD = get_score(all_preds, all_labels)
        self.log("KLD_val", val_KLD, on_step=False, on_epoch=True, prog_bar=True)
        
        if self.trainer.global_rank == 0:
            print(f"\nEpoch: {self.current_epoch}, KLD_val: {val_KLD}", flush=True)
           
    
    # The Model Architicture
    def forward(self, x):
        
        x1 = [x[:, :, :, i:i+1] for i in range(4)]
        x1 = torch.concat(x1, dim=1)

        x2 = [x[:, :, :, i+4:i+5] for i in range(4)]
        x2 = torch.concat(x2, dim=3)

        x1 = x1.permute(0, 3, 1, 2)
        x2 = x2.permute(0, 3, 1, 2)
        
```

--------------------------------

### Custom Time-Shift Augmentation and TTA Pipeline

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Defines a custom Albumentations transform for time-shifting spectrogram data and a factory function for generating Test-Time Augmentation (TTA) pipelines. This setup allows for consistent evaluation across various augmented versions of the input dataset.

```python
class TimeShift(ImageOnlyTransform):
    def __init__(self, shift_max, p=0.5):
        super(TimeShift, self).__init__(always_apply=False, p=p)
        self.shift_max = shift_max

    def apply(self, img, **params):
        shift = np.random.uniform(-self.shift_max, self.shift_max)
        shift = int(shift * img.shape[1])
        if shift > 0:
            img = np.pad(img, ((0, 0), (shift, 0), (0, 0)), mode='constant')[:, :-shift, :]
        elif shift < 0:
            img = np.pad(img, ((0, 0), (0, -shift), (0, 0)), mode='constant')[:, -shift:, :]
        return img

    def get_transform_init_args_names(self):
        return ("shift_max",)

def get_transforms_TTA(augment_list):
    aug = A.Compose(augment_list)
    return aug
```

--------------------------------

### Initialize Output Directory

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/15- Novozymes Enzyme Stability Prediction/Training Notebook - ProtBert.ipynb

Creates a versioned output directory for storing experiment results. It checks for the existence of the directory before creation to prevent errors.

```python
import os

OUTPUT_DIR = f'Debug_{VER}/'
if not os.path.exists(OUTPUT_DIR):
    os.makedirs(OUTPUT_DIR)
```

--------------------------------

### Calculate Mean Predictions using NumPy

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Computes the arithmetic mean of prediction arrays along the specified axis. This is typically used to ensemble predictions from multiple models or folds.

```python
predictions_63 = np.mean(predictions_63, axis=0)
predictions_64 = np.mean(predictions_64, axis=0)
```

--------------------------------

### Custom Model Forward Pass Logic

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Defines the forward pass logic for a multi-input model, where features are processed through a shared model, pooled, and concatenated before passing through a final head layer.

```python
x1 = self.model(x[:, :, 2:3])
x1 = self.global_avg_pooling(x1)
x1 = x1.squeeze(dim=2)
x2 = self.model(x[:, :, 3:4])
x2 = self.global_avg_pooling(x2)
x2 = x2.squeeze(dim=2)
z2 = torch.mean(torch.stack([x1, x2]), dim=0)
# ... (repeated for other slices)
y = torch.cat([z1, z2, z3, z4], dim=-1)
y = self.head(y)
return y
```

--------------------------------

### Training Arguments Setup (Python)

Source: https://context7.com/mohammad2012191/kaggle_competitions_solutions/llms.txt

Defines the training arguments for a sequence classification task using the Hugging Face Transformers library. It specifies the output directory, number of training epochs, batch size, learning rate, warmup steps, gradient checkpointing, and optimizer.

```python
# Training arguments
training_args = TrainingArguments(
    output_dir="Gemma2_QLoRA_ft",
    num_train_epochs=Config.n_epochs,
    per_device_train_batch_size=Config.per_device_train_batch_size,
    learning_rate=Config.lr,
    warmup_steps=20,
    gradient_checkpointing=True,
    optim="paged_adamw_32bit",
)
```

--------------------------------

### Define TrainDataset Class

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

A PyTorch Dataset implementation for processing spectrogram and EEG data. It handles data loading, log transformation, standardization, and optional augmentation for training or testing phases.

```python
class TrainDataset(Dataset):
    def __init__(self, cfg, data, train=True, transform=None, spectrograms=spectrograms, all_eegs=all_eegs):
        self.data  = data
        self.transforms = transform
        self.train = train
        self.cfg = cfg
        self.specs = spectrograms
        self.eeg_specs = all_eegs
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, index):
        return self.__getitems__([index])
    
    def __getitems__(self,indices):
        X, y = self._generate_data(indices)
        if self.train:
            if self.transforms:
                for i in range(X.shape[0]):
                    X[i,] = self.transforms(image=X[i,])['image']
            return list(zip(X, y))
        else:
            if self.transforms:
                for i in range(X.shape[0]):
                    X[i,] = self.transforms(image=X[i,])['image']
            return X
```

--------------------------------

### Calculate Competition Metrics

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Computes performance metrics by converting PyTorch tensors to DataFrames and invoking a scoring function. It assumes the existence of a global score function and requires inputs to be on the GPU.

```python
def comp_metric(y_pred,y_test):
    y_pred = pd.DataFrame(y_pred.cpu().numpy())
    y_pred['id'] = np.arange(len(y_pred))

    y_test = pd.DataFrame(y_test.cpu().numpy())
    y_test['id'] = np.arange(len(y_test))

    return score(solution=y_test, submission=y_pred, row_id_column_name='id')

def get_score(y_preds, y_trues):
    score = comp_metric(y_preds, y_trues)
    return score
```

--------------------------------

### Initialize Training Environment and Dependencies

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/16- SenNet + HOA - Hacking the Human Vasculature in 3D/Training Notebook.ipynb

Sets up the local cache directory for pretrained weights, copies necessary model files, and installs required deep learning libraries including segmentation-models-pytorch and albumentations.

```bash
!mkdir -p /root/.cache/torch/hub/checkpoints/
!cp /kaggle/input/se-net-pretrained-imagenet-weights/* /root/.cache/torch/hub/checkpoints/
!pip install -q albumentations
!python -m pip install --no-index --find-links=/kaggle/input/pip-download-for-segmentation-models-pytorch segmentation-models-pytorch
```

```python
import torch as tc
import torch.nn as nn
import numpy as np
from tqdm import tqdm
import os,sys,cv2
from torch.cuda.amp import autocast
import matplotlib.pyplot as plt
import albumentations as A
import segmentation_models_pytorch as smp
from albumentations.pytorch import ToTensorV2
from torch.utils.data import Dataset, DataLoader
from torch.nn.parallel import DataParallel
from glob import glob
```

--------------------------------

### Configure Kaggle Data Paths

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/15- Novozymes Enzyme Stability Prediction/Training Notebook - ThermoNet.ipynb

Sets up the file system paths for input data, including PDB structures, feature files, and public submission CSVs required for the enzyme stability prediction task.

```python
WILDTYPE_PDB = '../input/novozymes-enzyme-stability-prediction/wildtype_structure_prediction_af2.pdb'
PDB_PATH = '../input/thermonet-wildtype-relaxed'
TRAIN_FEATURES_PATH = '../input/thermonet-features/Q3214.npy'
TEST_CSV = '../input/novozymes-enzyme-stability-prediction/test.csv'
TEST_FEATURES_PATH = '../input/thermonet-features/nesp_features.npy'

os.makedirs(MODELS_PATH, exist_ok=True)
```

--------------------------------

### Execute Multi-Checkpoint Ensemble Inference

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Iterates through a directory of model checkpoints, loads each into a custom model architecture, and performs inference. Results are collected and aggregated using mean averaging to produce final predictions.

```python
checkpoint_dir = f"/kaggle/input/hms-ckpts-2/wavenet/wavenet_PLs"
checkpoint_paths = glob(f"{checkpoint_dir}/*.ckpt")

for checkpoint_path in tqdm(checkpoint_paths):
    model = CustomModel_4.load_from_checkpoint(checkpoint_path, map_location=device,cfg=CFG, criterion=None, train_folds=None, valid_folds=None, fold=None)
    prediction = inference_fn(valid_loader, model, device)
    prediction = np.concatenate(prediction)
    predictions.append(prediction)
    del model, prediction; gc.collect()
    torch.cuda.empty_cache()

predictions_77 = np.mean(predictions, axis=0)
```

--------------------------------

### Optimizer and Scheduler Configuration with PyTorch Lightning (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Implements the configure_optimizers method for the CustomModel. This method sets up the optimizer (AdamW or Adam8bit) and learning rate scheduler based on configuration parameters. It handles layer-wise learning rate decay and optional 8-bit optimizers. Dependencies include PyTorch, PyTorch Lightning, and bitsandbytes.

```python
def configure_optimizers(self):
    optimizer_parameters = get_optimizer_params([self.backbone1,self.backbone2],
                                                    encoder_lr=CFG.encoder_lr,
                                                    decoder_lr=CFG.decoder_lr,
                                                    layerwise_learning_rate_decay=CFG.layerwise_learning_rate_decay,
                                                    weight_decay=CFG.weight_decay)
    
    if self.cfg.use_8bit_optimizer:
        import bitsandbytes as bnb
        optimizer = bnb.optim.Adam8bit(optimizer_parameters,
              lr=CFG.encoder_lr,
              eps=CFG.eps,
              betas=CFG.betas)
    else:
        optimizer = AdamW(optimizer_parameters,
                          lr=CFG.encoder_lr,
                          eps=CFG.eps,
                          betas=CFG.betas)
    
    scheduler = get_scheduler(CFG, optimizer, len(self.train_folds))
    lr_scheduler_dict = {"scheduler": scheduler, "interval": CFG.sch_interval}
    if self.cfg.use_scheduler:
        return {'optimizer': optimizer, 'lr_scheduler': lr_scheduler_dict}
    else:
        return {'optimizer': optimizer}
```

--------------------------------

### Set Up Output Directory (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/19- Arabic Poem Classification/Training Notebook.ipynb

Configures the output directory for the project. It checks if the specified directory exists and creates it if it doesn't. Defines paths for training and testing data files.

```python
# ====================================================
# Directory settings
# ====================================================
import os

OUTPUT_DIR = './'
if not os.path.exists(OUTPUT_DIR):
    os.makedirs(OUTPUT_DIR)
    
TRAIN_PATH = 'arabic-poem-classification/poems.csv'
TEST_PATH = 'arabic-poem-classification/test.csv'

```

--------------------------------

### Print Separator Line (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

A simple utility function to print a horizontal line of 100 hyphens to visually separate sections of output in the console. This aids in organizing and reading logs or intermediate results.

```python
def sep():
    print('-'*100)
```

--------------------------------

### Average Predictions (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

This snippet calculates the mean of prediction arrays. It is typically used after collecting predictions from multiple sources, such as different model checkpoints or data augmentations, to produce a single, more robust prediction.

```python
predictions_38 = np.mean(predictions_38,axis=0)
predictions_39 = np.mean(predictions_39,axis=0)
predictions_40 = np.mean(predictions_40,axis=0)
```

--------------------------------

### Install PyTorch TabNet from Local Wheel File

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/01- UM - Game-Playing Strength of MCTS Variants/Inference Code.ipynb

Installs the PyTorch TabNet library from a local wheel file located in the Kaggle input directory. This is useful when the package is not available in the Kaggle environment's default channels. It ensures that the specified version of PyTorch TabNet is installed correctly.

```python
!pip install --no-index --find-links /kaggle/input/pytorchtabnet/pytorch_tabnet-4.1.0-py3-none-any.whl pytorch-tabnet
```

--------------------------------

### Initialize Computational Resources (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/08- BirdCLEF 2023/Inference Notebook.ipynb

Initializes the computational resources (GPU/TPU/TPU-VM) by calling the `get_device` function and setting the number of replicas based on the detected strategy. This is a common setup step for deep learning tasks.

```python
# Initialize GPU/TPU/TPU-VM
strategy, CFG.device, tpu = get_device()
CFG.replicas = strategy.num_replicas_in_sync
```

--------------------------------

### Calculate Competition Metric

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Defines a function to calculate the competition's scoring metric. It takes predicted and true values, formats them into pandas DataFrames with an 'id' column, and uses a provided 'score' function.

```python
def comp_metric(y_pred,y_test):
    y_pred = pd.DataFrame(y_pred)
    y_pred['id'] = np.arange(len(y_pred))

    y_test = pd.DataFrame(y_test)
    y_test['id'] = np.arange(len(y_test))

    return score(solution=y_test, submission=y_pred, row_id_column_name='id')
```

--------------------------------

### Prepare Input and Dataset for Transformers

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/12- Feedback Prize - English Language Learning/Training Notebook.ipynb

Functions and classes to tokenize input text and manage data loading. It handles tokenization via HuggingFace transformers and converts data into PyTorch tensors for model consumption.

```python
def prepare_input(cfg, text):
    inputs = cfg.tokenizer.encode_plus(
        text, 
        return_tensors=None, 
        add_special_tokens=True, 
        max_length=CFG.max_len,
        pad_to_max_length=True,
        truncation=True
    )
    for k, v in inputs.items():
        inputs[k] = torch.tensor(v, dtype=torch.long)
    return inputs

class TrainDataset(Dataset):
    def __init__(self, cfg, df):
        self.cfg = cfg
        self.texts = df['full_text'].values
        self.labels = df[cfg.target_cols].values

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, item):
        inputs = prepare_input(self.cfg, self.texts[item])
        label = torch.tensor(self.labels[item], dtype=torch.float)
        return inputs, label
```

--------------------------------

### Perform Ensemble Inference with TTA

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

Iterates through model checkpoints to generate predictions using various augmentation strategies. It aggregates results across multiple inference passes and clears GPU memory to prevent OOM errors.

```python
checkpoint_paths = glob(f"{checkpoint_dir}/*.ckpt")
for checkpoint_path in tqdm(checkpoint_paths):
    model = CustomModel_4.load_from_checkpoint(checkpoint_path, map_location=device, cfg=CFG, criterion=None, train_folds=None, valid_folds=None, fold=None)
    prediction_1 = inference_fn(valid_loader_original, model, device)
    prediction_2 = inference_fn(valid_loader_vflip, model, device)
    prediction_3 = inference_fn(valid_loader_time_shift, model, device)
    prediction_4 = inference_fn(valid_loader_xy_mask, model, device)
    
    predictions_49.append(np.concatenate(prediction_1))
    predictions_50.append(np.concatenate(prediction_2))
    predictions_51.append(np.concatenate(prediction_3))
    predictions_52.append(np.concatenate(prediction_4))
    del model; gc.collect(); torch.cuda.empty_cache()
```

--------------------------------

### Import Libraries and Setup Environment (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/12- Feedback Prize - English Language Learning/Inference Notebook.ipynb

Imports essential Python libraries for data science, machine learning, and deep learning, including PyTorch and Hugging Face Transformers. It also sets up the device for computation (GPU if available, otherwise CPU) and configures environment variables for tokenizers.

```python
import os
import gc
import re
import ast
import sys
import copy
import json
import time
import math
import string
import pickle
import random
import joblib
import itertools
import warnings
warnings.filterwarnings("ignore")

import scipy as sp
import numpy as np
import pandas as pd
from tqdm.auto import tqdm
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold

import torch
import torch.nn as nn
from torch.nn import Parameter
import torch.nn.functional as F
from torch.optim import Adam, SGD, AdamW
from torch.utils.data import DataLoader, Dataset

import tokenizers
import transformers
print(f"tokenizers.__version__: {tokenizers.__version__}")
print(f"transformers.__version__: {transformers.__version__}")
from transformers import AutoTokenizer, AutoModel, AutoConfig
from transformers import get_linear_schedule_with_warmup, get_cosine_schedule_with_warmup
from transformers import DataCollatorWithPadding
%env TOKENIZERS_PARALLELISM=false

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

--------------------------------

### Combine Predictions with Optimal Weights (Python)

Source: https://github.com/mohammad2012191/kaggle_competitions_solutions/blob/main/18- HMS - Harmful Brain Activity Classification/Inference Notebook - Ensemble.ipynb

This snippet demonstrates how to combine multiple prediction arrays (e.g., from different models) by multiplying each prediction array by a corresponding weight from an 'optimal_weights' array. This is typically done column-wise for each prediction set.

```python
(predictions_17[:,i] * optimal_weights[0]) +
(predictions_18[:,i] * optimal_weights[0]) +
(predictions_19[:,i] * optimal_weights[0]) +
(predictions_20[:,i] * optimal_weights[0]) +

(predictions_21[:,i] * optimal_weights[1]) +
(predictions_22[:,i] * optimal_weights[1]) +
(predictions_23[:,i] * optimal_weights[1]) +
(predictions_24[:,i] * optimal_weights[1]) +

(predictions_29[:,i] * optimal_weights[2]) +
(predictions_30[:,i] * optimal_weights[2]) +
(predictions_31[:,i] * optimal_weights[2]) +
(predictions_32[:,i] * optimal_weights[2]) +

(predictions_33[:,i] * optimal_weights[3]) +
(predictions_34[:,i] * optimal_weights[3]) +
(predictions_35[:,i] * optimal_weights[3]) +
(predictions_36[:,i] * optimal_weights[3]) +

(predictions_37[:,i] * optimal_weights[4]) +
(predictions_38[:,i] * optimal_weights[4]) +
(predictions_39[:,i] * optimal_weights[4]) +
(predictions_40[:,i] * optimal_weights[4]) +

(predictions_45[:,i] * optimal_weights[5]) +
(predictions_46[:,i] * optimal_weights[5]) +
(predictions_47[:,i] * optimal_weights[5]) +
(predictions_48[:,i] * optimal_weights[5]) +

(predictions_53[:,i] * optimal_weights[6]) +
(predictions_54[:,i] * optimal_weights[6]) +
(predictions_55[:,i] * optimal_weights[6]) +
(predictions_56[:,i] * optimal_weights[6]) +

(predictions_57[:,i] * optimal_weights[7]) +
(predictions_58[:,i] * optimal_weights[7]) +
(predictions_59[:,i] * optimal_weights[7]) +
(predictions_60[:,i] * optimal_weights[7]) +

(predictions_61[:,i] * optimal_weights[8]) +
(predictions_62[:,i] * optimal_weights[8]) +
(predictions_63[:,i] * optimal_weights[8]) +
(predictions_64[:,i] * optimal_weights[8])
```