### Build and Install WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/BuildWeightWatcherOnColab.ipynb

Navigates into the cloned WeightWatcher directory and builds/installs the project using its setup script. This prepares the project for testing and use.

```bash
%cd WeightWatcher
!python setup.py build install
```

--------------------------------

### Setup and Imports for MLP Training

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/Epoch-Wise-DoubleDescent.ipynb

Installs necessary libraries and imports modules for PyTorch, data handling, and WeightWatcher. Sets up the computation device (MPS, CUDA, or CPU) and output directory.

```python
#!pip -q install torch torchvision pandas matplotlib weightwatcher

import os, math, time, random
from dataclasses import dataclass
import numpy as np
import pandas as pd
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import matplotlib
import weightwatcher as ww

%matplotlib inline
import matplotlib.pyplot as plt


# =========================
# Repro + device
# =========================
def set_seed(s):
    random.seed(s); np.random.seed(s)
    torch.manual_seed(s); torch.cuda.manual_seed_all(s)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False



if torch.backends.mps.is_available():
    device = torch.device("mps")
elif torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

import os
OUTDIR = "./runs/mlp3_mnist_dd"
os.makedirs(OUTDIR, exist_ok=True)
```

--------------------------------

### Install WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-ONNX.ipynb

Use pip to install the library.

```python
!pip install weightwatcher
```

--------------------------------

### Install WeightWatcher Library

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Albert.ipynb

Installs the WeightWatcher package from the test PyPI repository.

```bash
!python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple weightwatcher
```

--------------------------------

### Installing Dependencies

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW_Sentence_Transformers.ipynb

Commands to install the necessary packages for model analysis.

```bash
!pip install transformers weightwatcher gwpy matplotlib==3.1.3
!pip install -U sentence-transformers
```

--------------------------------

### WeightWatcher Installation Output

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Albert.ipynb

Console output showing the successful installation and dependency verification of WeightWatcher.

```text
Output:
Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple
Requirement already satisfied: weightwatcher in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages/weightwatcher-0.6.3.3-py3.9.egg (0.6.3.3)
Requirement already satisfied: numpy in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (1.19.5)
Requirement already satisfied: pandas in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (1.4.0)
Requirement already satisfied: matplotlib in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (3.6.0)
Requirement already satisfied: matplotlib-inline in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (0.1.6)
Requirement already satisfied: powerlaw in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (1.5)
Requirement already satisfied: scikit-learn in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (1.1.2)
Requirement already satisfied: deprecated in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (1.2.13)
Requirement already satisfied: telly in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (0.9.3)
Requirement already satisfied: tqdm in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from weightwatcher) (4.64.1)
Requirement already satisfied: wrapt<2,>=1.10 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from deprecated->weightwatcher) (1.12.1)
Requirement already satisfied: cycler>=0.10 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (0.11.0)
Requirement already satisfied: pillow>=6.2.0 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (9.2.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (1.4.4)
Requirement already satisfied: pyparsing>=2.2.1 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (2.8.2)
Requirement already satisfied: contourpy>=1.0.1 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (1.0.5)
Requirement already satisfied: packaging>=20.0 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (21.3)
Requirement already satisfied: fonttools>=4.22.0 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib->weightwatcher) (4.37.4)
Requirement already satisfied: traitlets in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from matplotlib-inline->weightwatcher) (5.4.0)
Requirement already satisfied: pytz>=2020.1 in /Users/charleshmartin/anaconda3/envs/ww0.6.5/lib/python3.9/site-packages (from pandas->weightwatcher) (2022.4)
```

--------------------------------

### Install Dependencies

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-GPT.ipynb

Install transformers and weightwatcher libraries if running in a Google Colab environment.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install transformers weightwatcher
```

--------------------------------

### Install and Import WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-VGG11-Debug.ipynb

Installs the WeightWatcher library if running in Google Colab and imports the necessary components.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install weightwatcher
```

```python
import logging

import weightwatcher as ww
import torchvision.models as models

logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.DEBUG)

ww.__version__
```

--------------------------------

### Import WeightWatcher and Configure Logging

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-LayerIterator.ipynb

Initializes the WeightWatcher logger to the WARNING level and verifies the installed version.

```python
import logging

import weightwatcher as ww
import torchvision.models as models

logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.WARNING)

ww.__version__
```

--------------------------------

### Install WeightWatcher using pip

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/README.md

Install the WeightWatcher package using pip. This is the standard method for installing Python packages.

```sh
pip install weightwatcher
```

--------------------------------

### Variable Swap Example

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSmoothing-VGG16-Keras.ipynb

Demonstrates a simple tuple unpacking swap.

```python
N, M = 4,5
N, M = M, N
N, M
```

--------------------------------

### Setup Training Components

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/MLP3-MNIST-AdamW.ipynb

Initializes the model, optimizer with differential learning rates for stem and head, learning rate scheduler, and gradient scaler for mixed precision.

```python
# Setup training
model = MLP3(hp.hidden1, hp.hidden2, hp.dropout).to(device)
pgs = [
    {"params": list(model.stem.parameters()), "lr": hp.base_lr*hp.stem_lr_mul, "initial_lr": hp.base_lr*hp.stem_lr_mul, "weight_decay": hp.weight_decay},
    {"params": list(model.head.parameters()), "lr": hp.base_lr*hp.head_lr_mul, "initial_lr": hp.base_lr*hp.head_lr_mul, "weight_decay": hp.weight_decay}
]
opt = torch.optim.AdamW(pgs, betas=(0.9, 0.999), eps=1e-8)
sched = WarmupCosine(opt, hp.epochs, hp.warmup_epochs, hp.min_lr_ratio)
scaler = torch.cuda.amp.GradScaler(enabled=hp.amp and device.type=="cuda")
```

--------------------------------

### Prepare for PyPI Deployment

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/BuildWeightWatcherOnColab.ipynb

Installs necessary packaging tools (setuptools, wheel, twine), builds source and wheel distributions, and checks the distribution files for errors before uploading to PyPI.

```bash
!python -m pip install --upgrade setuptools wheel twine
!python setup.py sdist bdist_wheel
!twine check dist/*
!ls dist
```

--------------------------------

### Install and Import WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-VGG11-SoftRank.ipynb

Installs the WeightWatcher library if running in Google Colab and imports it along with torchvision models. Sets the logging level for WeightWatcher to INFO.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install weightwatcher
```

```python
import logging

import weightwatcher as ww
import torchvision.models as models

logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.INFO)

ww.__version__
```

--------------------------------

### Install WeightWatcher from TestPyPI

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/README.md

Install a specific version of WeightWatcher from TestPyPI. Use this if the standard pip installation fails or for testing pre-release versions.

```sh
python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple weightwatcher
```

--------------------------------

### Training Log Output Example

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/Epoch-Wise-DoubleDescent.ipynb

Example output format for training logs, showing epoch number, loss, accuracy metrics, and learning rates.

```text
SMALL N PL FIT
Epoch 01/100  train_loss 0.6711  train_acc 87.77%  train_clean_acc 96.41%  test_acc 96.73%  LRs [0.0008, 0.0016]  (20.0s)
SMALL N PL FIT
Epoch 02/100  train_loss 0.4724  train_acc 94.85%  train_clean_acc 97.91%  test_acc 97.83%  LRs [0.0016, 0.0032]  (14.0s)
SMALL N PL FIT
Epoch 03/100  train_loss 0.4685  train_acc 94.90%  train_clean_acc 97.74%  test_acc 97.72%  LRs [0.0024, 0.0048]  (11.7s)
SMALL N PL FIT
Epoch 04/100  train_loss 0.4665  train_acc 95.11%  train_clean_acc 97.94%  test_acc 97.84%  LRs [0.002399, 0.004799]  (12.0s)
SMALL N PL FIT
Epoch 05/100  train_loss 0.4421  train_acc 95.77%  train_clean_acc 98.39%  test_acc 98.20%  LRs [0.002398, 0.004795]  (12.1s)
SMALL N PL FIT
Epoch 06/100  train_loss 0.4275  train_acc 96.23%  train_clean_acc 98.62%  test_acc 98.43%  LRs [0.002394, 0.004789]  (12.8s)
SMALL N PL FIT
Epoch 07/100  train_loss 0.4155  train_acc 96.69%  train_clean_acc 98.75%  test_acc 98.71%  LRs [0.00239, 0.00478]  (13.1s)
SMALL N PL FIT
Epoch 08/100  train_loss 0.4106  train_acc 96.81%  train_clean_acc 98.74%  test_acc 98.53%  LRs [0.002384, 0.004769]  (11.7s)
SMALL N PL FIT
Epoch 09/100  train_loss 0.4036  train_acc 97.08%  train_clean_acc 98.79%  test_acc 98.73%  LRs [0.002378, 0.004755]  (12.2s)
SMALL N PL FIT
Epoch 10/100  train_loss 0.4021  train_acc 97.09%  train_clean_acc 98.84%  test_acc 98.75%  LRs [0.00237, 0.004739]  (11.6s)
SMALL N PL FIT
Epoch 11/100  train_loss 0.4020  train_acc 97.16%  train_clean_acc 98.77%  test_acc 98.55%  LRs [0.00236, 0.004721]  (11.4s)
SMALL N PL FIT
Epoch 12/100  train_loss 0.4083  train_acc 97.01%  train_clean_acc 98.70%  test_acc 98.51%  LRs [0.00235, 0.0047]  (10.9s)
SMALL N PL FIT
Epoch 13/100  train_loss 0.4097  train_acc 96.98%  train_clean_acc 98.84%  test_acc 98.50%  LRs [0.002338, 0.004676]  (10.7s)
SMALL N PL FIT
Epoch 14/100  train_loss 0.4130  train_acc 97.05%  train_clean_acc 98.71%  test_acc 98.61%  LRs [0.002325, 0.004651]  (10.8s)
SMALL N PL FIT
Epoch 15/100  train_loss 0.4284  train_acc 96.75%  train_clean_acc 98.90%  test_acc 98.60%  LRs [0.002311, 0.004623]  (10.9s)
SMALL N PL FIT
Epoch 16/100  train_loss 0.4342  train_acc 96.69%  train_clean_acc 98.90%  test_acc 98.70%  LRs [0.002296, 0.004592]  (10.8s)
SMALL N PL FIT
Epoch 17/100  train_loss 0.4628  train_acc 96.07%  train_clean_acc 98.55%  test_acc 98.10%  LRs [0.00228, 0.00456]  (11.4s)
SMALL N PL FIT
Epoch 18/100  train_loss 0.4776  train_acc 95.50%  train_clean_acc 98.50%  test_acc 98.39%  LRs [0.002263, 0.004525]  (11.2s)
SMALL N PL FIT
Epoch 19/100  train_loss 0.5238  train_acc 94.55%  train_clean_acc 98.77%  test_acc 98.77%  LRs [0.002244, 0.004488]  (10.9s)
SMALL N PL FIT
Epoch 20/100  train_loss 0.5479  train_acc 93.65%  train_clean_acc 98.76%  test_acc 98.62%  LRs [0.002224, 0.004449]  (10.9s)
SMALL N PL FIT
Epoch 21/100  train_loss 0.6267  train_acc 92.30%  train_clean_acc 97.23%  test_acc 97.20%  LRs [0.002204, 0.004408]  (10.9s)
SMALL N PL FIT
Epoch 22/100  train_loss 0.6255  train_acc 92.21%  train_clean_acc 97.56%  test_acc 97.35%  LRs [0.002182, 0.004364]  (11.3s)
SMALL N PL FIT
Epoch 23/100  train_loss 0.7148  train_acc 91.29%  train_clean_acc 98.54%  test_acc 98.50%  LRs [0.002159, 0.004319]  (11.5s)
SMALL N PL FIT
Epoch 24/100  train_loss 0.7687  train_acc 90.70%  train_clean_acc 98.32%  test_acc 98.43%  LRs [0.002136, 0.004271]  (11.4s)
SMALL N PL FIT
Epoch 25/100  train_loss 0.8705  train_acc 90.03%  train_clean_acc 98.10%  test_acc 97.92%  LRs [0.002111, 0.004222]  (11.6s)
SMALL N PL FIT
Epoch 26/100  train_loss 0.8835  train_acc 90.25%  train_clean_acc 98.81%  test_acc 98.81%  LRs [0.002085, 0.004171]  (23.2s)
SMALL N PL FIT
Epoch 27/100  train_loss 0.9212  train_acc 90.19%  train_clean_acc 98.72%  test_acc 98.76%  LRs [0.002059, 0.004118]  (27.2s)
SMALL N PL FIT
Epoch 28/100  train_loss 0.9592  train_acc 90.46%  train_clean_acc 98.71%  test_acc 98.70%  LRs [0.002031, 0.004063]  (11.8s)
SMALL N PL FIT
Epoch 29/100  train_loss 1.0599  train_acc 90.64%  train_clean_acc 98.67%  test_acc 98.48%  LRs [0.002003, 0.004006]  (11.0s)
SMALL N PL FIT
Epoch 30/100  train_loss 1.1602  train_acc 90.59%  train_clean_acc 98.23%  test_acc 97.90%  LRs [0.001974, 0.003948]  (11.8s)
```

--------------------------------

### Installing packages in Colab

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-LegalNER.ipynb

Conditional installation of weightwatcher and transformers for Google Colab environments.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install weightwatcher transformers
```

--------------------------------

### Setup and Hyperparameters for Muon-style MLP

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/MLP3-MNIST-Muon.ipynb

Initializes the environment, sets the random seed, configures the device, and defines the hyperparameters for the model and training process.

```python
# Colab-ready: Muon-style MLP on MNIST (no EMA)
# (Optional) fresh runtime:
# !pip -q install torch torchvision pandas matplotlib

import os, math, time, random
from dataclasses import dataclass
import numpy as np
import pandas as pd
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import matplotlib

%matplotlib inline
import matplotlib.pyplot as plt

# =========================
# Hyperparameters
# =========================
@dataclass
class HParams:
    # Model
    hidden1: int = 1792
    hidden2: int = 896
    dropout: float = 0.15

    # Training
    epochs: int = 40
    batch_size: int = 256
    base_lr: float = 0.03              # for SGD+Nesterov
    head_lr_mul: float = 1.6
    stem_lr_mul: float = 0.8
    weight_decay: float = 8e-5
    warmup_epochs: int = 3
    min_lr_ratio: float = 0.01
    label_smoothing: float = 0.05
    clip_grad_norm: float = 0.0

    # Muon-style spectral control
    s_max: float = 1.0                # clamp per Linear layer ||W||_2 ≤ s_max
    power_iters: int = 1                # power-iteration steps (1 = fast & fine)

    # Data augmentation
    degrees: float = 10.0
    translate: tuple = (0.10, 0.10)
    scale: tuple = (0.95, 1.05)
    random_erasing: float = 0.05

    # System
    num_workers: int = 2
    seed: int = 42
    amp: bool = True                    # only enabled if CUDA is available
    out_dir: str = "./runs/mlp3_mnist_muon"

hp = HParams()

# =========================
# Repro + device
# =========================
def set_seed(s):
    random.seed(s); np.random.seed(s)
    torch.manual_seed(s); torch.cuda.manual_seed_all(s)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

set_seed(hp.seed)

if torch.backends.mps.is_available():
    device = torch.device("mps")
elif torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")
    
amp_enabled = hp.amp and (device.type == "cuda")  # <— robust AMP gating
os.makedirs(hp.out_dir, exist_ok=True)
print("Device:", device, "| AMP enabled:", amp_enabled)
```

--------------------------------

### Get WeightWatcher Version

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSharpness-VGG11.ipynb

Retrieves and displays the installed version of the WeightWatcher library.

```python
import weightwatcher as ww
logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.INFO)

ww.__version__
```

--------------------------------

### Import and Version Check

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-ONNX.ipynb

Import the library and verify the installed version.

```python
import weightwatcher as ww
```

```python
ww.__version__
```

--------------------------------

### Install Transformers Library

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/BuildWeightWatcherOnColab.ipynb

Installs the 'transformers' library, which is a dependency for the WeightWatcher project. This is typically done using pip.

```bash
!pip install transformers
```

--------------------------------

### Install and Import WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-GloveVectors.ipynb

Installs the WeightWatcher library if running in Google Colab and imports it along with torchvision models. Sets custom logging level for WeightWatcher.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install weightwatcher
```

```python
import weightwatcher as ww
import torchvision.models as models

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.INFO)

ww.__version__#,  torchvision.__version__
```

--------------------------------

### List Slicing Example

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSmoothing-VGG16-Keras.ipynb

Demonstrates list slicing syntax.

```python
a = [1,2,3,4,5]
```

```python
a[:-2]
```

--------------------------------

### Create Dummy Image Files

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-VGG.ipynb

Creates empty image files. This is likely a placeholder or setup step before actual image loading.

```python
!touch VGG16.1.png VGG16.2.png CV-models.png
```

--------------------------------

### Import Libraries and Setup

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/Marchenko_Pastur_Plots.ipynb

Imports necessary libraries for numerical computation, deep learning, plotting, and utility functions. Sets up Matplotlib for inline plotting and font sizes.

```python
import sys
import pickle, time
from copy import deepcopy
from shutil import copy
from tqdm import tqdm_notebook as tqdm

import numpy as np
import tensorflow as tf
import keras
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.layers.convolutional import Conv2D, MaxPooling2D
from keras.layers.normalization import BatchNormalization
from keras.callbacks import TensorBoard, EarlyStopping
from keras.optimizers import SGD
from keras.initializers import Constant
import keras.backend as K

import import_ipynb
import RMT_Util

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

import sklearn
from sklearn.decomposition import TruncatedSVD
from sklearn.random_projection import sparse_random_matrix

print(sys.version)
print("numpy version {}".format(np.__version__))
print("tensforflow version {}".format(tf.__version__))
print("keras version {}".format(keras.__version__))
print("sklearn version {}".format(sklearn.__version__))
```

```python
plt.rcParams['figure.figsize'] = [6,5]
matplotlib.rcParams.update({'font.size': 16})
```

--------------------------------

### Install WeightWatcher in Colab

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Calibrate_Alpha.ipynb

Installs the weightwatcher package if running in a Google Colab environment.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install weightwatcher
```

--------------------------------

### Import Libraries for VGG11 Example

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSharpness-VGG11.ipynb

Imports necessary libraries including numpy, pandas, tensorflow, torchvision, tqdm, and matplotlib for the VGG11 example.

```python
import numpy as np
import pandas as pd

import tensorflow
import torchvision.models as models
from tqdm import tqdm

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

--------------------------------

### Install Dependencies

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Albert-Paper-Example.ipynb

Installs necessary libraries including transformers, weightwatcher, and a specific matplotlib version. This is required for running the WeightWatcher analysis in a Google Colab environment.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install transformers weightwatcher gwpy matplotlib==3.1.3

```

--------------------------------

### Run Unit Tests

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/BuildWeightWatcherOnColab.ipynb

Executes the unit tests for the WeightWatcher project using the setup script. This verifies the project's functionality.

```bash
!python setup.py test
```

--------------------------------

### Run Weight Watcher Analysis

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/TestFrameworks.ipynb

Execute the weight watcher analysis. Ensure the 'weightwatcher' library is installed and imported.

```python
watcher.analyze()
```

--------------------------------

### Verify Versions

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-GPT-Debug.ipynb

Check the installed versions of torch and weightwatcher.

```python
import torch
import weightwatcher as ww
torch.__version__, ww.__version__
```

```text
Result:
('1.6.0', '0.6.4')
```

--------------------------------

### WeightWatcher Debug and Info Logs

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-VGG11-Debug.ipynb

Example output logs showing layer processing, SVD parameter configuration, and eigenvalue extraction.

```text
DEBUG:weightwatcher:layer_supported  N 25088 max evals 10000
DEBUG:weightwatcher:Layer 9 classifier.0 is skipped
```

```text
DEBUG:weightwatcher:keeping layer 10 classifier.3 by id
DEBUG:weightwatcher:layer_supported  N 4096 max evals 10000
INFO:weightwatcher:Getting ESD for layer 10 ; ww_layer id = 10
DEBUG:weightwatcher:apply ESD  on Layer 10 classifier.3 
DEBUG:weightwatcher:running SVD on Layer 10 classifier.3 
DEBUG:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': True, 'savedir': 'ww-img', 'savefig': True, 'rescale': True, 'plot': False, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': False, 'vectors': True, 'smooth': None, 'stacked': False, 'svd_method': 'accurate', 'fix_fingers': None, 'fit': 'power_law', 'sparsify': True, 'detX': True, 'mp_fit': False, 'min_evals': 50, 'max_evals': 10000, 'max_N': 10, 'tolerance': 1e-06, 'layer_ids_start': 0, 'add_biases': False} 
DEBUG:weightwatcher:Running accurate SVD:  W.shape=(4096, 4096)  n_comp = 4096
```

```text
FILTER 10
```

```text
DEBUG:weightwatcher:Found 4096 eiganvalues for 10 classifier.3
```

```text
4096
```

```text
INFO:weightwatcher: Smoothing method svd
INFO:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': True, 'savedir': 'ww-img', 'savefig': True, 'rescale': True, 'plot': False, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': False, 'vectors': True, 'smooth': 'svd', 'stacked': False, 'svd_method': 'accurate', 'fix_fingers': None, 'fit': 'PL', 'sparsify': True, 'detX': True, 'mp_fit': False, 'min_evals': 50, 'max_evals': 10000, 'max_N': 10, 'tolerance': 1e-06, 'layer_ids_start': 0, 'add_biases': False, 'layers': [10]}
INFO:weightwatcher:Saving all images to ww-img
INFO:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': True, 'savedir': 'ww-img', 'savefig': True, 'rescale': True, 'plot': False, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': False, 'vectors': True, 'smooth': 'svd', 'stacked': False, 'svd_method': 'accurate', 'fix_fingers': None, 'fit': 'power_law', 'sparsify': True, 'detX': True, 'mp_fit': False, 'min_evals': 50, 'max_evals': 10000, 'max_N': 10, 'tolerance': 1e-06, 'layer_ids_start': 0, 'add_biases': False, 'layers': [10]}
INFO:weightwatcher:Saving all images to ww-img
INFO:weightwatcher:torch version 1.12.1
INFO:weightwatcher:framework from model = 16
INFO:weightwatcher:Filtering layer by id 10
DEBUG:weightwatcher:conv2D_Wmats
DEBUG:weightwatcher:channels Last tensor shape: 64x3 (NxM), 3x3 (i,j)
DEBUG:weightwatcher:get_conv2D_Wmats N=64 M=3 rf= 9 channels= 4
DEBUG:weightwatcher:skipping layer 1 features.0 by id
DEBUG:weightwatcher:layer_supported  N 64 max evals 10000
DEBUG:weightwatcher:Layer 1 features.0 is skipped
DEBUG:weightwatcher:conv2D_Wmats
DEBUG:weightwatcher:channels Last tensor shape: 128x64 (NxM), 3x3 (i,j)
DEBUG:weightwatcher:get_conv2D_Wmats N=128 M=64 rf= 9 channels= 4
DEBUG:weightwatcher:skipping layer 2 features.3 by id
DEBUG:weightwatcher:layer_supported  N 128 max evals 10000
DEBUG:weightwatcher:Layer 2 features.3 is skipped
DEBUG:weightwatcher:conv2D_Wmats
DEBUG:weightwatcher:channels Last tensor shape: 256x128 (NxM), 3x3 (i,j)
DEBUG:weightwatcher:get_conv2D_Wmats N=256 M=128 rf= 9 channels= 4
DEBUG:weightwatcher:skipping layer 3 features.6 by id
DEBUG:weightwatcher:layer_supported  N 256 max evals 10000
DEBUG:weightwatcher:Layer 3 features.6 is skipped
DEBUG:weightwatcher:conv2D_Wmats
DEBUG:weightwatcher:channels Last tensor shape: 256x256 (NxM), 3x3 (i,j)
DEBUG:weightwatcher:get_conv2D_Wmats N=256 M=256 rf= 9 channels= 4
DEBUG:weightwatcher:skipping layer 4 features.8 by id
DEBUG:weightwatcher:layer_supported  N 256 max evals 10000
DEBUG:weightwatcher:Layer 4 features.8 is skipped
DEBUG:weightwatcher:conv2D_Wmats
DEBUG:weightwatcher:channels Last tensor shape: 512x256 (NxM), 3x3 (i,j)
DEBUG:weightwatcher:get_conv2D_Wmats N=512 M=256 rf= 9 channels= 4
DEBUG:weightwatcher:skipping layer 5 features.11 by id
DEBUG:weightwatcher:layer_supported  N 512 max evals 10000
DEBUG:weightwatcher:Layer 5 features.11 is skipped
DEBUG:weightwatcher:conv2D_Wmats
DEBUG:weightwatcher:channels Last tensor shape: 512x512 (NxM), 3x3 (i,j)
DEBUG:weightwatcher:get_conv2D_Wmats N=512 M=512 rf= 9 channels= 4
DEBUG:weightwatcher:skipping layer 6 features.13 by id
DEBUG:weightwatcher:layer_supported  N 512 max evals 10000
DEBUG:weightwatcher:Layer 6 features.13 is skipped
DEBUG:weightwatcher:conv2D_Wmats
DEBUG:weightwatcher:channels Last tensor shape: 512x512 (NxM), 3x3 (i,j)
DEBUG:weightwatcher:get_conv2D_Wmats N=512 M=512 rf= 9 channels= 4
DEBUG:weightwatcher:skipping layer 7 features.16 by id
DEBUG:weightwatcher:layer_supported  N 512 max evals 10000
DEBUG:weightwatcher:Layer 7 features.16 is skipped
```

--------------------------------

### Verify Library Versions

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-GPT.ipynb

Check the installed versions of torch and weightwatcher.

```python
import torch
import weightwatcher as ww
torch.__version__, ww.__version__
```

--------------------------------

### Load VGG16 Model and Initialize WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSmoothing-VGG16.ipynb

Loads a pre-trained VGG16 model from PyTorch and initializes the WeightWatcher with the model. Ensure PyTorch and WeightWatcher are installed.

```python
vgg16_pt = models.vgg16(pretrained=True)
watcher = ww.WeightWatcher(model=vgg16_pt)
details = watcher.describe()
```

--------------------------------

### Initialize WeightWatcher and Describe Model

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Inverse.ipynb

Imports the WeightWatcher library and initializes a WeightWatcher object. The describe method is then called on the model to get initial layer information.

```python
import weightwatcher as ww
print(ww.__version__)

watcher = ww.WeightWatcher()

watcher.describe(model=model)
```

--------------------------------

### Load Models for Framework Detection

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/TestFrameworks.ipynb

Examples of loading models from Transformers or Keras to be used with WeightWatcher.

```python
from transformers import AutoModel
albert_model = AutoModel.from_pretrained('albert-base-v2')
```

```python
import tensorflow as tf
from tensorflow.keras.models import load_model
from tensorflow.keras.applications import vgg16

vgg16_model = vgg16.VGG16(weights='imagenet')
```

--------------------------------

### Initialize WeightWatcher and Analyze VGG11

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSharpness-VGG11.ipynb

Initializes WeightWatcher with a pre-trained VGG11 model, describes its layers, and performs an initial analysis. Use this to get a baseline understanding of the model's weight distributions.

```python
vgg11 = models.vgg11(pretrained=True)
watcher = ww.WeightWatcher(model=vgg11)
details = watcher.describe()
esd = watcher.get_ESD(model=vgg11, layer=8)
analyzed = watcher.analyze(plot=True, layers=[8], randomize=True)
```

--------------------------------

### Initialize VGG19 Model and WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-VGG-RandDistance.ipynb

Loads a pre-trained VGG19 model from torchvision.models and initializes the WeightWatcher with this model. This setup is the first step towards analyzing the model's weights.

```python
modelname = 'VGG19'
model = models.vgg19(pretrained=True)
watcher = ww.WeightWatcher(model=model)
```

--------------------------------

### Initialize Environment

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-LayerIterator.ipynb

Imports necessary libraries and configures the environment for data analysis and visualization.

```python
import numpy as np
import pandas as pd

from tqdm import tqdm

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

%load_ext watermark
%watermark
```

--------------------------------

### Initialize WeightWatcher and Models

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-DenseNet.ipynb

Sets up logging and imports the WeightWatcher and torchvision model modules.

```python
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

import weightwatcher as ww
import torchvision.models as models

ww.__version__
```

--------------------------------

### Import and Configure WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Custom-Plot-Title.ipynb

Initializes the WeightWatcher library and configures logging levels.

```python
import weightwatcher as ww
import torchvision
import torchvision.models as models


import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.INFO) 


ww.__version__,  torchvision.__version__
```

--------------------------------

### Install Dependencies for Google Colab

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Albert.ipynb

Installs necessary libraries if running within a Google Colab environment.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install transformers  gwpy matplotlib==3.1.3
```

--------------------------------

### Initializing WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-LegalNER.ipynb

Configure logging and verify library versions.

```python
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

import weightwatcher as ww
import transformers

ww.__version__, transformers.__version__
```

--------------------------------

### Install Dependencies in Google Colab

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-BERT-BlogExample.ipynb

Installs the required libraries (transformers, weightwatcher, gwpy, matplotlib) if running in a Google Colab environment.

```python
import sys
if 'google.colab' in sys.modules:
    !pip install transformers weightwatcher gwpy matplotlib==3.1.3
```

--------------------------------

### Install powerlaw package

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/LSA.ipynb

Installs the powerlaw package, which is used for fitting and analyzing power-law distributions. Ensure you have pip and a compatible Python environment.

```python
pip install powerlaw
```

--------------------------------

### Upload to TestPyPI (Commented Out)

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/BuildWeightWatcherOnColab.ipynb

This command is commented out but shows how to upload the built distributions to the TestPyPI repository. Replace 'testpypi' with 'pypi' for the official release.

```bash
#!twine upload --repository testpypi dist/*
```

--------------------------------

### Get ESD for a Layer

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-VGG11-Debug.ipynb

This Python snippet shows how to get the Empirical Spectral Distribution (ESD) for a specific layer using the Weight Watcher library.

```python
esd_aftr = watcher.get_ESD(layer=fc2_layer)
```

--------------------------------

### Initialize WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Calibrate_Alpha.ipynb

Configures logging for the WeightWatcher library.

```python
import weightwatcher as ww
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(ww.__name__) 
logger.setLevel(logging.INFO)
```

--------------------------------

### Describe Model with WeightWatcher

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WWPyStateDictFileIterator.ipynb

Initialize WeightWatcher with a model directory and generate a description of the weights.

```python
logger = logging.getLogger(ww.__name__) 
logger.setLevel(logging.DEBUG)

watcher = ww.WeightWatcher(model=weights_dir)
pysd_details = watcher.describe()
```

--------------------------------

### Initialize WeightWatcher and Describe Model

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSmoothing.ipynb

Load a pre-trained VGG16 model and use WeightWatcher to analyze its layer structure.

```python
from tensorflow.keras.applications.vgg16 import VGG16
vgg16 = VGG16()
watcher = ww.WeightWatcher(model=vgg16)
watcher.describe()
```

--------------------------------

### Initialize Model, Optimizer, and Scheduler

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/Epoch-Wise-DoubleDescent.ipynb

Sets up the MLP model, AdamW optimizer with differential learning rates for stem and head, and a WarmupCosine learning rate scheduler.

```python
model = MLP3(hp.hidden1, hp.hidden2, hp.dropout).to(device)
pgs = [
    {"params": list(model.stem.parameters()),
     "lr": hp.base_lr*hp.stem_lr_mul,
     "initial_lr": hp.base_lr*hp.stem_lr_mul,
     "weight_decay": hp.weight_decay},
    {"params": list(model.head.parameters()),
     "lr": hp.base_lr*hp.head_lr_mul,
     "initial_lr": hp.base_lr*hp.head_lr_mul,
     "weight_decay": hp.weight_decay}
]
opt = torch.optim.AdamW(pgs, betas=(0.9, 0.999), eps=1e-8)
sched = WarmupCosine(opt, hp.epochs, hp.warmup_epochs, hp.min_lr_ratio)
scaler = torch.cuda.amp.GradScaler(enabled=hp.amp and device.type=="cuda")
```

--------------------------------

### Initialize WeightWatcher and Describe Model

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-Custom-Plot-Title.ipynb

Creates a WeightWatcher instance for the model and generates a summary of its layers.

```python
%%capture

import warnings
warnings.filterwarnings('ignore')


watcher = ww.WeightWatcher(model=vgg16) 
watcher.describe()
```

--------------------------------

### Environment and Device Initialization

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/MLP3-MNIST-AdamW.ipynb

Sets up the random seed for reproducibility and detects the available compute device (MPS, CUDA, or CPU).

```python
# !pip -q install torch torchvision pandas matplotlib

import os, math, time, random
from dataclasses import dataclass
import numpy as np
import pandas as pd
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import matplotlib

%matplotlib inline
import matplotlib.pyplot as plt


# =========================
# Repro + device
# =========================
def set_seed(s):
    random.seed(s); np.random.seed(s)
    torch.manual_seed(s); torch.cuda.manual_seed_all(s)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

set_seed(hp.seed)

if torch.backends.mps.is_available():
    device = torch.device("mps")
elif torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

import os
OUTDIR = "./runs/mlp3_mnist"
os.makedirs(OUTDIR, exist_ok=True)
```

--------------------------------

### Get Layer IDs

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSmoothing-VGG16-Keras.ipynb

Retrieves the layer IDs as a numpy array.

```python
details.layer_id.to_numpy()
```

--------------------------------

### Define Layer Indices

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/ModelPlots.ipynb

Example index selection for first and last layers.

```python
#first_n_last_ids = [0, len(all_details)-1]
```

--------------------------------

### Initialize Model and Optimizer

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/MLP3-MNIST-Muon.ipynb

Sets up the MLP model with distinct parameter groups for stem and head, configures SGD, and initializes the learning rate scheduler and AMP scaler.

```python
model = MLP3(hp.hidden1, hp.hidden2, hp.dropout).to(device)
param_groups = [
    {"params": list(model.stem.parameters()),
     "lr": hp.base_lr * hp.stem_lr_mul,
     "initial_lr": hp.base_lr * hp.stem_lr_mul,
     "weight_decay": hp.weight_decay},
    {"params": list(model.head.parameters()),
     "lr": hp.base_lr * hp.head_lr_mul,
     "initial_lr": hp.base_lr * hp.head_lr_mul,
     "weight_decay": hp.weight_decay},
]
opt = torch.optim.SGD(param_groups, momentum=0.9, nesterov=True)
sched = WarmupCosine(opt, hp.epochs, hp.warmup_epochs, hp.min_lr_ratio)

# New torch.amp GradScaler API (only if CUDA/AMP enabled)
scaler = torch.amp.GradScaler("cuda", enabled=amp_enabled)
```

--------------------------------

### Get Empirical Spectral Density

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSmoothing.ipynb

Retrieve the ESD for a specific layer of the smoothed model.

```python
smoother_esd_31 = watcher.get_ESD(model=smoothed_vgg11, layer=31)
```

--------------------------------

### Configure WeightWatcher Logging

Source: https://github.com/calculatedcontent/weightwatcher/blob/master/examples/WW-SVDSharpness-VGG11.ipynb

Sets up basic logging for the application and WeightWatcher library to the WARN level.

```python
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
```

```python
import weightwatcher as ww
logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.INFO)
```