### Install Python Wrapper for GraphViz (Pip)

Source: https://github.com/waleedka/hiddenlayer/blob/master/README.md

Installs the Python wrapper for GraphViz using pip after manually installing GraphViz. Use this method if you are not using Conda.

```bash
pip3 install graphviz
```

--------------------------------

### Create Data Directory

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Creates a directory named 'test_data' in the project root for storing datasets. This is a common setup step before downloading datasets.

```python
# Create data directory in project root (to download dataset to)
ROOT_DIR = os.path.dirname(os.path.abspath(os.getcwd()))
DATA_DIR = os.path.join(ROOT_DIR, "test_data")
```

--------------------------------

### Install HiddenLayer using PIP

Source: https://github.com/waleedka/hiddenlayer/blob/master/README.md

Installs the latest stable release of the HiddenLayer library directly from PyPI using pip.

```bash
pip install hiddenlayer
```

--------------------------------

### Install GraphViz and Python Wrapper (Conda)

Source: https://github.com/waleedka/hiddenlayer/blob/master/README.md

Installs GraphViz and its Python wrapper using Conda. This is the recommended method if you are using the Conda package manager.

```bash
conda install graphviz python-graphviz
```

--------------------------------

### Install HiddenLayer Directly from GitHub using PIP

Source: https://github.com/waleedka/hiddenlayer/blob/master/README.md

Installs the latest version of the HiddenLayer library directly from its GitHub repository using pip. This is useful for accessing the most recent updates.

```bash
pip install git+https://github.com/waleedka/hiddenlayer.git
```

--------------------------------

### Import TensorFlow and HiddenLayer Libraries

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Imports necessary libraries for TensorFlow and HiddenLayer. Ensure these are installed before running.

```python
import os
import time
import random
import numpy as np
import tensorflow as tf
import hiddenlayer as hl
```

--------------------------------

### Clone HiddenLayer Repository and Install in Dev Mode

Source: https://github.com/waleedka/hiddenlayer/blob/master/README.md

Clones the HiddenLayer repository from GitHub and installs it in development mode using pip. This allows for local modifications and customization.

```bash
# Clone the repository
git clone git@github.com:waleedka/hiddenlayer.git
cd hiddenlayer

# Install in dev mode
pip install -e .
```

--------------------------------

### Build and Display TensorFlow Graph with HiddenLayer

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_graph.ipynb

This snippet shows how to construct a TensorFlow graph, build a HiddenLayer graph representation, and display it. It requires TensorFlow and HiddenLayer to be installed.

```python
import tensorflow as tf
import hiddenlayer as hl
from hiddenlayer import nets

with tf.Session() as sess:
    with tf.Graph().as_default() as tf_graph:
        # Setup input placeholder
        inputs = tf.placeholder(tf.float32, shape=(1, 231, 231, 3))
        # Build model
        predictions, _ = nets.overfeat.overfeat(inputs)
        # Build layout
        hl_graph = hl.build_graph(tf_graph)
```

--------------------------------

### Initialize History and Canvas

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Initializes hiddenlayer's History object for logging and Canvas object for visualization. These are typically set up before starting the training loop.

```python
step = (0, 0)  # tuple of (epoch, batch_ix)
cifar_history = hl.History()
cifar_canvas = hl.Canvas()
```

--------------------------------

### Clear Keras Session and Set Learning Phase

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Clears the current Keras session and sets the learning phase to inference mode (0). This is a common setup step before building or visualizing models for evaluation.

```python
K.clear_session()
K.set_learning_phase(0)
```

--------------------------------

### Text-Based Progress for Non-GUI Environments

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Demonstrates using the `progress()` method of the History object to display training status as text. This is useful for running training on servers without a graphical interface.

```python
# If the training loop is running on a server without a GUI, then use the `History` `progress()` method to print a text status.
```

--------------------------------

### Build InceptionV3 Model Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Initializes an InceptionV3 model and prepares for graph visualization. This snippet sets up the model and clears the Keras session.

```python
# Create a new graph and set the learning phase to inference
K.clear_session()
K.set_learning_phase(0)

# Build the model
model = InceptionV3(input_shape=(299, 299, 3))


```

--------------------------------

### Visualize Training Data Samples

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Display the first 5 training images using hiddenlayer.show_images to visually inspect the dataset.

```Python
# Visual check
hl.show_images(x_train[:5])
```

--------------------------------

### Basic Training Loop with History and Canvas

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Simulates a training loop, logging loss and accuracy metrics using HiddenLayer's History and Canvas for visualization. Use this to track single experiment progress.

```python
# A History object to store metrics
history1 = hl.History()

# A Canvas object to draw the metrics
canvas1 = hl.Canvas()

# Simulate a training loop with two metrics: loss and accuracy
loss = 1
accuracy = 0
for step in range(800):
    # Fake loss and accuracy
    loss -= loss * np.random.uniform(-.09, 0.1)
    accuracy = max(0, accuracy + (1 - accuracy) * np.random.uniform(-.09, 0.1))

    # Log metrics and display them at certain intervals
    if step % 10 == 0:
        # Store metrics in the history object
        history1.log(step, loss=loss, accuracy=accuracy)

        # Plot the two metrics in one graph
        canvas1.draw_plot([history1["loss"], history1["accuracy"]])

        time.sleep(0.1)
```

--------------------------------

### Import necessary libraries

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_graph.ipynb

Imports PyTorch, torchvision models, and the HiddenLayer library for graph visualization.

```python
import torch
import torchvision.models
import hiddenlayer as hl
```

--------------------------------

### Initialize CIFAR10 Dataset

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Initializes the CIFAR10 dataset object with a specified batch size and data directory.

```python
# CIFAR10 dataset and model
batch_size = 50
cifar10 = CIFAR10(batch_size=batch_size, data_dir=DATA_DIR)
```

--------------------------------

### Build AlexNet graph with custom theme

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_graph.ipynb

Builds the graph for an AlexNet model and applies a custom blue color theme. The resulting graph object is stored and can be displayed.

```python
# AlexNet
model = torchvision.models.alexnet()

# Build HiddenLayer graph
hl_graph = hl.build_graph(model, torch.zeros([1, 3, 224, 224]))

# Use a different color theme
hl_graph.theme = hl.graph.THEMES["blue"].copy()  # Two options: basic and blue
hl_graph
```

--------------------------------

### Import CIFAR10 Model

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Imports the CIFAR10 model class and sets up the data directory for dataset downloads.

```python
# Import the model code
from tf_cifar10 import CIFAR10

# Create data directory in project root (to download dataset to)
ROOT_DIR = os.path.dirname(os.path.abspath(os.getcwd()))
DATA_DIR = os.path.join(ROOT_DIR, "test_data")
```

--------------------------------

### Saving and Loading Training Histories

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Saves the current training history to a file and loads it back into new History objects. Use this to persist experiment results or resume training.

```python
# Save experiments 1 and 2
history1.save("experiment1.pkl")
history2.save("experiment2.pkl")
```

```python
# Load them again. To verify it's working, load them into new objects.
h1 = hl.History()
h2 = hl.History()
h1.load("experiment1.pkl")
h2.load("experiment2.pkl")
```

```python
# Show a summary of the experiment
h1.summary()
```

```python
# Draw a plot of experiment 2
hl.Canvas().draw_plot(h2["accuracy"])
```

--------------------------------

### Print Training Progress

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Logs and displays the current step's loss and accuracy during training. Call this within your training loop.

```python
history1.progress()
```

--------------------------------

### Load and Prepare MNIST Dataset

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Load the MNIST dataset using the default train/validation split and extract image dimensions and the number of classes.

```Python
# Load the data using the default train/validation split
(x_train, y_train), (x_val, y_val) = mnist.load_data()

# Extract image size
_, x_height, x_width = x_train.shape

# Extract number of classes
y_classes = len(np.unique(y_train))
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Import all required libraries for building and training the Keras model, including data loading, model architecture, optimization, callbacks, and evaluation metrics.

```Python
from tensorflow.keras.datasets import mnist
from tensorflow.keras.applications.vgg16 import VGG16
import tensorflow.keras.backend as K
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Flatten, Dense, Dropout
from tensorflow.keras.callbacks import Callback
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.optimizers import Adam
from sklearn.metrics import confusion_matrix, accuracy_score
import cv2
import itertools
from matplotlib import pyplot as plt
```

--------------------------------

### Build Xception Model Graph with Custom Transforms

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Builds an Xception model graph and applies custom transforms to group nodes into Xception modules. It folds Conv-BatchNorm, SeparableConv-ConvBn, and Relu-SepConvConvBn sequences, then combines them into an 'Xception' module.

```python
# Create a new graph and set the learning phase to inference
K.clear_session()
K.set_learning_phase(0)

# Build the model
model = Xception(input_shape=(399, 399, 3))

# Define custom transforms to group nodes of the Xception module
transforms = [
    # Build basic folds first
    ht.Fold("Conv > BatchNorm", "ConvBn"),
    # ht.Fold("ConvBn > Relu", "ConvBnRelu"),
    # Next, build higher level folds that use basic folds
    ht.Fold("SeparableConv > ConvBn", "SepConvConvBn"),
    ht.Fold("Relu > SepConvConvBn", "ReluSepConvConvBn"),
    # Finally, build the Xception module
    ht.Fold("ReluSepConvConvBn > ReluSepConvConvBn > ReluSepConvConvBn > Add", "Xception", "Xception Module"),
    # Fold repeated nodes
    ht.FoldDuplicates(),
]

# Build model graph view
hl_graph = hl.build_graph(K.get_session().graph, transforms=transforms)

# Display graph view
hl_graph
```

--------------------------------

### Build Model Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Builds and visualizes the computational graph of the PyTorch model using hiddenlayer. Requires the model and a dummy input tensor.

```python
hl.build_graph(model, torch.zeros([1, 3, 32, 32]).to(device))
```

--------------------------------

### Save Training Graph Snapshot

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Saves a snapshot of the current training graphs (loss and accuracy) to a PNG file. Call this periodically within your training loop.

```python
# Print a text progress status in the loop
history.progress()

# Occasionally, save a snapshot of the graphs.
canvas.draw_plot([h["loss"], h["accuracy"]])
canvas.save("training_graph.png")
```

--------------------------------

### Import Keras Models and HiddenLayer

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Imports necessary Keras pre-trained models and HiddenLayer components for graph visualization. Ensures GPUs are not used if not needed.

```python
# Hide GPUs. Not needed for this demo.
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""

# Import Keras pre-trained models
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.applications.vgg19 import VGG19
from tensorflow.keras.applications.xception import Xception
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.applications.inception_v3 import InceptionV3
from tensorflow.keras.applications.mobilenet import MobileNet
from tensorflow.keras.applications.densenet import DenseNet121
import tensorflow.keras.backend as K

import hiddenlayer as hl
import hiddenlayer.transforms as ht
```

--------------------------------

### Build VGG16 graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_graph.ipynb

Builds the graph for a VGG16 model and renders it. The graph is automatically displayed in Jupyter Notebook environments.

```python
# VGG16 with BatchNorm
model = torchvision.models.vgg16()

# Build HiddenLayer graph
# Jupyter Notebook renders it automatically
hl.build_graph(model, torch.zeros([1, 3, 224, 224]))
```

--------------------------------

### Import Libraries

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Imports necessary libraries for HiddenLayer and numerical operations.

```python
import os
import time
import random
import numpy as np
import hiddenlayer as hl
```

--------------------------------

### Build graph with framework transforms only

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_graph.ipynb

Builds the graph for a model using only framework transforms, which map PyTorch or TensorFlow graphs into a standard format based on ONNX naming conventions. This is an alternative to applying custom simplification transforms.

```python
# Without simplification transforms, but include framework transforms
# Framework transforms map PyTorch or TensorFlow graphs into a standard format
# based, mostly, on ONNX naming conventions.

# hl.build_graph(model, torch.zeros([1, 3, 224, 224]), transforms=[])
```

--------------------------------

### Set Learning Phase to Training

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Configures the Keras backend to operate in training mode. This is a global setting for the Keras session.

```python
# Set the learning phase to training
K.set_learning_phase(1)
```

--------------------------------

### Determine Device (CPU/GPU)

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Determines whether to use the CUDA-enabled GPU or the CPU for computation. Prints the selected device.

```python
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("device = ", device)
```

--------------------------------

### Comparing Experiments with Multiple Canvases

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Compares two training experiments by logging metrics to separate History objects and plotting them together using a Canvas. Use this to evaluate different model configurations or hyperparameters side-by-side.

```python
# New history and canvas objects
history2 = hl.History()
canvas2 = hl.Canvas()

# Simulate a training loop with two metrics: loss and accuracy
loss = 1
accuracy = 0
for step in range(800):
    # Fake loss and accuracy
    loss -= loss * np.random.uniform(-.09, 0.1)
    accuracy = max(0, accuracy + (1 - accuracy) * np.random.uniform(-.09, 0.1))

    # Log metrics and display them at certain intervals
    if step % 10 == 0:
        history2.log(step, loss=loss, accuracy=accuracy)

        # Draw two plots
        # Encluse them in a "with" context to ensure they render together
        with canvas2:
            canvas2.draw_plot([history1["loss"], history2["loss"]],
                              labels=["Loss 1", "Loss 2"])
            canvas2.draw_plot([history1["accuracy"], history2["accuracy"]],
                              labels=["Accuracy 1", "Accuracy 2"])
        time.sleep(0.1)
```

--------------------------------

### Run TensorFlow Training Loop

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Executes the training loop for the CIFAR10 model using TensorFlow. It initializes the session, iterates through epochs and batches, fetches data, performs training steps, logs metrics, and visualizes the training progress with HiddenLayer.

```python
# Run training loop for a few epochs
with tf_graph.as_default():
    # Instantiate history and canvas
    cifar_history = hl.History()
    cifar_canvas = hl.Canvas()

    # Setup TF training session
    sess = tf.Session()
    
    # Run the initializer
    sess.run(tf.global_variables_initializer())

    for epoch in range(3):
        batches, _ = divmod(cifar10.train_len, batch_size)
        for batch in range(batches):
            # Fetch training samples
            _input = cifar10.train_data[batch*batch_size : (batch+1)*batch_size]
            _output = cifar10.train_labels[batch*batch_size : (batch+1)*batch_size]

            # Train
            _, loss, accuracy = sess.run([optimizer, loss_op, accuracy_op], 
                                         feed_dict={inputs : _input, outputs : _output})

            # Print stats
            if batch & batch % 100 == 0:
                # Log metrics: loss, accuracy, and one weights matrix
                conv1_weights = tf_graph.get_tensor_by_name('conv1/conv2d/kernel:0').eval(session=sess)
                cifar_history.log((epoch, batch),
                                  loss=loss,
                                  accuracy=accuracy,
                                  conv1_weights=conv1_weights)
                # Display graphs
                with cifar_canvas:
                    cifar_canvas.draw_plot([cifar_history["loss"], cifar_history["accuracy"]])
                    cifar_canvas.draw_hist(cifar_history["conv1_weights"])
```

--------------------------------

### Load CIFAR10 Dataset

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Loads the CIFAR10 dataset using PyTorch's torchvision.datasets. Includes transformations to convert images to tensors. Sets up training and testing data loaders.

```python
# CIFAR10 Dataset
t = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.CIFAR10(DATA_DIR, train=True, download=True, transform=t)
test_dataset = datasets.CIFAR10(DATA_DIR, train=False, download=True, transform=t)

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=50, shuffle=True)
testloader = torch.utils.data.DataLoader(test_dataset, batch_size=50, shuffle=True)
```

--------------------------------

### Import TensorFlow and HiddenLayer Libraries

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_graph.ipynb

Imports necessary libraries for TensorFlow graph manipulation and visualization with HiddenLayer. It also hides GPUs as they are not needed for this demo.

```python
import os
import tensorflow as tf
import tensorflow.contrib.slim.nets as nets
import hiddenlayer as hl
import hiddenlayer.transforms as ht

# Hide GPUs. Not needed for this demo.
os.environ["CUDA_VISIBLE_DEVICES"] = ""
```

--------------------------------

### Inception v1 with Simplified Inception Modules

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_graph.ipynb

Applies custom transforms to simplify the Inception v1 graph, folding inception blocks and Conv-Relu layers. This demonstrates advanced graph manipulation using `ht.Fold` and `ht.FoldDuplicates`.

```python
# Define custom transforms to replice the default ones
transforms = [
    # Fold inception blocks into one node
    ht.Fold("""
        ( (MaxPool > Conv > Relu) | \
          (Conv > Relu > Conv > Relu) | \
          (Conv > Relu > Conv > Relu) | \
          (Conv > Relu)
        ) > Concat
        """, "Inception", "Inception Module"),
    # Fold Conv and Relu together if they come together
    ht.Fold("Conv > Relu", "ConvRelu"),
    # Fold repeated nodes
    ht.FoldDuplicates(),
]

with tf.Session() as sess:
    with tf.Graph().as_default() as tf_graph:
        # Setup input placeholder
        inputs = tf.placeholder(tf.float32, shape=(1, 224, 224, 3))
        # Build model
        predictions, _ = nets.inception.inception_v1(inputs)
        # Build layout
        hl_graph = hl.build_graph(tf_graph, transforms=transforms)

# Display
hl_graph.theme = hl.graph.THEMES["blue"].copy()
hl_graph
```

--------------------------------

### PyTorch Training Loop

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

This snippet shows a complete training loop for a PyTorch model. It iterates through epochs and batches, performs forward and backward passes, updates model weights, and logs metrics. Ensure `train_loader`, `device`, `optimizer`, `criterion`, `model`, `cifar_history`, and `cifar_canvas` are properly initialized before use.

```python
for epoch in range(2):
    train_iter = iter(train_loader)
    for batch_ix, (inputs, labels) in enumerate(train_iter):
        # Update global step counter
        step = (epoch, batch_ix)

        optimizer.zero_grad()
        inputs = inputs.to(device)
        labels = labels.to(device)

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # Print statistics
        if batch_ix and batch_ix % 100 == 0:
            # Compute accuracy
            pred_labels = np.argmax(outputs.detach().cpu().numpy(), 1)
            accuracy = np.mean(pred_labels == labels.detach().cpu().numpy())
            # Log metrics to history
            cifar_history.log((epoch, batch_ix),
                              loss=loss, accuracy=accuracy,
                              conv1_weight=model.features[0].weight)
            # Visualize metrics
            with cifar_canvas:
                cifar_canvas.draw_plot([cifar_history["loss"], cifar_history["accuracy"]])
                cifar_canvas.draw_image(cifar_history["conv1_output"])
                cifar_canvas.draw_hist(cifar_history["conv1_weight"])
```

--------------------------------

### Build ResNet50 Graph with Bottleneck and Residual Blocks

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Builds a ResNet50 model graph and applies custom transforms to group nodes into Bottleneck and Residual blocks. It folds Conv-BatchNorm, Conv-BatchNorm-Relu, and then combines them into the respective block types.

```python
# Create a new graph and set the learning phase to inference
K.clear_session()
K.set_learning_phase(0)

# Build the model
model = ResNet50(input_shape=(224, 224, 3))

# Define custom transforms to group nodes of the bottleneck and residual blocks
transforms = [
    # Build basic folds first
    ht.Fold("Conv > BatchNorm", "ConvBn"),
    ht.Fold("ConvBn > Relu", "ConvBnRelu"),
    # Fold bottleneck blocks
    ht.Fold("((ConvBnRelu > ConvBnRelu > ConvBn) | ConvBn) > Add > Relu", "BottleneckBlock", "Bottleneck Block"),
    # Fold residual blocks
    ht.Fold("ConvBnRelu > ConvBnRelu > ConvBn > Add > Relu", "ResBlock", "Residual Block"),
    ht.FoldDuplicates(),
]

# Build model graph view and display it
hl_graph = hl.build_graph(K.get_session().graph, transforms=transforms)
hl_graph
```

--------------------------------

### Build ResNet101 graph with custom transforms

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_graph.ipynb

Builds the graph for a ResNet101 model, applying custom transforms to fold residual and bottleneck blocks for simplification. This helps in visualizing complex architectures.

```python
# Resnet101
model = torchvision.models.resnet101()

# Rather than using the default transforms, build custom ones to group
# nodes of residual and bottleneck blocks.
transforms = [
    # Fold Conv, BN, RELU layers into one
    hl.transforms.Fold("Conv > BatchNorm > Relu", "ConvBnRelu"),
    # Fold Conv, BN layers together
    hl.transforms.Fold("Conv > BatchNorm", "ConvBn"),
    # Fold bottleneck blocks
    hl.transforms.Fold("""
        ((ConvBnRelu > ConvBnRelu > ConvBn) | ConvBn) > Add > Relu
        """, "BottleneckBlock", "Bottleneck Block"),
    # Fold residual blocks
    hl.transforms.Fold("""ConvBnRelu > ConvBnRelu > ConvBn > Add > Relu""",
                       "ResBlock", "Residual Block"),
    # Fold repeated blocks
    hl.transforms.FoldDuplicates(),
]

# Display graph using the transforms above
hl.build_graph(model, torch.zeros([1, 3, 224, 224]), transforms=transforms)
```

--------------------------------

### Train Keras Model with Callbacks

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Trains the Keras model using the specified training data, batch size, and number of epochs. Includes custom callbacks for monitoring training progress.

```python
# Train the model
model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, callbacks=callbacks);

```

--------------------------------

### Custom Visualizations with Mixed Metrics

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Uses a custom Canvas to display a pie chart of accuracy, a plot of accuracy and loss, and an image visualization within a single training loop. This showcases combining various visualization types.

```python
history3 = hl.History()
canvas3 = MyCanvas()  # My custom Canvas

# Simulate a training loop
loss = 1
accuracy = 0
for step in range(400):
    # Fake loss and accuracy
    loss -= loss * np.random.uniform(-.09, 0.1)
    accuracy = max(0, accuracy + (1 - accuracy) * np.random.uniform(-.09, 0.1))

    if step % 10 == 0:
        # Log loss and accuracy
        history3.log(step, loss=loss, accuracy=accuracy)

        # Log a fake image metric (e.g. image generated by a GAN)
        image = np.sin(np.sum(((np.indices([32, 32]) - 16) * 0.5 * accuracy) ** 2, 0))
        history3.log(step, image=image)
        
        # Display
        with canvas3:
            canvas3.draw_pie(history3["accuracy"])
            canvas3.draw_plot([history3["accuracy"], history3["loss"]])
            canvas3.draw_image(history3["image"])

        time.sleep(0.1)
```

--------------------------------

### Build and Display VGG19 Graph with Custom Transforms

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Builds a VGG19 model graph and applies custom transforms to group nodes like Conv-Relu and Linear-Relu layers. Also folds duplicate nodes and customizes the graph theme.

```python
# Build the model
model = VGG19(input_shape=(224, 224, 3))

# Define custom transforms to group nodes differently
transforms = [
    # Build basic folds first
    ht.Fold("Conv > Relu", "ConvRelu"),
    # Display fully-connected layers differently
    ht.Fold("Linear > Relu", "FC", "FC"),
    # Fold repeated nodes
    ht.FoldDuplicates(),
]

# Build model graph view
hl_graph = hl.build_graph(K.get_session().graph, transforms=transforms)

# Customize the theme. The theme is a simple dict defined in graph.py
hl_graph.theme.update({
    "fill_color": "#789263",
    "outline_color": "#789263",
    "font_color": "#FFFFFF",
})

# Display graph view
hl_graph
```

--------------------------------

### Build and Display Inception v1 Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_graph.ipynb

Builds a TensorFlow graph for the Inception v1 model and visualizes it using HiddenLayer. This snippet showcases the visualization of a different pre-trained model.

```python
with tf.Session() as sess:
    with tf.Graph().as_default() as tf_graph:
        # Setup input placeholder
        inputs = tf.placeholder(tf.float32, shape=(1, 224, 224, 3))
        # Build model
        predictions, _ = nets.inception.inception_v1(inputs)
        # Build layout
        hl_graph = hl.build_graph(tf_graph)

# Display
hl_graph
```

--------------------------------

### ResNet-50 with Simplified Residual and Bottleneck Blocks

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_graph.ipynb

Applies custom transforms to group residual and bottleneck blocks in a ResNet-50 graph. This demonstrates using `ht.Fold` for complex pattern matching and grouping in graph visualization.

```python
# Custom transforms to group nodes of residual and bottleneck blocks
transforms = [
    # Fold Pad into the Conv that follows it 
    ht.Fold("Pad > Conv", "__last__"),
    # Fold Conv/Relu
    ht.Fold("Conv > Relu", "ConvRelu"),
    # Fold bottleneck blocks
    hl.transforms.Fold("""((ConvRelu > ConvRelu > Conv) | Conv) > Add > Relu
        """, "BottleneckBlock", "Bottleneck Block"),
    # Fold residual blocks
    hl.transforms.Fold("ConvRelu > ConvRelu > Conv > Add > Relu",
                       "ResBlock", "Residual Block"),
]

# Build TensorFlow graph
with tf.Session() as sess:
    with tf.Graph().as_default() as tf_graph:
        # Setup input placeholder
        inputs = tf.placeholder(tf.float32, shape=(1, 224, 224, 3))
        # Build model
        predictions, _ = nets.resnet_v1.resnet_v1_50(inputs)
        # Build HiddenLayer graph
        hl_graph = hl.build_graph(tf_graph, transforms=transforms)

# Customize the theme. The theme is a simple dict defined in graph.py
hl_graph.theme.update({
    "fill_color": "#789263",
    "outline_color": "#789263",
    "font_color": "#FFFFFF",
})


```

--------------------------------

### Build Keras Model Architecture

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Defines a classifier head with dropout and a softmax layer, then integrates it with a convolutional base to form the final model.

```python
# two fully-connected layers with dropout and a final softmax layer
classifier = Flatten(input_shape=convnet.output_shape[1:])(convnet.output)
classifier = Dense(512, activation='relu')(classifier) # fc1
classifier = Dropout(0.5)(classifier)
classifier = Dense(512, activation='relu')(classifier) # fc2
classifier = Dropout(0.5)(classifier)
classifier = Dense(10, activation='softmax')(classifier) # softmax classifier for classes 0..9

```

```python
# Finally, put the model together
model = Model(inputs=convnet.input, outputs=classifier)

```

--------------------------------

### Sanity Check Preprocessed Tensors

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Log the shapes and data types of the preprocessed dataset tensors using hiddenlayer.write to confirm the transformations.

```Python
# Sanity check
hl.write("x_train", x_train)
hl.write("x_val", x_val)
hl.write("y_train", y_train)
hl.write("y_val", y_val)
```

--------------------------------

### Customizing Graph Folding Rules

Source: https://github.com/waleedka/hiddenlayer/blob/master/README.md

This rule folds all the nodes of a bottleneck block of a ResNet101 into one node. It uses graph expressions and transforms for customization.

```python
ht.Fold("((ConvBnRelu > ConvBnRelu > ConvBn) | ConvBn) > Add > Relu", 
            "BottleneckBlock", "Bottleneck Block"),
```

--------------------------------

### Visualize Model Architecture

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Generate and display a graph of the model's architecture using HiddenLayer's build_graph function, based on the Keras session graph.

```Python
# Visual check of the "before" model architecture using HiddenLayer
hl.build_graph(K.get_session().graph)
```

--------------------------------

### Sanity Check Dataset Tensors

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Use hiddenlayer.write to log the shapes and data types of the loaded dataset tensors for verification.

```Python
# Sanity check
hl.write("x_train", x_train)
hl.write("x_val", x_val)
hl.write("y_train", y_train)
hl.write("y_val", y_val)
print(f"Sample size: {x_height}x{x_width}")
print(f"Number of classes: {y_classes}")
```

--------------------------------

### Inspect CIFAR10 Data and Labels

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Writes the CIFAR10 training and testing data and labels to HiddenLayer for inspection. This helps in understanding the dataset's structure and values.

```python
# Inspect data and labels
hl.write("cifar10.train_data", cifar10.train_data)
hl.write("cifar10.train_labels", cifar10.train_labels)
hl.write("cifar10.test_data", cifar10.test_data)
hl.write("cifar10.test_labels", cifar10.test_labels)
```

--------------------------------

### Preprocess Dataset for VGG16

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Adapt the MNIST dataset to meet VGG16 requirements: normalize pixel values to [0,1], resize images to the expected input shape, and replicate grayscale channels to RGB.

```Python
# The VGG16 backbone was trained on RGB data in the [0.,1.] range
# -> convert data to float and perform min-max normalization
x_train = x_train.astype('float32') / 255.
x_val = x_val.astype('float32') / 255

# The VGG16 backbone was trained on color RGB data (imagenet)
# -> resize the samples from (#, 28, 28) to (#, input_shape[0], input_shape[1])
if input_shape[0:2] != (x_height, x_width):
    x_train = [cv2.resize(x, input_shape[0:2]) for x in x_train]
    x_val = [cv2.resize(x, input_shape[0:2]) for x in x_val]

# -> reshape the samples from (#, H, W) to (#, H, W, 3)
#    by copying the "gray" MNIST images identically across the 3 RGB channels
x_train = np.stack((x_train, x_train, x_train), axis=-1)
x_val = np.stack((x_val, x_val, x_val), axis=-1)
```

--------------------------------

### Build TensorFlow Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Constructs the TensorFlow computation graph, including placeholders for inputs and outputs, the model definition, loss operation, optimizer, and accuracy metrics.

```python
# Build the TensorFlow Graph
tf_graph = tf.Graph()

with tf_graph.as_default():
    # Placeholders
    inputs = tf.placeholder(
        tf.float32, 
        shape=(batch_size, cifar10.img_size, cifar10.img_size, cifar10.num_channels))
    outputs = tf.placeholder(tf.float32, shape=[batch_size, cifar10.num_classes])
    # Build model
    predictions = cifar10.model(inputs)

    # Loss and optimizer
    loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(
        logits=predictions, labels=outputs))
    optimizer = tf.train.MomentumOptimizer(
        learning_rate=0.01, momentum=0.9).minimize(loss_op)

    # Metrics
    accurate_preds = tf.equal(tf.argmax(predictions, axis=1), tf.argmax(outputs, axis=1))
    accuracy_op = tf.reduce_mean(tf.cast(accurate_preds, tf.float32))
```

--------------------------------

### Build VGG16 Feature Extractor

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Instantiate the VGG16 model from Keras applications, specifying 'imagenet' weights, excluding the top classification layer, and setting the input shape.

```Python
# Build the VGG16 model as a feature extractor
convnet = VGG16(weights='imagenet', include_top=False, input_shape=input_shape)
```

--------------------------------

### Extending Canvas for Custom Metrics Visualization

Source: https://github.com/waleedka/hiddenlayer/blob/master/README.md

Extends the Canvas class to add a new method for drawing a pie chart to visualize a metric, such as model accuracy. Requires NumPy for clipping values.

```python
import numpy as np

class MyCanvas(hl.Canvas):
    """Extending Canvas to add a pie chart method."""
    def draw_pie(self, metric):
        # set square aspect ratio
        self.ax.axis('equal')
        # Get latest value of the metric
        value = np.clip(metric.data[-1], 0, 1)
        # Draw pie chart
        self.ax.pie([value, 1-value], labels=["Accuracy", ""])
```

--------------------------------

### Build Classifier on Top of VGG16

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Construct a custom classifier by adding layers on top of the VGG16 feature extractor. This snippet is incomplete and requires further definition.

```Python
# Build our own classifier on top of the VGG16 feature extractor using
```

--------------------------------

### Compile Keras Model for Classifier Training

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Compiles the Keras model with the Adam optimizer and categorical crossentropy loss, specifying accuracy as a metric. This prepares the model for training the classifier head.

```python
# Compile the model using the Adam optimizer
model.compile(loss='categorical_crossentropy', optimizer=Adam(1e-3), metrics=['acc'])

```

--------------------------------

### Custom Visualization: Pie Chart for Accuracy

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Extends the HiddenLayer Canvas class to add a custom visualization method for drawing accuracy as a pie chart. This demonstrates how to integrate custom plotting functions.

```python
class MyCanvas(hl.Canvas):
    """Extending Canvas to add a pie chart method."""
    
    def draw_pie(self, metric):
        # Method name must start with 'draw_' for the Canvas to automatically manage it
        
        # Use the provided matplotlib Axes in self.ax
        self.ax.axis('equal')  # set square aspect ratio

        # Get latest value of the metric
        value = np.clip(metric.data[-1], 0, 1)
        
        # Draw pie chart
        self.ax.pie([value, 1-value], labels=["Accuracy", ""])
```

--------------------------------

### Set Matplotlib Backend

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Configures the matplotlib backend to 'Agg' for non-interactive plotting. This must be done before importing hiddenlayer.

```python
import matplotlib
matplotlib.use("Agg")
```

--------------------------------

### Build and Display VGG-16 Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_graph.ipynb

Builds a TensorFlow graph for the VGG-16 model and visualizes it using HiddenLayer. This snippet shows the basic usage of `hl.build_graph`.

```python
with tf.Session() as sess:
    with tf.Graph().as_default() as tf_graph:
        # Setup input placeholder
        inputs = tf.placeholder(tf.float32, shape=(1, 224, 224, 3))
        # Build model
        predictions, _ = nets.vgg.vgg_16(inputs)
        # Build HiddenLayer graph
        hl_graph = hl.build_graph(tf_graph)

# Display graph
# Jupyter Notebook renders it automatically
hl_graph
```

--------------------------------

### Build raw graph without any transforms

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_graph.ipynb

Builds the raw graph of a model by removing all simplification and framework transforms. This displays the original, unmodified graph structure.

```python
# Remove all transforms completely
# Override both, simplicity transforms and framework transforms, to get
# the original raw graph.

# hl.build_graph(model, torch.zeros([1, 3, 224, 224]), transforms=[], framework_transforms=[])
```

--------------------------------

### Clear Session and Set Learning Phase to Inference

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Clears the current Keras session and sets the learning phase to inference mode. This is often done before building a new model or visualizing a graph for inference.

```python
# Create a new graph and set the learning phase to inference
K.clear_session()
K.set_learning_phase(0)
```

--------------------------------

### Custom Confusion Matrix Canvas

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Extends HiddenLayer's Canvas to include functionality for drawing confusion matrices. Requires matplotlib and itertools.

```python
class ConfusionMatrixCanvas(hl.Canvas):
    """Extending HiddenLayer's Canvas to plot a confusion matrix."""

    def __init__(self, classes):
        super().__init__()
        self.classes = range(0, classes)
    
    def draw_confusion_matrix(self, metric):
        """This function plots the confusion matrix.
        Ref:
            https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
        """
        # Get confusion matrix of the last step
        cm = metric.data[-1]

        # Build the plot
        plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
        tick_marks = np.arange(len(self.classes))
        plt.xticks(tick_marks, self.classes, rotation=45)
        plt.yticks(tick_marks, self.classes)

        thresh = cm.max() / 2.
        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
            plt.text(j, i, format(cm[i, j], 'd'),
                     horizontalalignment="center",
                     color="white" if cm[i, j] > thresh else "black")

        plt.ylabel('True label')
        plt.xlabel('Predicted label')
        plt.tight_layout()
        

```

--------------------------------

### Display TensorFlow Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/tf_train.ipynb

Builds and displays the TensorFlow computation graph using HiddenLayer. This provides a visual representation of the model's architecture.

```python
# Display graph
hl.build_graph(tf_graph)
```

--------------------------------

### Keras Callback for Confusion Matrix

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

A Keras callback that computes and visualizes the confusion matrix and accuracy during training. It uses HiddenLayer for history logging and visualization.

```python
class ConfusionMatrixCallback(Callback):
    """Extending Keras' Callback to compute a confusion matrix."""
    
    def __init__(self, x_val, y_val, y_classes):
        # Keep a reference to the validation data
        self.x_val = x_val
        self.y_val = [np.argmax(y) for y in y_val]
        
        # Use a HiddenLayer History object to store metrics
        self.history = hl.History()

        # Use our custom HiddenLayer Canvas object to draw the metrics
        self.canvas = ConfusionMatrixCanvas(y_classes)        
        
    def on_train_begin(self, logs={}):
        self.epoch = 0
              
    def on_epoch_end(self, batch, logs={}):
        # Generate predictions for validation data
        y_val_hat = self.model.predict(self.x_val)
        y_val_hat = [np.argmax(y) for y in y_val_hat]
              
        # Compute the accuracy and generate a confusion matrix using the validation data
        val_acc = accuracy_score(self.y_val, y_val_hat)
        cm = confusion_matrix(self.y_val, y_val_hat)
              
        # Store the metrics in the history object
        self.history.log(self.epoch, acc=logs.get('acc'), val_acc=val_acc, cm=cm)
        
        # Plot the metrics
        with self.canvas:
            self.canvas.draw_confusion_matrix(self.history["cm"])
            self.canvas.draw_plot([self.history["acc"], self.history["val_acc"]])
              
        # Onto the next epoch...
        self.epoch += 1

callbacks = [ConfusionMatrixCallback(x_val, y_val, y_classes)]

```

--------------------------------

### Write Dataset Information

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Writes dataset information (data and labels) to hiddenlayer's log. This is useful for inspecting dataset characteristics before training.

```python
# Print dataset stats
hl.write("train_dataset.data", train_dataset.train_data)
hl.write("train_dataset.labels", train_dataset.train_labels)
hl.write("test_dataset.data", test_dataset.test_data)
hl.write("test_dataset.labels", test_dataset.test_labels)
```

--------------------------------

### Register Activation Hook

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Registers a forward hook on a specific layer (e.g., the first convolutional layer) to intercept and log its activations during the forward pass. This is useful for analyzing intermediate feature maps.

```python
def activations_hook(self, inputs, output):
    """Intercepts the forward pass and logs activations.
    """
    batch_ix = step[1]
    if batch_ix and batch_ix % 100 == 0:
        # The output of this layer is of shape [batch_size, 16, 32, 32]
        # Take a slice that represents one feature map
        cifar_history.log(step, conv1_output=output.data[0, 0])
    
# A hook to extract the activations of intermediate layers
model.features[0].register_forward_hook(activations_hook);
```

--------------------------------

### Define CIFAR10 Convolutional Network

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb

Defines a simple convolutional neural network model for CIFAR10 classification using PyTorch's nn.Module. Includes convolutional layers, batch normalization, ReLU activation, max pooling, and adaptive pooling.

```python
# Simple Convolutional Network
class CifarModel(nn.Module):
    def __init__(self):
        super(CifarModel, self).__init__()
        
        self.features = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.Conv2d(16, 16, kernel_size=3, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),

            nn.Conv2d(16, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),

            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            
            nn.AdaptiveMaxPool2d(1)
        )
        self.classifier = nn.Sequential(
            nn.Linear(32, 32),
#             TODO: nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Linear(32, 10))

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

model = CifarModel().to(device)

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
```

--------------------------------

### Define Custom Transforms for MobileNet Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Defines custom transforms for visualizing a MobileNet graph, including folding ReLU6, Conv-BatchNorm, and SeparableConv blocks. It also includes a commented-out option to fold depthwise separable convolution blocks.

```python
# Build the model
model = MobileNet(input_shape=(224, 224, 3))

# Define custom transforms to group nodes into modules
transforms = [
    # Build basic folds first
    # Note that the activation function used by MobileNet is ReLU6
    # The authors of the MobileNet paper found it more robust than regular ReLU in fixed-point inference
    ht.Fold("Relu > Minimum > Maximum", "Relu", "ReLU6"),
    ht.Fold("Conv > BatchNorm", "ConvBn"),
    ht.Fold("ConvBn > Relu", "ConvBnRelu"),
    ht.Fold("SeparableConv > BatchNorm", "SeparableConvBn"),
    ht.Fold("SeparableConvBn > Relu", "SeparableConvBnRelu"),
    # You could further simplify the graph by folding nodes into "depthwise separable” convolution blocks
    # To do so, simply uncomment the line below:
    # ht.Fold("SeparableConvBnRelu > ConvBnRelu > Pad", "Depthwise Separable Conv Block", "Depthwise Separable Conv Block"),
    ht.FoldDuplicates(),
]

# Build model graph view and display it
hl_graph = hl.build_graph(K.get_session().graph, transforms=transforms)
hl_graph
```

--------------------------------

### Define Input Shape for VGG16

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Specify the input shape for the Keras VGG16 model, which is the minimum size of input images it will accept.

```Python
# Minimum size of input images Keras' VGG16 will accept
input_shape = (48, 48, 3)
```

--------------------------------

### Visualize Keras Model Graph with HiddenLayer

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_train.ipynb

Generates a visual representation of the Keras model's architecture using HiddenLayer. This is useful for understanding the model's structure.

```python
# Visual check of the "after" model architecture using HiddenLayer
hl.build_graph(K.get_session().graph)

```

--------------------------------

### Define Custom Transforms for DenseNet121 Graph

Source: https://github.com/waleedka/hiddenlayer/blob/master/demos/keras_graph.ipynb

Defines custom transforms for visualizing a DenseNet121 graph, focusing on folding BatchNorm-Relu-Conv layers into Residual Blocks. It also includes folding duplicates.

```python
# Build the model
model = DenseNet121(input_shape=(224, 224, 3))

# Define custom transforms to group nodes into modules
transforms = [
    # Build basic folds first
    ht.Fold("BatchNorm > Relu > Conv", "BnReluConv"),
    ht.Fold("BnReluConv > BnReluConv> Concat", "ResBlock", "Residual Block"),
    ht.FoldDuplicates(),
]

# Build model graph view and display it
hl_graph = hl.build_graph(K.get_session().graph, transforms=transforms)
hl_graph
```