### Install KD_Lib from Source

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/install.rst

Installs the latest unreleased version of KD_Lib directly from its GitHub repository. This method requires Git and involves cloning the repository and running the setup script.

```console
$ git clone https://github.com/SforAiDl/KD_Lib.git
$ cd KD_lib
$ python setup.py install
```

--------------------------------

### Install KD-Lib from Source

Source: https://github.com/sforaidl/kd_lib/blob/master/README.md

Installs the KD-Lib library by cloning the repository and running the setup script. This is the recommended installation method.

```shell
https://github.com/SforAiDl/KD_Lib.git
cd KD_Lib
python setup.py install
```

--------------------------------

### Install KD_Lib Stable Release via Pip

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/install.rst

Installs the latest stable release of KD_Lib from PyPI using pip. This is the recommended method for most users and requires Python 3.6+ and PyTorch.

```console
$ pip install KD-Lib
```

--------------------------------

### Set Up Local Development Environment

Source: https://github.com/sforaidl/kd_lib/blob/master/CONTRIBUTING.rst

Steps to set up a virtual environment and install KD_Lib for local development using virtualenvwrapper.

```shell
mkvirtualenv KD_Lib
cd KD_Lib/
python setup.py develop
```

--------------------------------

### Install Optuna using pip

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/optuna_with_KD_Lib.rst

This command installs the Optuna library using pip, a package installer for Python. Optuna is a hyperparameter optimization framework.

```console
$ pip install optuna
```

--------------------------------

### Install KD-Lib from Source

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/about.rst

This snippet shows how to clone the KD-Lib repository and install the latest unreleased version from source using setup.py. This method is recommended for developers who want the most up-to-date features.

```console
$ git clone https://github.com/SforAiDl/KD_Lib.git
$ cd KD_Lib
$ python setup.py install
```

--------------------------------

### Upgrade KD_Lib to Latest Version via Pip

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/install.rst

Upgrades an existing installation of KD_Lib to the most recent version available on PyPI using pip. This command ensures you have the latest features and bug fixes.

```console
$ pip install -U KD-Lib
```

--------------------------------

### Install KD-Lib from PyPI

Source: https://github.com/sforaidl/kd_lib/blob/master/README.md

Installs the KD-Lib library using pip, the Python package installer. This is a convenient way to get the latest stable version.

```shell
pip install KD-Lib
```

--------------------------------

### Install Optuna using conda

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/optuna_with_KD_Lib.rst

This command installs the Optuna library using conda, a package and environment manager. It installs Optuna from the conda-forge channel.

```console
$ conda install -c conda-forge optuna
```

--------------------------------

### Install KD-Lib using Pip

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/about.rst

This snippet demonstrates the standard installation of KD-Lib using pip, Python's package installer. It also includes instructions on how to upgrade to the latest version.

```console
$ pip install KD-Lib

$ pip install -U KD-Lib
```

--------------------------------

### Run Specific Tests

Source: https://github.com/sforaidl/kd_lib/blob/master/CONTRIBUTING.rst

Example command to run a subset of tests using py.test, targeting a specific test file.

```shell
py.test tests.test_KD_Lib
```

--------------------------------

### Implement CSKD with KD_Lib

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/CSKD.rst

This snippet demonstrates the basic implementation of Class-wise Self-Knowledge Distillation (CSKD) using the KD_Lib library. It covers dataset loading, model initialization, optimizer setup, and the training and evaluation process for a student network.

```python
import torch
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD.vision import CSKD
from KD_Lib.KD.vision.CSKD.sampler import load_dataset


# Define datasets, dataloaders, models and optimizers

# Note that the dataloader should sample according to pairwise sampling. (Refer to KD_Lib -> KD -> vision -> CSKD -> sampler.py)
train_loader, val_loader = load_dataset('cifar100', '~/data/', 'pair', batch_size=128)


student_model = <your model>

student_optimizer = optim.SGD(student_model.parameters(), 0.01)

# Now, this is where KD_Lib comes into the picture

distiller = CSKD(None, student_model, train_loader, test_loader, 
                      None, student_optimizer)  
distiller.train_student(epochs=5, plot_losses=True, save_model=True)    # Train the student network
distiller.evaluate(teacher=False)                                       # Evaluate the student network
distiller.get_parameters()                                              # A utility function to get the number of parameters in the teacher and the student network 

```

--------------------------------

### Format Code and Run Tests

Source: https://github.com/sforaidl/kd_lib/blob/master/CONTRIBUTING.rst

Commands to format code using Black and run tests using setup.py or py.test, including testing with tox for multiple Python versions.

```shell
black KD_Lib/
python setup.py test #or py.test
tox
```

--------------------------------

### Implement Vanilla Knowledge Distillation with KD_Lib

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/VanillaKD.rst

This snippet shows how to set up and use the VanillaKD class from KD_Lib for knowledge distillation. It includes dataset loading (MNIST), model and optimizer initialization, and the core training and evaluation steps for both teacher and student models. The `plot_losses` argument enables visualization of training progress.

```python
import torch
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import VanillaKD

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        )
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        )
    ),
    batch_size=32,
    shuffle=True,
)

teacher_model = <your model>
student_model = <your model>

teacher_optimizer = optim.SGD(teacher_model.parameters(), 0.01)
student_optimizer = optim.SGD(student_model.parameters(), 0.01)

# Now, this is where KD_Lib comes into the picture

distiller = VanillaKD(teacher_model, student_model, train_loader, test_loader, 
                      teacher_optimizer, student_optimizer)

distiller.train_teacher(epochs=5, plot_losses=True, save_model=True)    # Train the teacher network
distiller.train_student(epochs=5, plot_losses=True, save_model=True)    # Train the student network
distiller.evaluate(teacher=False)                                       # Evaluate the student network
distiller.get_parameters()                                              # A utility function to get the number of parameters in the teacher and the student network
```

--------------------------------

### Clone KD_Lib Repository

Source: https://github.com/sforaidl/kd_lib/blob/master/CONTRIBUTING.rst

Instructions for cloning the KD_Lib repository locally after forking it on GitHub.

```shell
git clone git@github.com:your_name_here/KD_Lib.git
```

--------------------------------

### Knowledge Distillation with VanillaKD

Source: https://github.com/sforaidl/kd_lib/blob/master/README.md

Demonstrates how to use the VanillaKD class from KD_Lib for knowledge distillation. It includes setting up datasets, models, optimizers, and training/evaluating the teacher and student models.

```python
import torch
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import VanillaKD

# This part is where you define your datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

teacher_model = <your model>
student_model = <your model>

teacher_optimizer = optim.SGD(teacher_model.parameters(), 0.01)
student_optimizer = optim.SGD(student_model.parameters(), 0.01)

# Now, this is where KD_Lib comes into the picture

distiller = VanillaKD(teacher_model, student_model, train_loader, test_loader, 
                      teacher_optimizer, student_optimizer)

distiller.train_teacher(epochs=5, plot_losses=True, save_model=True)    # Train the teacher network
distiller.train_student(epochs=5, plot_losses=True, save_model=True)    # Train the student network
distiller.evaluate(teacher=False)                                       # Evaluate the student network
distiller.get_parameters()                                              # A utility function to get the number of 
                                                                        # parameters in the  teacher and the student network
```

--------------------------------

### Train Student Model with KD_Lib Self-Training on MNIST

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/SelfTraining.rst

This snippet demonstrates how to initialize and use the SelfTraining class from KD_Lib to train a student model on the MNIST dataset. It covers setting up data loaders, a student model, an optimizer, and the training process.

```python
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import SelfTraining

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

# Set device to be trained on

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define student model

student_model = <your model>

# Define optimizer

student_optimizer = optim.SGD(student_model.parameters(), lr=0.01)


# Train using KD_Lib

distiller = SelfTraining(student_model, train_loader, test_loader, student_optimizer, 
                             device=device)  
distiller.train_student(epochs=5)                                      # Train the student model
distiller.evaluate()                                                    # Evaluate the student model

```

--------------------------------

### Hyperparameter Tuning for VanillaKD with Optuna

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/optuna_with_KD_Lib.rst

This Python code demonstrates how to tune hyperparameters for the VanillaKD algorithm using Optuna. It defines an objective function that Optuna will optimize by searching through suggested hyperparameter ranges for learning rate, momentum, optimizer type, temperature, distillation weight, and loss function.

```Python
import torch
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import VanillaKD

import optuna
from sklearn.externals import joblib

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,))
        ])
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,))
        ])
    ),
    batch_size=32,
    shuffle=True,
)

# Set device to be trained on

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Optuna requires defining an objective function 
# The hyperparameters are then optimized for maximizing/minimizing this objective function

def tune_VanillaKD(trial):

    teacher_model = <your model>
    student_model = <your model>

    # Define hyperparams and choose what ranges they should be trialled for

    lr = trial.suggest_float("lr", 1e-4, 1e-1)
    momentum = trial.suggest_float("momentum", 0.9, 0.99)
    optimizer = trial.suggest_categorical('optimizer',[optim.SGD, optim.Adam])

    teacher_optimizer = optimizer(teacher_model.parameters(), lr, momentum)
    student_optimizer = optimizer(student_model.parameters(), lr, momentum)

    temperature = trial.suggest_float("temperature", 5.0, 20.0)
    distil_weight = trial.suggest_float("distil_weight", 0.0, 1.0)

    loss_fn = trial.suggest_categorical("loss_fn",[nn.KLDivLoss(), nn.MSELoss()])

    # Instiate disitller object using KD_Lib and train

    distiller = VanillaKD(teacher_model, student_model, train_loader, test_loader,
                          teacher_optimizer, student_optimizer, loss_fn, 
                          temperature, distil_weight, device)
    distiller.train_teacher(epochs=10)
    distiller.train_student(epochs=10)
    test_accuracy = disitller.evaluate()

    # The objective function must return the quantity we're trying to maximize/minimize

    return test_accuracy

# Create a study

study = optuna.create_study(study_name="Hyperparameter Optimization",
                            direction="maximize")
study.optimize(tune_VanillaKD, n_trials=10)

# Access results

results = study.trials_dataframe()
results.head()

# Get best values of hyperparameter

for key, value in study.best_trial.__dict__.items():
    print("{} : {}".format(key, value))
    
# Write results of the study

joblib.dump(study, <your path>)

# Access results at a later time

study = joblib.load(<your path>)
results = study.trials_dataframe()
results.head()
```

--------------------------------

### Vanilla Knowledge Distillation with KD-Lib

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/about.rst

This Python snippet illustrates how to use the VanillaKD class from KD-Lib for knowledge distillation. It includes setting up datasets (MNIST), models, optimizers, and then training the teacher and student models, followed by evaluation.

```python
import torch
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import VanillaKD

# This part is where you define your datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

teacher_model = <your model>
student_model = <your model>

teacher_optimizer = optim.SGD(teacher_model.parameters(), 0.01)
student_optimizer = optim.SGD(student_model.parameters(), 0.01)

# Now, this is where KD_Lib comes into the picture

distiller = VanillaKD(teacher_model, student_model, train_loader, test_loader, 
                      teacher_optimizer, student_optimizer)  
distiller.train_teacher(epochs=5, plot_losses=True, save_model=True)    # Train the teacher network
distiller.train_student(epochs=5, plot_losses=True, save_model=True)    # Train the student network
distiller.evaluate(teacher=False)                                       # Evaluate the student network
distiller.get_parameters()                                              # A utility function to get the number of parameters in the teacher and the student network
```

--------------------------------

### Deploy Changes

Source: https://github.com/sforaidl/kd_lib/blob/master/CONTRIBUTING.rst

Commands for maintainers to deploy changes by version bumping, pushing to git, and pushing tags. Travis CI handles deployment to PyPI.

```shell
bumpversion patch # possible: major / minor / patch
git push
git push --tags
```

--------------------------------

### Deep Mutual Learning with KD-Lib

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/about.rst

This Python snippet demonstrates implementing Deep Mutual Learning (DML) using KD-Lib. It shows how to set up multiple student models, optimizers, and dataloaders, and then train them collectively while logging details to Tensorboard.

```python
import torch
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import DML
from KD_Lib.models import ResNet18, ResNet50                                   # To use models packaged in KD_Lib

# This part is where you define your datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

student_params = [4, 4, 4, 4, 4]
student_model_1 = ResNet50(student_params, 1, 10)
student_model_2 = ResNet18(student_params, 1, 10)

student_cohort = [student_model_1, student_model_2]

student_optimizer_1 = optim.SGD(student_model_1.parameters(), 0.01)
student_optimizer_2 = optim.SGD(student_model_2.parameters(), 0.01)

student_optimizers = [student_optimizer_1, student_optimizer_2]

# Now, this is where KD_Lib comes into the picture 

distiller = DML(student_cohort, train_loader, test_loader, student_optimizers, log=True, logdir="./Logs")

distiller.train_students(epochs=5)
distiller.evaluate()
distiller.get_parameters()
```

--------------------------------

### Train DML Students with KD_Lib on MNIST

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/DML.rst

This snippet demonstrates how to initialize and train multiple student models using the DML algorithm from KD_Lib. It covers setting up datasets, dataloaders, student models, optimizers, and the DML distiller for collective training and evaluation.

```Python
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import DML

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

# Set device to be trained on

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define a cohort of student models

student_model_1 = <your model>
student_model_2 = <your model>
student_model_3 = <your model>

student_cohort = (student_model_1, student_model_2, student_model_3)

# Make a list of optimizers for the models keeping in mind the order

student_optimizer_1 = optim.SGD(student_model_1.parameters(), 0.01)
student_optimizer_2 = optim.SGD(student_model_2.parameters(), 0.01)
student_optimizer_3 = optim.SGD(student_model_3.parameters(), 0.01)

optimizers = [student_optimizer_1, student_optimizer_2, student_optimizer_3]

# Train using KD_Lib

distiller = DML(student_cohort, train_loader, test_loader, optimizers,
                device=device)
distiller.train_students(epochs=5, plot_losses=True, save_model=True)  # Train the student cohort
distiller.evaluate()                                                    # Evaluate the student models

```

--------------------------------

### Deep Mutual Learning with DML

Source: https://github.com/sforaidl/kd_lib/blob/master/README.md

Illustrates the use of the DML class for Deep Mutual Learning, training multiple models collaboratively. It shows how to set up a cohort of student models and log training progress to Tensorboard.

```python
import torch
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import DML
from KD_Lib.models import ResNet18, ResNet50          # To use models packaged in KD_Lib

# Define your datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

student_params = [4, 4, 4, 4, 4]
student_model_1 = ResNet50(student_params, 1, 10)
student_model_2 = ResNet18(student_params, 1, 10)

student_cohort = [student_model_1, student_model_2]

student_optimizer_1 = optim.SGD(student_model_1.parameters(), 0.01)
student_optimizer_2 = optim.SGD(student_model_2.parameters(), 0.01)

student_optimizers = [student_optimizer_1, student_optimizer_2]

# Now, this is where KD_Lib comes into the picture 

distiller = DML(student_cohort, train_loader, test_loader, student_optimizers, log=True, logdir="./logs")

distiller.train_students(epochs=5)
distiller.evaluate()
distiller.get_parameters()
```

--------------------------------

### Implement Label Smooth Regularization with KD_Lib

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/LabelSmoothReg.rst

This snippet demonstrates how to use the LabelSmoothReg class from KD_Lib for knowledge distillation with label smoothing. It includes setting up datasets, models, optimizers, and the distillation process for both teacher and student models.

```python
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import LabelSmoothReg

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

# Set device to be trained on

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define student and teacher models

teacher_model = <your model>
student_model = <your model>

# Define optimizers

teacher_optimizer = optim.SGD(teacher_model.parameters(), lr=0.01)
student_optimizer = optim.SGD(student_model.parameters(), lr=0.01)

# Train using KD_Lib

distiller = LabelSmoothReg(teacher_model, student_model, train_loader, test_loader, teacher_optimizer, 
                               student_optimizer, correct_prob=0.9, device=device)  
distiller.train_teacher(epochs=5)                                       # Train the teacher model
distiller.train_students(epochs=5)                                      # Train the student model
distiller.evaluate(teacher=True)                                        # Evaluate the teacher model
distiller.evaluate()                                                    # Evaluate the student model

```

--------------------------------

### Commit and Push Changes

Source: https://github.com/sforaidl/kd_lib/blob/master/CONTRIBUTING.rst

Steps to stage, commit, and push local changes to a remote branch on GitHub.

```shell
git add .
git commit -m "Your detailed description of your changes."
git push origin name-of-your-bugfix-or-feature
```

--------------------------------

### Train Student Model with Virtual Teacher

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/VirtualTeacher.rst

This snippet shows how to initialize and use the VirtualTeacher class from KD_Lib to train a student model. It includes setting up the dataset (MNIST), dataloaders, student model, and optimizer. The VirtualTeacher is configured with a correct probability and then used to train the student model for a specified number of epochs, followed by evaluation.

```python
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import VirtualTeacher

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

# Set device to be trained on

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define student and teacher models

student_model = <your model>

# Define optimizer

student_optimizer = optim.SGD(student_model.parameters(), lr=0.01)


# Train using KD_Lib

distiller = VirtualTeacher(student_model, train_loader, test_loader, student_optimizer, 
                             correct_prob=0.9, device=device)  
distiller.train_student(epochs=5)                                       # Train the student model
distiller.evaluate()                                                    # Evaluate the student model

```

--------------------------------

### Train Student Model with Probability Shift on MNIST

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/ProbShift.rst

This Python code snippet demonstrates how to use the ProbShift class from KD_Lib to train a student model on the MNIST dataset. It includes setting up datasets, dataloaders, models, optimizers, and then utilizing the distiller object to train both teacher and student models, followed by evaluation.

```python
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import ProbShift

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

# Set device to be trained on

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define student and teacher models

teacher_model = <your model>
student_model = <your model>

# Define optimizers

teacher_optimizer = optim.SGD(teacher_model.parameters(), lr=0.01)
student_optimizer = optim.SGD(student_model.parameters(), lr=0.01)

# Train using KD_Lib

distiller = ProbShift(teacher_model, student_model, train_loader, test_loader, teacher_optimizer, 
                      student_optimizer, device=device)
    distiller.train_teacher(epochs=5)                                       # Train the teacher model
    distiller.train_students(epochs=5)                                      # Train the student model
    distiller.evaluate(teacher=True)                                        # Evaluate the teacher model
    distiller.evaluate()                                                    # Evaluate the student model
```

--------------------------------

### Create a New Branch for Development

Source: https://github.com/sforaidl/kd_lib/blob/master/CONTRIBUTING.rst

Command to create a new branch for local bug fixes or feature development.

```shell
git checkout -b name-of-your-bugfix-or-feature
```

--------------------------------

### Implement RCO with KD_Lib for MNIST

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/usage/tutorials/RCO.rst

This Python code demonstrates how to use the RCO class from KD_Lib to perform knowledge distillation on the MNIST dataset. It includes setting up datasets, dataloaders, student and teacher models, optimizers, and then training and evaluating both models using the RCO distiller with a specified epoch interval.

```python
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from KD_Lib.KD import RCO

# Define datasets, dataloaders, models and optimizers

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=True,
        download=True,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST(
        "mnist_data",
        train=False,
        transform=transforms.Compose(
            [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
        ),
    ),
    batch_size=32,
    shuffle=True,
)

# Set device to be trained on

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define student and teacher models

teacher_model = <your model>
student_model = <your model>

# Define optimizers

teacher_optimizer = optim.SGD(teacher_model.parameters(), lr=0.01)
student_optimizer = optim.SGD(student_model.parameters(), lr=0.01)

# Train using KD_Lib

distiller = RCO(teacher_model, student_model, train_loader, test_loader, teacher_optimizer, 
                    student_optimizer, epoch_interval=5, device=device)  
distiller.train_teacher(epochs=20)                                      # Train the teacher model
distiller.train_students(epochs=20)                                     # Train the student model
distiller.evaluate(teacher=True)                                        # Evaluate the teacher model
distiller.evaluate()                                                    # Evaluate the student model

```

--------------------------------

### KD_Lib Citation

Source: https://github.com/sforaidl/kd_lib/blob/master/docs/about.rst

This snippet provides the BibTeX citation for the KD_Lib project, which is useful for academic referencing. It includes details such as title, authors, year, and arXiv information.

```BibTeX
@misc{shah2020kdlib,
  title={KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization},
  author={Het Shah and Avishree Khare and Neelay Shah and Khizir Siddiqui},
  year={2020},
  eprint={2011.14691},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}
```

--------------------------------

### KD-Lib BibTeX Citation

Source: https://github.com/sforaidl/kd_lib/blob/master/README.md

This BibTeX entry provides citation details for the KD-Lib project, including authors, title, year, and arXiv information. It is useful for academic referencing when using the library.

```BibTeX
@misc{shah2020kdlib,
  title={KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization},
  author={Het Shah and Avishree Khare and Neelay Shah and Khizir Siddiqui},
  year={2020},
  eprint={2011.14691},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}
```

--------------------------------

### ResNet18 Static Quantization Results

Source: https://github.com/sforaidl/kd_lib/blob/master/logs.rst

This snippet presents the results for static quantization applied to a ResNet18 model on the CIFAR10 dataset. It compares the accuracy and size of the base model versus the quantized model, along with training epochs and time elapsed.

```Python
import pandas as pd

# Data representing static quantization results
static_quant_data = {
    'Algo Used': ['Static Quantization', 'Static Quantization'],
    'Model Used': ['ResNet18', 'ResNet18'],
    'Dataset': ['CIFAR10', 'CIFAR10'],
    'Model': ['Base', 'Quantized'],
    'Size (MB)': [46.8373, 11.7366],
    'Training epochs': [5, 5],
    'Accuracy': [0.7208, 0.7073],
    'Time elapsed': [26.1193, 12.9016]
}

df_static_quant = pd.DataFrame(static_quant_data)
print(df_static_quant.to_markdown(index=False))

```

--------------------------------

### ResNet18 Quantization Aware Training Results

Source: https://github.com/sforaidl/kd_lib/blob/master/logs.rst

This section shows the performance of quantization-aware training on a ResNet18 model using the CIFAR10 dataset. It contrasts the base model with the quantized model in terms of size, training epochs, accuracy, and time.

```Python
import pandas as pd

# Data representing quantization-aware training results
qat_data = {
    'Algo Used': ['Quantization Aware Training', 'Quantization Aware Training'],
    'Model Used': ['ResNet18', 'ResNet18'],
    'Dataset': ['CIFAR10', 'CIFAR10'],
    'Model': ['Base', 'Quantized'],
    'Size (MB)': [46.8373, 11.8408],
    'Training epochs': [5, 3],
    'Accuracy': [0.7208, 0.7128],
    'Time elapsed': [26.1193, 13.0377]
}

df_qat = pd.DataFrame(qat_data)
print(df_qat.to_markdown(index=False))

```

--------------------------------

### LSTM Dynamic Quantization Results

Source: https://github.com/sforaidl/kd_lib/blob/master/logs.rst

This snippet details the results of dynamic quantization applied to an LSTM model on the IMDB dataset. It compares the base LSTM model with its quantized version, focusing on size, training epochs, accuracy, and time elapsed.

```Python
import pandas as pd

# Data representing dynamic quantization results
dynamic_quant_data = {
    'Algo Used': ['Dynamic Quantization', 'Dynamic Quantization'],
    'Model Used': ['LSTM', 'LSTM'],
    'Dataset': ['IMDB', 'IMDB'],
    'Model': ['Base', 'Quantized'],
    'Size (MB)': [5.3829, 4.3757],
    'Training epochs': [3, 3],
    'Accuracy': [0.6964, 0.6953],
    'Time elapsed': [56.6991, 54.2108]
}

df_dynamic_quant = pd.DataFrame(dynamic_quant_data)
print(df_dynamic_quant.to_markdown(index=False))

```

--------------------------------

### ResNet18 Pruning Results

Source: https://github.com/sforaidl/kd_lib/blob/master/logs.rst

This section details the accuracy of a ResNet18 model on the MNIST dataset after applying pruning at different epochs. It shows the percentage of the model pruned and the number of training epochs used.

```Python
import pandas as pd

# Data representing the pruning results
pruning_data = {
    'Model Used': ['ResNet18', 'ResNet18', 'ResNet18'],
    'Dataset': ['MNIST', 'MNIST', 'MNIST'],
    'Pruning epoch': ['1/3', '2/3', '3/3'],
    '% Model pruned': [0.0, 10.0, 18.99],
    'Training epochs': [5, 5, 5],
    'Accuracy': ['98.78 %', '98.91 %', '98.90 %']
}

df_pruning = pd.DataFrame(pruning_data)
print(df_pruning.to_markdown(index=False))

```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.