TorchDistill

https://github.com/yoshitomo-matsubara/torchdistill
Admin
A modular, configuration-driven framework for knowledge distillation and general deep learning...
Tokens:41,363
Snippets:185
Trust Score:9.1
License:MIT
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Hugging Face Transformers Dependencies

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/hf_transformers/README.md

Installs the necessary Hugging Face libraries (`transformers`, `datasets`, `accelerate`) using pip or pipenv from a requirements file.

```bash
pip3 install -r examples/hf_transformers/requirements.txt
```

```bash
# If you use pipenv, use the following instead
pipenv install -r examples/hf_transformers/requirements.txt
```

--------------------------------

### Install Dependencies and TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Installs the required dependencies for Hugging Face Transformers examples and the torchdistill library itself using pip.

```bash
!pip install -r torchdistill/examples/hf_transformers/requirements.txt
!pip install torchdistill
```

--------------------------------

### Install TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_training.ipynb

Installs the TorchDistill library using pip. This command fetches and installs the package and its dependencies.

```Python
!pip install torchdistill
```

--------------------------------

### Install sc2bench Package

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/docs/source/projects.rst

This command installs the sc2bench framework, which is built on PyTorch for benchmarking SC2 methods. It is available as a PyPI package.

```bash
pip3 install sc2bench
```

--------------------------------

### Install TorchDistill from source using pipenv (Python)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This sequence of commands clones the TorchDistill repository, navigates into the directory, and then installs the library in editable mode using pipenv. This method also manages dependencies and virtual environments for the source installation.

```Shell
git clone https://github.com/yoshitomo-matsubara/torchdistill.git
cd torchdistill/
pipenv install "-e ."
```

--------------------------------

### Install TorchDistill from source using pip (Python)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This sequence of commands clones the TorchDistill repository, navigates into the directory, and then installs the library in editable mode using pip. This is useful for development or when you need the latest changes directly from the source.

```Shell
git clone https://github.com/yoshitomo-matsubara/torchdistill.git
cd torchdistill/
pip3 install -e .
```

--------------------------------

### Install TorchDistill using pip (Python)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This command installs the TorchDistill library using pip, the standard package installer for Python. Ensure you have Python and pip installed on your system.

```Shell
pip3 install torchdistill
```

--------------------------------

### Install TorchDistill using pip

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/docs/source/usage.rst

This snippet shows the command to install the torchdistill library using pip. It's a straightforward installation process.

```console
$ pip install torchdistill
```

--------------------------------

### Install TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_kd.ipynb

Installs the torchdistill library using pip. This is a prerequisite for using the library's functionalities.

```bash
!pip install torchdistill
```

--------------------------------

### Clone TorchDistill Repository

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Clones the torchdistill repository from GitHub to access example code and configuration files necessary for fine-tuning.

```bash
!git clone https://github.com/yoshitomo-matsubara/torchdistill
```

--------------------------------

### Install Dependencies and TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_kd_and_submission.ipynb

Installs the required Python dependencies for the Hugging Face Transformers examples and the TorchDistill library itself. This ensures all necessary packages are available for running the distillation scripts.

```bash
!pip install -r torchdistill/examples/hf_transformers/requirements.txt
!pip install torchdistill
```

--------------------------------

### Run Semantic Segmentation Experiment (Single GPU)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

This command runs a semantic segmentation experiment on a single GPU. It specifies the Python script, a configuration file, and an output log file. This is suitable for training on a single machine with one or more GPUs but without distributed setup.

```shell
python3 examples/torchvision/semantic_segmentation.py --config configs/sample/coco2017/ktaad/lraspp_mobilenet_v3_large_from_deeplabv3_resnet50.yaml --log log/coco2017/ktaad/lraspp_mobilenet_v3_large_from_deeplabv3_resnet50.txt
```

```shell
python3 examples/torchvision/semantic_segmentation.py --config configs/sample/pascal_voc2012/ce/deeplabv3_resnet50.yaml --log log/pascal_voc2012/ce/deeplabv3_resnet50.txt
```

--------------------------------

### Clone TorchDistill Repository

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_training.ipynb

Clones the official TorchDistill GitHub repository to access example code and configuration files. This is necessary for running the provided training scripts.

```Python
!git clone https://github.com/yoshitomo-matsubara/torchdistill.git
```

--------------------------------

### Run WNLI Task with TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_kd_and_submission.ipynb

Launches the general language understanding example script for the WNLI task using `accelerate launch`. It specifies the configuration file, task type, run log path, and private output directory.

```python
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/wnli/kd/bert_base_uncased_from_bert_large_uncased.yaml \
  --task wnli \
  --run_log log/glue/wnli/kd/bert_base_uncased_from_bert_large_uncased.txt \
  --private_output leaderboard/glue/kd/bert_base_uncased_from_bert_large_uncased/
```

--------------------------------

### Configure Hugging Face Accelerate

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This snippet shows the interactive prompts for configuring Hugging Face Accelerate, a tool for distributed training. It covers environment setup, machine type, process count, and mixed precision settings.

```shell
accelerate config
# In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): 0
# Which type of machine are you using? ([0] No distributed training, [1] multi-GPU, [2] TPU): 0
# How many processes in total will you use? [1]: 1
# Do you wish to use FP16 (mixed precision)? [yes/NO]: yes
```

--------------------------------

### Download ImageNet Datasets

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Downloads the ImageNet training and validation datasets using wget. Requires manual URL acquisition from the ImageNet website.

```shell
wget ${TRAIN_DATASET_URL} ./
wget ${VAL_DATASET_URL} ./
```

--------------------------------

### Object Detection Examples in TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This snippet links to example Python scripts for object detection tasks within the TorchDistill project. These examples showcase the application of knowledge distillation for improving object detection models.

```Python
# Placeholder for object detection distillation logic
# This would involve models that output bounding boxes and class probabilities

# Example structure for a distillation loss in object detection
def object_detection_distillation_loss(student_outputs, teacher_outputs, **kwargs):
    # student_outputs and teacher_outputs would contain bounding box predictions, class scores, etc.
    # Calculate losses for classification and regression components, potentially with distillation terms
    pass

# Example usage within a training loop:
# student_detector = ...
# teacher_detector = ...
# ...
# for images, targets in dataloader:
#     student_preds = student_detector(images)
#     with torch.no_grad():
#         teacher_preds = teacher_detector(images)
#     
#     loss = calculate_total_loss(student_preds, targets) + 
#            distillation_weight * object_detection_distillation_loss(student_preds, teacher_preds)
#     ...

```

--------------------------------

### Clone TorchDistill Repository

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_kd.ipynb

Clones the official torchdistill repository from GitHub. This allows access to example code and configuration files necessary for running distillation experiments.

```bash
!git clone https://github.com/yoshitomo-matsubara/torchdistill.git
```

--------------------------------

### Download COCO 2017 Datasets

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Downloads the COCO 2017 training images, validation images, and annotations using wget.

```shell
wget http://images.cocodataset.org/zips/train2017.zip ./
wget http://images.cocodataset.org/zips/val2017.zip ./
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip ./
```

--------------------------------

### Configure Accelerate for Multi-GPU/TPU/fp16

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/hf_transformers/README.md

Configures the 'accelerate' library for distributed training across multiple GPUs, TPUs, or with mixed-precision (fp16).

```bash
accelerate config
```

```bash
# If you use pipenv, use the following instead
pipenv run accelerate config
```

--------------------------------

### Install TorchDistill using pipenv (Python)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This command installs the TorchDistill library using pipenv, a tool that aims to bring the best of all packaging worlds to Python. It manages dependencies and virtual environments.

```Shell
pipenv install torchdistill
```

--------------------------------

### Image Classification Examples in TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This snippet provides links to example Python scripts for image classification tasks using TorchDistill. These examples likely demonstrate how to implement and utilize various knowledge distillation techniques for image classification models.

```Python
import torch
import torch.nn as nn
import torch.optim as optim

# Example placeholder for a distillation process
class Distiller(nn.Module):
    def __init__(self, student, teacher):
        super(Distiller, self).__init__()
        self.student = student
        self.teacher = teacher

    def forward(self, inputs):
        student_outputs = self.student(inputs)
        with torch.no_grad():
            teacher_outputs = self.teacher(inputs)
        
        # Add distillation loss calculation here
        # For example, using KL divergence or MSE between student and teacher outputs
        loss = nn.KLDivLoss()(torch.log_softmax(student_outputs, dim=1), torch.softmax(teacher_outputs, dim=1))
        return loss

# Placeholder for model definitions and training loop
# student_model = ...
# teacher_model = ...
# distiller = Distiller(student_model, teacher_model)
# optimizer = optim.Adam(student_model.parameters(), lr=0.001)
# ... training loop ...
```

--------------------------------

### Download PASCAL VOC 2012 Dataset

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

This command downloads the PASCAL VOC 2012 dataset archive using wget. The dataset is essential for semantic segmentation tasks and is typically downloaded to a specified resource directory.

```shell
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
```

--------------------------------

### Prepare PASCAL VOC 2012 Dataset

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

This script prepares the PASCAL VOC 2012 dataset by creating a resource directory, moving the downloaded tar file into it, extracting the contents, and then returning to the parent directory. This process is crucial for making the dataset available for training.

```shell
# Go to the root of this repository
mkdir ./resource/dataset/ -p
mv VOCtrainval_11-May-2012.tar ./resource/dataset/
cd ./resource/dataset/
tar -xvf VOCtrainval_11-May-2012.tar
cd ../../
```

--------------------------------

### Fine-tune BERT-Base for Text Classification (GoEmotions)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/hf_transformers/README.md

Fine-tunes a BERT-Base (uncased) model for multi-label text classification on the GoEmotions dataset using 'accelerate launch'.

```bash
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/sample/go_emotions/ce/bert_base_uncased.yaml \
  --run_log log/go_emotions/ce/bert_base_uncased.txt \
  --seed 123 \
  -disable_cudnn_benchmark

# use `pipenv run accelerate launch ...` if you're using pipenv
```

--------------------------------

### Generate GLUE Leaderboard Submissions

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/hf_transformers/README.md

Generates prediction files for GLUE leaderboard submission after fine-tuning a BERT-Base model, specifying private output directory.

```bash
export TASK_NAME=cola
export TEST_DIR=submission/standard/bert_base_uncased/

accelerate launch examples/hf_transformers/general_language_understanding.py \
  --config configs/sample/glue/${TASK_NAME}/ce/bert_base_uncased.yaml \
  --task ${TASK_NAME} \
  --private_output ${TEST_DIR} \
  -disable_cudnn_benchmark

# use `pipenv run accelerate launch ...` if you're using pipenv

# Note that you should re-configure `accelerate` by `accelerate config` and avoid the distributed processing mode
# when writing out the prediction file. If you have a fine-tuned model, you can skip training by adding `-test_only`.
```

--------------------------------

### Run Semantic Segmentation Experiment (CPU)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

This command executes a semantic segmentation experiment using the CPU. It specifies the Python script, the device as 'cpu', a configuration file, and an output log file. This is useful for testing or when GPU resources are unavailable.

```shell
python3 examples/torchvision/semantic_segmentation.py --device cpu --config configs/sample/coco2017/ktaad/lraspp_mobilenet_v3_large_from_deeplabv3_resnet50.yaml --log log/coco2017/ktaad/lraspp_mobilenet_v3_large_from_deeplabv3_resnet50.txt
```

```shell
python3 examples/torchvision/semantic_segmentation.py --device cpu --config configs/sample/pascal_voc2012/ce/deeplabv3_resnet50.yaml --log log/pascal_voc2012/ce/deeplabv3_resnet50.txt
```

--------------------------------

### Clone TorchDistill Repository

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_kd_and_submission.ipynb

Clones the official TorchDistill repository from GitHub. This action is necessary to access example code and configuration files for running distillation experiments.

```bash
!git clone https://github.com/yoshitomo-matsubara/torchdistill
```

--------------------------------

### Fine-tune BERT-Base for GLUE Tasks

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/hf_transformers/README.md

Fine-tunes a BERT-Base (uncased) model on a specified GLUE task (e.g., CoLA) using 'accelerate launch'.

```bash
export TASK_NAME=cola

accelerate launch examples/hf_transformers/general_language_understanding.py \
  --config configs/sample/glue/${TASK_NAME}/ce/bert_base_uncased.yaml \
  --task ${TASK_NAME} \
  -disable_cudnn_benchmark

# use `pipenv run accelerate launch ...` if you're using pipenv
```

--------------------------------

### Prepare ImageNet Datasets

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Extracts and organizes the downloaded ImageNet dataset files. This involves untarring archives, moving files to specific directories, and running a preparation script for validation data.

```shell
# Go to the root of this repository
mkdir ./resource/dataset/ilsvrc2012/{train,val} -p
mv ILSVRC2012_img_train.tar ./resource/dataset/ilsvrc2012/train/
mv ILSVRC2012_img_val.tar ./resource/dataset/ilsvrc2012/val/
cd ./resource/dataset/ilsvrc2012/train/
tar -xvf ILSVRC2012_img_train.tar
mv ILSVRC2012_img_train.tar ../
for f in *.tar; do
  d=`basename $f .tar`
  mkdir $d
  (cd $d && tar xf ../$f)
done
rm -r *.tar
cd ../../../../

wget https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh
mv valprep.sh ./resource/dataset/ilsvrc2012/val/
cd ./resource/dataset/ilsvrc2012/val/
tar -xvf ILSVRC2012_img_val.tar
mv ILSVRC2012_img_val.tar ../
sh valprep.sh
mv valprep.sh ../
cd ../../../../
```

--------------------------------

### Run Semantic Segmentation Experiment (Multi-GPU Distributed)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

This command executes a semantic segmentation experiment using multiple GPUs for distributed training. It requires specifying the number of GPUs, the Python script, and a configuration file. The output logs are directed to a specified text file.

```shell
torchrun --nproc_per_node=${NUM_GPUS} examples/torchvision/semantic_segmentation.py --world_size ${NUM_GPUS} --config configs/sample/coco2017/ktaad/lraspp_mobilenet_v3_large_from_deeplabv3_resnet50.yaml --log log/coco2017/ktaad/lraspp_mobilenet_v3_large_from_deeplabv3_resnet50.txt
```

```shell
torchrun --nproc_per_node=${NUM_GPUS} examples/torchvision/semantic_segmentation.py --world_size ${NUM_GPUS} --config configs/sample/pascal_voc2012/single_stage/ce/deeplabv3_resnet50.yaml --log log/pascal_voc2012/single_stage/ce/deeplabv3_resnet50.txt
```

--------------------------------

### Semantic Segmentation Examples in TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This snippet points to example Python scripts for semantic segmentation tasks in TorchDistill. These examples demonstrate how knowledge distillation can be applied to enhance semantic segmentation models.

```Python
# Placeholder for semantic segmentation distillation logic
# This would involve models outputting pixel-wise class predictions

# Example structure for a distillation loss in semantic segmentation
def semantic_segmentation_distillation_loss(student_outputs, teacher_outputs, **kwargs):
    # student_outputs and teacher_outputs would be segmentation masks (e.g., probability maps per class)
    # Calculate segmentation loss (e.g., CrossEntropyLoss) and add distillation loss (e.g., KL divergence on probability maps)
    pass

# Example usage within a training loop:
# student_segmenter = ...
# teacher_segmenter = ...
# ...
# for images, masks in dataloader:
#     student_preds = student_segmenter(images)
#     with torch.no_grad():
#         teacher_preds = teacher_segmenter(images)
#     
#     loss = calculate_segmentation_loss(student_preds, masks) + 
#            distillation_weight * semantic_segmentation_distillation_loss(student_preds, teacher_preds)
#     ...

```

--------------------------------

### Fine-tune BERT for QQP Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Launches the fine-tuning process for the BERT-Base (uncased) model on the QQP dataset using Accelerate and a specified configuration file. It logs the run and outputs results to a private directory.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/qqp/ce/bert_base_uncased.yaml \
  --task qqp \
  --run_log log/glue/qqp/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Configure Hugging Face Accelerate

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This section shows the interactive prompts for configuring Hugging Face Accelerate for distributed training. It covers environment, machine type, process count, and mixed precision settings.

```shell
accelerate config
# In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): 0
# Which type of machine are you using? ([0] No distributed training, [1] multi-GPU, [2] TPU): 0
# How many processes in total will you use? [1]: 1
# Do you wish to use FP16 (mixed precision)? [yes/NO]: yes
```

--------------------------------

### Fine-tune BERT-Base for QQP Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Base model for the QQP task using Hugging Face Accelerate. It specifies the configuration file, task type, log file, and private output directory.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/qqp/ce/bert_base_uncased.yaml \
  --task qqp \
  --run_log log/glue/qqp/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Text Classification Examples with Hugging Face Transformers

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/README.md

This snippet links to an example Python script for text classification using Hugging Face Transformers within the TorchDistill project. It demonstrates knowledge distillation for NLP tasks.

```Python
# Placeholder for text classification distillation logic using Hugging Face Transformers
# This would involve models like BERT, RoBERTa, etc.

# Example structure for a distillation loss in text classification
def text_classification_distillation_loss(student_logits, teacher_logits, **kwargs):
    # student_logits and teacher_logits are the raw output scores from the models
    # Calculate cross-entropy loss for the ground truth labels and add distillation loss (e.g., KL divergence on softmax outputs)
    pass

# Example usage within a training loop:
# from transformers import AutoModelForSequenceClassification, AutoTokenizer
# student_model = AutoModelForSequenceClassification.from_pretrained(...) 
# teacher_model = AutoModelForSequenceClassification.from_pretrained(...) 
# tokenizer = AutoTokenizer.from_pretrained(...)
# ...
# for batch in dataloader:
#     inputs = tokenizer(batch['text'], return_tensors='pt', padding=True, truncation=True)
#     student_outputs = student_model(**inputs)
#     with torch.no_grad():
#         teacher_outputs = teacher_model(**inputs)
#     
#     loss = calculate_classification_loss(student_outputs.logits, batch['labels']) + 
#            distillation_weight * text_classification_distillation_loss(student_outputs.logits, teacher_outputs.logits)
#     ...

```

--------------------------------

### Distill Knowledge for QQP Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_kd_and_submission.ipynb

Launches the knowledge distillation process for the QQP (Quora Question Pairs) task. The command specifies the configuration, task, and output paths for the distillation run.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/qqp/kd/bert_base_uncased_from_bert_large_uncased.yaml \
  --task qqp \
  --run_log log/glue/qqp/kd/bert_base_uncased_from_bert_large_uncased.txt \
  --private_output leaderboard/glue/kd/bert_base_uncased_from_bert_large_uncased/
```

--------------------------------

### Fine-tune BERT for QNLI Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Launches the fine-tuning process for the BERT-Base (uncased) model on the QNLI dataset using Accelerate and a specified configuration file. It logs the run and outputs results to a private directory.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/qnli/ce/bert_base_uncased.yaml \
  --task qnli \
  --run_log log/glue/qnli/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Unzip COCO 2017 Datasets

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Extracts the downloaded COCO 2017 dataset zip files into the specified directory structure.

```shell
# Go to the root of this repository
mkdir ./resource/dataset/coco2017/ -p
mv train2017.zip ./resource/dataset/coco2017/
mv val2017.zip ./resource/dataset/coco2017/
mv annotations_trainval2017.zip ./resource/dataset/coco2017/
cd ./resource/dataset/coco2017/
unzip train2017.zip
unzip val2017.zip
unzip annotations_trainval2017.zip
cd ../../../
```

--------------------------------

### Fine-tune BERT-Large for QQP Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Large model for the QQP task using Hugging Face Accelerate. It specifies the configuration file, task type, log file, and private output directory.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/qqp/ce/bert_large_uncased.yaml \
  --task qqp \
  --run_log log/glue/qqp/ce/bert_large_uncased.txt \
  --private_output leaderboard/glue/standard/bert_large_uncased/
```

--------------------------------

### Fine-tune BERT-Base for QNLI Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Base model for the QNLI task using Hugging Face Accelerate. It specifies the configuration file, task type, log file, and private output directory.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/qnli/ce/bert_base_uncased.yaml \
  --task qnli \
  --run_log log/glue/qnli/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Configure Hugging Face Accelerate

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This interactive shell command configures Hugging Face Accelerate for distributed training. It prompts the user for environment details, machine type, number of processes, and precision settings.

```shell
accelerate config
# In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): 0
# Which type of machine are you using? ([0] No distributed training, [1] multi-GPU, [2] TPU): 0
# How many processes in total will you use? [1]: 1
# Do you wish to use FP16 (mixed precision)? [yes/NO]: yes
```

--------------------------------

### Run ImageNet Classification (Single GPU/CPU)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Runs an ImageNet image classification experiment on a single GPU or CPU. Configuration and logging paths are specified.

```shell
python3 examples/torchvision/image_classification.py --config configs/sample/ilsvrc2012/single_stage/kd/alexnet_from_resnet152.yaml --log log/ilsvrc2012/kd/alexnet_from_resnet152.txt
python3 examples/torchvision/image_classification.py --device cpu --config configs/sample/ilsvrc2012/single_stage/kd/alexnet_from_resnet152.yaml --log log/ilsvrc2012/kd/alexnet_from_resnet152.txt
```

--------------------------------

### Fine-tune BERT for RTE Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Launches the fine-tuning process for the BERT-Base (uncased) model on the RTE dataset using Accelerate and a specified configuration file. It logs the run and outputs results to a private directory.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/rte/ce/bert_base_uncased.yaml \
  --task rte \
  --run_log log/glue/rte/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Setup Forward Hook Manager

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/extract_intermediate_representations.ipynb

Initializes a ForwardHookManager to capture intermediate representations. Hooks are added to specific layers ('conv1', 'layer1.0.bn2', 'fc') to record their inputs and/or outputs.

```python
from torchdistill.core.forward_hook import ForwardHookManager

device = torch.device('cpu')
forward_hook_manager = ForwardHookManager(device)
forward_hook_manager.add_hook(model, 'conv1', requires_input=True, requires_output=False)
forward_hook_manager.add_hook(model, 'layer1.0.bn2', requires_input=True, requires_output=True)
forward_hook_manager.add_hook(model, 'fc', requires_input=False, requires_output=True)
```

--------------------------------

### Define and Register a Custom Module in TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/docs/source/usage.rst

This Python code demonstrates how to define a custom module (MyNewCoolModel) that inherits from nn.Module and registers it using the @register_model decorator. It also shows how to print initialization parameters.

```python
from torch import nn
from torchdistill.models.registry import register_model

@register_model
class MyNewCoolModel(nn.Module):
    def __init__(self, some_value, some_list, some_dict):
        super().__init__()
        print('some_value: ', some_value)
        print('some_list: ', some_list)
        print('some_dict: ', some_dict)
    ...
```

--------------------------------

### Run ImageNet Classification (Distributed)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Executes an ImageNet image classification experiment using multiple GPUs for distributed training. Requires specifying the number of GPUs and configuration file.

```shell
torchrun --nproc_per_node=${NUM_GPUS} examples/torchvision/image_classification.py --world_size ${NUM_GPUS} --config configs/sample/ilsvrc2012/single_stage/kd/alexnet_from_resnet152.yaml --log log/ilsvrc2012/kd/alexnet_from_resnet152.txt
```

--------------------------------

### Fine-tune BERT-Large for QNLI Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Large model for the QNLI task using Hugging Face Accelerate. It specifies the configuration file, the task, and logging/output paths.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/qnli/ce/bert_large_uncased.yaml \
  --task qnli \
  --run_log log/glue/qnli/ce/bert_large_uncased.txt \
  --private_output leaderboard/glue/standard/bert_large_uncased/
```

--------------------------------

### Fine-tune BERT for WNLI Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Launches the fine-tuning process for the BERT-Base (uncased) model on the WNLI dataset using Accelerate and a specified configuration file. It logs the run and outputs results to a private directory.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/wnli/ce/bert_base_uncased.yaml \
  --task wnli \
  --run_log log/glue/wnli/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Run COCO Object Detection (Single GPU/CPU)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Runs a COCO 2017 object detection experiment on a single GPU or CPU. Configuration and logging paths are specified, and cuDNN benchmark is disabled.

```shell
python3 examples/torchvision/object_detection.py --config configs/sample/coco2017/ft/custom_fasterrcnn_resnet18_fpn_from_fasterrcnn_resnet50_fpn.yaml --log log/coco2017/ft/custom_fasterrcnn_resnet18_fpn_from_fasterrcnn_resnet50_fpn.txt -disable_cudnn_benchmark
python3 examples/torchvision/object_detection.py --device cpu --config configs/sample/coco2017/ft/custom_fasterrcnn_resnet18_fpn_from_fasterrcnn_resnet50_fpn.yaml --log log/coco2017/ft/custom_fasterrcnn_resnet18_fpn_from_fasterrcnn_resnet50_fpn.txt -disable_cudnn_benchmark
```

--------------------------------

### Test Pretrained Models with TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/cifar100/yoshitomo-matsubara/nlp-oss2023/README.md

This command tests pretrained models using TorchDistill. It requires modifying the configuration to set 'pretrained: True' and uses the '-test_only' flag. The example shows how to test the WRN-40-4 model.

```Shell
python3 examples/torchvision/image_classification.py \
    --config configs/official/cifar100/yoshitomo-matsubara/nlp-oss2023/wide_resnet40_4-final_run.yaml \
    -test_only
```

--------------------------------

### Configure YAML for Custom Module in TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/docs/source/usage.rst

This YAML snippet illustrates how to configure a script to use a registered custom module ('MyNewCoolModel') and pass specific parameters ('some_value', 'some_list', 'some_dict') during its initialization. It also includes a dependency on a custom module file.

```yaml
dependencies:
  - name: 'my_module'
...
models:
  model:
    key: 'MyNewCoolModel'
    kwargs:
      some_value: 777
      some_list: ['this', 'is', 'some_list']
      some_dict:
        some_key: 'some_value'
        test: 0.123
    src_ckpt:
...
```

--------------------------------

### Project Citation (BibTeX)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/ilsvrc2012/yoshitomo-matsubara/nlp-oss2023/README.md

This BibTeX entry provides the citation details for the project 'torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP'.

```bibtex
@inproceedings{matsubara2023torchdistill,
  title={{torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP}},
  author={Matsubara, Yoshitomo},
  booktitle={Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)},
  publisher={Empirical Methods in Natural Language Processing},
  pages={153--164},
  year={2023}
}
```

--------------------------------

### Run COCO Object Detection (Distributed)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/examples/torchvision/README.md

Executes a COCO 2017 object detection experiment using multiple GPUs for distributed training. Specifies configuration for Faster R-CNN models and disables cuDNN benchmark.

```shell
torchrun --nproc_per_node=${NUM_GPUS} examples/torchvision/object_detection.py --world_size ${NUM_GPUS} --config configs/sample/coco2017/ft/custom_fasterrcnn_resnet18_fpn_from_fasterrcnn_resnet50_fpn.yaml --log log/coco2017/ft/custom_fasterrcnn_resnet18_fpn_from_fasterrcnn_resnet50_fpn.txt -disable_cudnn_benchmark
```

--------------------------------

### Fine-tune BERT for SST-2 Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Launches the fine-tuning process for the BERT-Base (uncased) model on the SST-2 dataset using Accelerate and a specified configuration file. It logs the run and outputs results to a private directory.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/sst2/ce/bert_base_uncased.yaml \
  --task sst2 \
  --run_log log/glue/sst2/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Test Pretrained Models with TorchDistill

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/cifar10/yoshitomo-matsubara/nlp-oss2023/README.md

This command tests pretrained models using TorchDistill. It requires modifying the configuration file to set 'pretrained: True' and running the script with the '-test_only' flag. The example uses a ResNet-20 model.

```Shell
python3 examples/torchvision/image_classification.py \
    --config configs/official/cifar10/yoshitomo-matsubara/nlp-oss2023/resnet20-final_run.yaml \
    -test_only
```

--------------------------------

### Fine-tune BERT for MNLI Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Launches the fine-tuning process for the BERT-Base (uncased) model on the MNLI dataset using Accelerate and a specified configuration file. It logs the run and outputs results to a private directory.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/mnli/ce/bert_base_uncased.yaml \
  --task mnli \
  --run_log log/glue/mnli/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Run Factor Transfer (Non-Distributed)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/ilsvrc2012/yoshitomo-matsubara/rrpr2020/README.md

This command executes the Factor Transfer method for knowledge distillation in a non-distributed setup. It uses 'python3' to run the image classification script with the provided configuration and log file. This command is intended for training models from scratch.

```Shell
python3 examples/torchvision/image_classification.py \
    --config configs/official/ilsvrc2012/yoshitomo-matsubara/rrpr2020/ft-resnet18_from_resnet34.yaml \
    --run_log log/ilsvrc2012/ft-resnet18_from_resnet34.log
```

--------------------------------

### Fine-tune BERT-Base for RTE Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Base model for the RTE task using Hugging Face Accelerate. It specifies the configuration file, task type, log file, and private output directory.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/rte/ce/bert_base_uncased.yaml \
  --task rte \
  --run_log log/glue/rte/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Run L2 (CE + L2) Distillation (Distributed)

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/ilsvrc2012/yoshitomo-matsubara/rrpr2020/README.md

This command executes the L2 (CE + L2) distillation method in a distributed setup using torchrun. It specifies the number of GPUs, the training script, configuration, and log file. The batch size in the configuration should be adjusted so that (batch size) * ${NUM_GPUS} = 512.

```Shell
torchrun  --nproc_per_node=${NUM_GPUS} examples/torchvision/image_classification.py \
    --config configs/official/ilsvrc2012/yoshitomo-matsubara/rrpr2020/ce_l2-resnet18_from_resnet34.yaml \
    --run_log log/ilsvrc2012/ce_l2-resnet18_from_resnet34.log \
    --world_size ${NUM_GPUS}
```

--------------------------------

### Train DeepLabv3 ResNet-101 from Scratch

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/pascal_voc2012/yoshitomo-matsubara/nlp-oss2023/README.md

This command trains a DeepLabv3 model with ResNet-101 from scratch on the PASCAL VOC 2012 dataset. Ensure no checkpoint exists at the specified destination to initiate training from the beginning. It logs the training process.

```Python
python3 examples/torchvision/semantic_segmentation.py \
    --config configs/official/pascal_voc2012/yoshitomo-matsubara/nlp-oss2023/deeplabv3_resnet101.yaml \
    --run_log log/deeplabv3_resnet101.log 
```

--------------------------------

### Fine-tune BERT-Base for WNLI Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Base model for the WNLI task using Hugging Face Accelerate. It specifies the configuration file, task type, log file, and private output directory.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/wnli/ce/bert_base_uncased.yaml \
  --task wnli \
  --run_log log/glue/wnli/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Fine-tune BERT for MRPC Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Launches the fine-tuning process for the BERT-Base (uncased) model on the MRPC dataset using Accelerate and a specified configuration file. It logs the run and outputs results to a private directory.

```bash
!accelerate launch torchdistill/examples/hf_transformers/general_language_understanding.py \
  --config torchdistill/configs/sample/glue/mrpc/ce/bert_base_uncased.yaml \
  --task mrpc \
  --run_log log/glue/mrpc/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```

--------------------------------

### Configure Accelerate for Mixed-Precision Training

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb

Configures the Accelerate library for mixed-precision training, which can significantly speed up training times, especially on limited hardware.

```bash
!accelerate config
```

--------------------------------

### Fine-tune BERT-Large for RTE Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Large model for the RTE task using Hugging Face Accelerate. It specifies the configuration file, the task, and logging/output paths.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/rte/ce/bert_large_uncased.yaml \
  --task rte \
  --run_log log/glue/rte/ce/bert_large_uncased.txt \
  --private_output leaderboard/glue/standard/bert_large_uncased/
```

--------------------------------

### Distill BERT-Large to BERT-Base for QQP Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command distills knowledge from BERT-Large to BERT-Base for the QQP task. It specifies the configuration file, task, and output directories for the distillation process.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/qqp/kd/bert_base_uncased_from_bert_large_uncased.yaml \
  --task qqp \
  --run_log log/glue/qqp/kd/bert_base_uncased_from_bert_large_uncased.txt \
  --private_output leaderboard/glue/kd/bert_base_uncased_from_bert_large_uncased/
```

--------------------------------

### Fine-tune BERT-Base for MNLI Task

Source: https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/official/glue/yoshitomo-matsubara/nlp-oss2023/README.md

This command fine-tunes a BERT-Base model for the MNLI task using Hugging Face Accelerate. It specifies the configuration file, task type, log file, and private output directory.

```shell
accelerate launch examples/hf_transformers/text_classification.py \
  --config configs/official/glue/yoshitomo-matsubara/nlp-oss2023/mnli/ce/bert_base_uncased.yaml \
  --task mnli \
  --run_log log/glue/mnli/ce/bert_base_uncased.txt \
  --private_output leaderboard/glue/standard/bert_base_uncased/
```
Tokens:
Raw