### Distillation Configuration Example

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Experiments.md

Example of setting up a distillation configuration with a specified temperature and intermediate layer matches. Other arguments default to standard values.

```python
distill_config = DistillationConfig(temperature = 8, intermediate_matches = matches)
# Others arguments take the default values
```

--------------------------------

### Distillation Configuration Example

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/_build/html/Quickstart copy.html

An example of how to set up a distillation configuration using DistillationConfig. Other arguments will take default values.

```python
distill_config = DistillationConfig(temperature = 8, intermediate_matches = matches)
```

--------------------------------

### Install TextBrewer from Github Source

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Tutorial.rst

Install TextBrewer by cloning the Github repository and then installing it locally. This is useful for developers or when needing the latest unreleased features.

```shell
git clone https://github.com/airaria/TextBrewer.git
pip install ./textbrewer
```

--------------------------------

### Load SQuAD v1.1 training examples

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

Initializes the SquadV1Processor and loads the training examples from the specified directory.

```python
processor = SquadV1Processor()
examples = processor.get_train_examples('/content/')
```

--------------------------------

### Install TextBrewer from PyPI

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Tutorial.rst

Install the TextBrewer library using pip. This is the recommended method for most users.

```shell
pip install textbrewer
```

--------------------------------

### Install Required Libraries

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/msra_ner.ipynb

Installs necessary Python packages: datasets, transformers, seqeval, and textbrewer.

```bash
!pip install datasets
!pip install transformers
!pip install seqeval
!pip install textbrewer
```

--------------------------------

### Install Required Libraries

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sst2.ipynb

Installs the transformers, datasets, and textbrewer libraries. These are essential for model training and distillation.

```bash
!pip install transformers #4.8.2
!pip install datasets
!pip install textbrewer
```

--------------------------------

### Instantiate TSDataset with Fake Data and DataLoader

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Concepts.rst

This example demonstrates how to create dummy teacher and student datasets, generate fake teacher cache data, and then instantiate the TSDataset and a DataLoader. This setup is useful for testing or demonstrating the teacher cache functionality.

```python
teacher_dataset = TensorDataset(torch.randn(32,3),torch.randn(32,3))
student_dataset = TensorDataset(torch.randn(32,2),torch.randn(32,2))

# We make some fake data and assume teacher model outputs are (logits, loss)
fake_logits = [torch.randn(3) for _ in range(32)]
fake_loss = [torch.randn(1)[0] for _ in range(32)]
teacher_cache = [(fake_logits[i],fake_loss[i]) for i in range(32)]

tsdataset = TSDataset(teacher_dataset=teacher_dataset,student_dataset=student_dataset, teacher_cache=teacher_cache)
dataloader = DataLoader(dataset=tsdataset, ... )
```

--------------------------------

### Configure and Start Training

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sst2.ipynb

Sets up training arguments and initializes a Trainer for the BERT model.

```python
#start training 
training_args = TrainingArguments(
    output_dir='/content/drive/MyDrive/results',          #output directory
    learning_rate=1e-4,
    num_train_epochs=5,
    per_device_train_batch_size=32,                #batch size per device during training
    per_device_eval_batch_size=32,                #batch size for evaluation
    logging_dir='/content/drive/MyDrive/logs',
    logging_steps=100,
    do_train=True,
    do_eval=True,
    no_cuda=False,
    load_best_model_at_end=True,
    # eval_steps=100,
    evaluation_strategy="epoch"
)

trainer = Trainer(
    model=model,                         
    args=training_args,                  
    train_dataset=train_dataset,         
    eval_dataset=val_dataset,            
    compute_metrics=compute_metrics
)

train_out = trainer.train()

#after training, you could find traing logs and checpoints in your own dirve. also you can reset the file address in training args
```

--------------------------------

### Configure Distillation and Train Model

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sst2.ipynb

Sets up distillation configuration, defines an adaptor, initializes a GeneralDistiller, and starts the distillation training process.

```python
scheduler_args = {'num_warmup_steps':int(0.1*num_training_steps), 'num_training_steps':num_training_steps}

def simple_adaptor(batch, model_outputs):
    return {'logits': model_outputs.logits, 'hidden': model_outputs.hidden_states}

distill_config = DistillationConfig(
    intermediate_matches=[
     {'layer_T':0, 'layer_S':0, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1},
     {'layer_T':8, 'layer_S':2, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1}])
train_config = TrainingConfig()

distiller = GeneralDistiller(
    train_config=train_config, distill_config=distill_config,
    model_T=teacher_model, model_S=student_model, 
    adaptor_T=simple_adaptor, adaptor_S=simple_adaptor)


with distiller:
    distiller.train(optimizer, train_dataloader, num_epochs, scheduler_class=scheduler_class, scheduler_args = scheduler_args, callback=None)
```

--------------------------------

### SimpleAdaptor Example

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Concepts.rst

An example of a SimpleAdaptor function that maps model outputs to the expected format for TextBrewer. It specifies how to return logits, hidden states, and input masks.

```python
def SimpleAdaptor(batch, model_outputs):
  return {'logits': (model_outputs[0],),
          'hidden': model.outputs[1],
          'inputs_mask': batch[1]}
```

--------------------------------

### Quickstart: Initialize Distiller

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/_build/html/GettingStarted.html

Initialize a GeneralDistiller for knowledge distillation. This requires a trained teacher model, an untrained student model, and necessary training components like dataloader, optimizer, and scheduler.

```python
import textbrewer
from textbrewer import GeneralDistiller
```

--------------------------------

### Quickstart: Distilling BERT with TextBrewer

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Tutorial.rst

Demonstrates the core TextBrewer workflow for knowledge distillation. It includes displaying model parameters, defining an adaptor, setting up training and distillation configurations, and initiating the training process.

```python
import textbrewer
from textbrewer import GeneralDistiller
from textbrewer import TrainingConfig, DistillationConfig

# Show the statistics of model parameters
print("\nteacher_model's parametrers:")
result, _ = textbrewer.utils.display_parameters(teacher_model,max_level=3)
print (result)

print("student_model's parametrers:")
result, _ = textbrewer.utils.display_parameters(student_model,max_level=3)
print (result)

# Define an adaptor for interpreting the model inputs and outputs
def simple_adaptor(batch, model_outputs):
    # The second and third elements of model outputs are the logits and hidden states
    return {'logits': model_outputs[1],
            'hidden': model_outputs[2]}

# Training configuration 
train_config = TrainingConfig()
# Distillation configuration
# Matching different layers of the student and the teacher
# We match 0-0 and 8-2 here for demonstration
distill_config = DistillationConfig(
    intermediate_matches=[
    {'layer_T':0, 'layer_S':0, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1},
    {'layer_T':8, 'layer_S':2, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1}])

# Build distiller
distiller = GeneralDistiller(
    train_config=train_config, distill_config = distill_config,
    model_T = teacher_model, model_S = student_model, 
    adaptor_T = simple_adaptor, adaptor_S = simple_adaptor)

# Start!
with distiller:
    distiller.train(optimizer, dataloader, num_epochs=1, scheduler_class=scheduler_class, scheduler_args=scheduler_args, callback=None)
```

--------------------------------

### Install Required Packages

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

Installs the 'transformers' and 'textbrewer' libraries using pip. These are necessary for model fine-tuning and distillation.

```bash
!pip install transformers
!pip install textbrewer
```

--------------------------------

### Initialize and Train GeneralDistiller

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

This snippet shows the setup and training process for a GeneralDistiller, including defining distillation configuration, training configuration, and the distillation training loop.

```python
scheduler_args = {'num_warmup_steps':int(0.1*num_training_steps), 'num_training_steps':num_training_steps}

def simple_adaptor(batch, model_outputs):
    return {'logits': (model_outputs.start_logits,model_outputs.end_logits), 
        'hidden': model_outputs.hidden_states, 
        'attention': model_outputs.attentions}

distill_config = DistillationConfig(
    temperature = 1,
    intermediate_matches=[{"layer_T":[0,0],  "layer_S":[0,0], "feature":"hidden", "loss":"mmd", "weight":1},
               {"layer_T":[4,4],  "layer_S":[1,1], "feature":"hidden", "loss":"mmd", "weight":1},
               {"layer_T":[8,8],  "layer_S":[2,2], "feature":"hidden", "loss":"mmd", "weight":1},
               {"layer_T":[12,12], "layer_S":[3,3], "feature":"hidden", "loss":"mmd", "weight":1}])
train_config = TrainingConfig()

distiller = GeneralDistiller(
    train_config=train_config, distill_config=distill_config,
    model_T=teacher_model, model_S=student_model, 
    adaptor_T=simple_adaptor, adaptor_S=simple_adaptor)

def batch_postprocessor(batch):
  return {"input_ids": batch[0],
       "attention_mask": batch[1],
       "token_type_ids": batch[2],
       "start_positions": batch[3],
       "end_positions": batch[4]}

with distiller:
    distiller.train(optimizer, train_dataloader, num_epochs, scheduler_class=scheduler_class, scheduler_args = scheduler_args, batch_postprocessor=batch_postprocessor, callback=None)
```

--------------------------------

### Configure Distillation

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/_build/html/GettingStarted.html

Sets up training and distillation configurations. This example shows matching different layers of the student and teacher models using intermediate matches.

```python
# Training configuration 
train_config = TrainingConfig()
# Distillation configuration
# Matching different layers of the student and the teacher
distill_config = DistillationConfig(
    intermediate_matches=[
     {'layer_T':0, 'layer_S':0, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1},
     {'layer_T':8, 'layer_S':2, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1}])
```

--------------------------------

### FSP Preset Configuration

Source: https://github.com/airaria/textbrewer/wiki/Presets

Example configuration for the Fast Optimization, Network Minimization and Transfer Learning (FSP) preset. Use this when student and teacher hidden sizes differ and projection is needed.

```python
intermediate_matches = [
    {'layer_T':[0,0], 'layer_S':[0,0], 'feature':'hidden','loss': 'fsp', 'weight' : 1, 'proj':['linear',384,768]},
    ...]
```

--------------------------------

### Build and Run General Distiller

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/_build/html/GettingStarted.html

Builds a GeneralDistiller instance and starts the distillation training process. Ensure optimizer, scheduler, and dataloader are properly defined.

```python
# Build distiller
distiller = GeneralDistiller(
    train_config=train_config, distill_config = distill_config, 
    model_T = teacher_model, model_S = student_model, 
    adaptor_T = simple_adaptor, adaptor_S = simple_adaptor)

# Start!
with distiller:
    distiller.train(optimizer, scheduler, dataloader, num_epochs=1, callback=None)
```

--------------------------------

### Start Model Training

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/msra_ner.ipynb

Initiates the training process for the token classification model using the configured Trainer.

```python
trainer.train()
```

--------------------------------

### Training Loop Example

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

Iterates through epochs and batches, performs model training, calculates loss, and logs progress. Includes gradient clipping and optimizer/scheduler steps.

```python
print("  Num examples = ", len(train_dataset))
print("  Total optimization steps = ", t_total)

steps = 1
tr_loss = 0.0
model.zero_grad()

for epoch in range(epochs):
    print('Epoch:{}'.format(epoch+1))
    epoch_iterator = tqdm.notebook.tqdm(train_dataloader, desc="Iteration", disable=False)
    for step, batch in enumerate(epoch_iterator):

        model.train()
        batch = tuple(t.to(device) for t in batch)

        inputs = {
            "input_ids": batch[0],
            "attention_mask": batch[1],
            "token_type_ids": batch[2],
            "start_positions": batch[3],
            "end_positions": batch[4],
        }

        outputs = model(**inputs)
        loss = outputs.loss

        loss.backward()

        tr_loss += loss.item()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        optimizer.step()
        scheduler.step()  
        model.zero_grad()
        steps += 1
        
        # Log 
        if steps % 500 == 0:
            print('steps = {}, logging_loss = {}'.format(steps,tr_loss))
```

--------------------------------

### GeneralDistiller.train

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Distillers.rst

Initializes a GeneralDistiller object and calls its train method to start training/distillation.

```APIDOC
## GeneralDistiller.train

### Description
Trains the student model using the GeneralDistiller.

### Method
`train`

### Parameters
This method is documented via the autoclass directive and its members are listed.
```

--------------------------------

### MultiTaskDistiller Training Batch Postprocessor Example

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Illustrates how batch postprocessors are applied within the MultiTaskDistiller's training loop for specific tasks.

```python
batch = next(dataloaders[taskname])
# if batch_postprocessors is not None:
batch = batch_postprocessors[taskname](batch)
# check batch datatype
# passes batch to the model and adaptors
```

--------------------------------

### BasicDistiller.train

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Distillers.rst

Initializes a BasicDistiller object and calls its train method to start training/distillation.

```APIDOC
## BasicDistiller.train

### Description
Trains the student model using the BasicDistiller.

### Method
`train`

### Parameters
This method is documented via the autoclass directive and its members are listed.
```

--------------------------------

### DistillationConfig Initialization Parameters

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Example of parameters used to initialize DistillationConfig, specifying layer mappings, feature types, loss functions, weights, and projection details.

```python
[
      {'layer_T':0, 'layer_S':0, 'feature':'hidden','loss': 'hidden_mse', 'weight' : 1,'proj':['linear',384,768]},
      {'layer_T':4, 'layer_S':1, 'feature':'hidden','loss': 'hidden_mse', 'weight' : 1,'proj':['linear',384,768]},
      {'layer_T':8, 'layer_S':2, 'feature':'hidden','loss': 'hidden_mse', 'weight' : 1,'proj':['linear',384,768]},
      {'layer_T':12, 'layer_S':3, 'feature':'hidden','loss': 'hidden_mse', 'weight' : 1,'proj':['linear',384,768]}
  ]
```

--------------------------------

### Configuring 'flsw' Temperature Scheduler

Source: https://github.com/airaria/textbrewer/wiki/Presets

Example of configuring the 'flsw' (Fast Learning, Slow Weight) temperature scheduler in DistillationConfig, requiring beta and gamma parameters.

```python
distill_config = DistillationConfig(
    temperature_scheduler = ['flsw', 1, 1]
    ...)
```

--------------------------------

### GeneralDistiller

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Initializes a GeneralDistiller for single-model, single-task distillation. The 'train' method is used to start the distillation process.

```APIDOC
## GeneralDistiller

### Description
Initializes a GeneralDistiller for single-model, single-task distillation. This class is recommended for standard distillation tasks.

### Class Signature
`textbrewer.GeneralDistiller(train_config, distill_config, model_T, model_S, adaptor_T, adaptor_S, custom_matches=None)`

### Parameters
* **train_config** (`TrainingConfig`): Configuration for the training process.
* **distill_config** (`DistillationConfig`): Configuration for the distillation process.
* **model_T** (`torch.nn.Module`): The teacher model.
* **model_S** (`torch.nn.Module`): The student model.
* **adaptor_T** (`Callable`): A function to adapt the teacher model's outputs. It takes `batch` and `model_outputs` and returns a dict.
* **adaptor_S** (`Callable`): A function to adapt the student model's outputs. It takes `batch` and `model_outputs` and returns a dict.
* **custom_matches** (`List`, optional): Supports more flexible self-defined matches for advanced use cases.

### Adaptor Function Signature
`adaptor(batch, model_outputs) -> Dict`

### Train Method Signature
`train(optimizer, scheduler, dataloader, num_epochs, num_steps=None, callback=None, batch_postprocessor=None, **args)`

### Parameters for train method
* **optimizer**: The optimizer to use for training.
* **scheduler**: Learning rate scheduler (optional).
* **dataloader**: Dataset iterator.
* **num_epochs** (`int`): Number of training epochs.
* **num_steps** (`int`, optional): Number of training steps. If provided, `num_epochs` is ignored. The dataloader will cycle automatically.
* **callback** (`Callable`, optional): A function called after each epoch, e.g., for evaluation. Signature: `callback(model=self.model_S, step=global_step)`.
* **batch_postprocessor** (`Callable`, optional): A function to post-process batches before they are passed to the model. Signature: `batch_postprocessor(batch) -> batch`.
* **`**args`**: Additional arguments to be fed to the model.
```

--------------------------------

### Training and Distillation Configuration

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/_build/html/Quickstart copy.html

Configure training parameters and distillation matching strategies for TextBrewer. This setup is used to build a GeneralDistiller for knowledge distillation.

```python
train_config = TrainingConfig()
distill_config = DistillationConfig(
    intermediate_matches=[
     {'layer_T':0, 'layer_S':0, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1},
     {'layer_T':8, 'layer_S':2, 'feature':'hidden', 'loss': 'hidden_mse','weight' : 1}])

distiller = GeneralDistiller(
    train_config=train_config, distill_config = distill_config,
    model_T = teacher_model, model_S = student_model, 
    adaptor_T = simple_adaptor, adaptor_S = simple_adaptor)

with distiller:
    distiller.train(optimizer, scheduler, dataloader, num_epochs=1, callback=None)
```

--------------------------------

### Start Distillation Training

Source: https://github.com/airaria/textbrewer/blob/master/README.md

Executes the distillation training process using the initialized distiller. This includes specifying the optimizer, dataloader, number of epochs, and optional scheduler and callback.

```python
with distiller:
    distiller.train(optimizer, dataloader, num_epochs=1, scheduler_class=scheduler_class, scheduler_args = scheduler_args, callback=None)
```

--------------------------------

### Distill BERT-base to 3-layer BERT with TextBrewer

Source: https://github.com/airaria/textbrewer/blob/master/README.md

Demonstrates the basic setup for knowledge distillation using TextBrewer's GeneralDistiller. It includes displaying model parameters, defining a data adaptor, and initializing training and distillation configurations.

```python
import textbrewer
from textbrewer import GeneralDistiller
from textbrewer import TrainingConfig, DistillationConfig

# Show the statistics of model parameters
print("\nteacher_model's parametrers:")
result, _ = textbrewer.utils.display_parameters(teacher_model,max_level=3)
print (result)

print("student_model's parametrers:")
result, _ = textbrewer.utils.display_parameters(student_model,max_level=3)
print (result)

# Define an adaptor for interpreting the model inputs and outputs
def simple_adaptor(batch, model_outputs):
      # The second and third elements of model outputs are the logits and hidden states
    return {'logits': model_outputs[1],
            'hidden': model_outputs[2]}

# Training configuration 
train_config = TrainingConfig()
# Distillation configuration

```

--------------------------------

### Configuring 'cwsm' Temperature Scheduler

Source: https://github.com/airaria/textbrewer/wiki/Presets

Example of configuring the 'cwsm' (Constant Weight, Slow Momentum) temperature scheduler in DistillationConfig, requiring a beta parameter.

```python
distill_config = DistillationConfig(
    temperature_scheduler = ['cwsm', 1]
    ...)
```

--------------------------------

### GeneralDistiller Adaptor Example

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Defines a custom adaptor function for a model that outputs logits, sequence output, and total loss. This adaptor maps model outputs and batch inputs to a dictionary format expected by the distiller.

```python
def BertForGLUESimpleAdaptor(batch, model_outputs):
  return {'logits': (model_outputs[0],),
      'hidden': model.outputs[1],
      'inputs_mask': batch[1]}
```

--------------------------------

### Install pyemd Package

Source: https://github.com/airaria/textbrewer/blob/master/examples/mnli_example/README.md

This command installs the pyemd package, which is a dependency for the EMDDistiller.

```bash
pip install pyemd
```

--------------------------------

### Configure and Run Distillation

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/msra_ner.ipynb

Sets up the optimizer, learning rate scheduler, and distillation configuration. Initiates the distillation training process using GeneralDistiller.

```python
from transformers import get_linear_schedule_with_warmup
from torch.optim import AdamW
from textbrewer import DistillationConfig, TrainingConfig, GeneralDistiller

# Assuming student_model, teacher_model, device, train_dataloader, label_list are defined
# Assuming batch_size is defined

num_epochs = 20
num_training_steps = len(train_dataloader) * num_epochs

optimizer = AdamW(student_model.parameters(), lr=1e-5)

scheduler_class = get_linear_schedule_with_warmup
scheduler_args = {'num_warmup_steps':int(0.1*num_training_steps), 'num_training_steps':num_training_steps}

def simple_adaptor(batch, model_outputs):
  return {"logits":model_outputs.logits, 'hidden': model_outputs.hidden_states}

distill_config = DistillationConfig(
    intermediate_matches=[{"layer_T":0, "layer_S":0, "feature":"hidden", "loss":"hidden_mse", "weight":1},
               {"layer_T":4, "layer_S":1, "feature":"hidden", "loss":"hidden_mse", "weight":1},
               {"layer_T":8, "layer_S":2, "feature":"hidden", "loss":"hidden_mse", "weight":1},
               {"layer_T":12,"layer_S":3, "feature":"hidden", "loss":"hidden_mse", "weight":1}])

train_config = TrainingConfig()
distiller = GeneralDistiller(
    train_config=train_config, distill_config=distill_config,
    model_T=teacher_model, model_S=student_model, 
    adaptor_T=simple_adaptor, adaptor_S=simple_adaptor)


with distiller:
    distiller.train(optimizer, train_dataloader, num_epochs, scheduler_class=scheduler_class, scheduler_args = scheduler_args, callback=None,
                    batch_postprocessor=proc_fn)
```

--------------------------------

### Convert SQuAD Examples to Features

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

This snippet demonstrates converting SQuAD development examples into features using the SquadV1Processor and a tokenizer. It specifies sequence length, stride, and query length parameters.

```python
processor = SquadV1Processor()
examples = processor.get_dev_examples('/content/')

features,eval_dataset = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=384,
    doc_stride=128,
    max_query_length=64,
    is_training=False,
    return_dataset="pt")
```

--------------------------------

### Convert SQuAD Examples to Features

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

Converts SQuAD dataset examples into features for model training. This function handles tokenization, sequence length, query length, and training mode. It can return a PyTorch dataset.

```python
features,train_dataset = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=384,
    doc_stride=128,
    max_query_length=64,
    is_training=True,
    return_dataset="pt"
)

```

--------------------------------

### BasicTrainer.train

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Distillers.rst

Initializes a BasicTrainer object and calls its train method to start training/distillation.

```APIDOC
## BasicTrainer.train

### Description
Trains the student model using the BasicTrainer.

### Method
`train`

### Parameters
This method is documented via the autoclass directive and its members are listed.
```

--------------------------------

### MultiTaskDistiller.train

Source: https://github.com/airaria/textbrewer/blob/master/docs/source/Distillers.rst

Initializes a MultiTaskDistiller object and calls its train method to start training/distillation.

```APIDOC
## MultiTaskDistiller.train

### Description
Trains the student model using the MultiTaskDistiller.

### Method
`train`

### Parameters
This method is documented via the autoclass directive and its members are listed.
```

--------------------------------

### Initialize TrainingConfig

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Initialize TrainingConfig with essential parameters like log directory and output directory. Other parameters can be left as default.

```python
from textbrewer import TrainingConfig

train_config = TrainingConfig(log_dir=my_log_dir, output_dir=my_output_dir)
```

--------------------------------

### Get Label List

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/msra_ner.ipynb

Extracts the list of possible NER tags from the training split of the dataset.

```python
label_list = datasets["train"].features[f"{task}_tags"].feature.names
label_list
```

--------------------------------

### Initialize Training Parameters

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

Sets up essential parameters for the training loop, including the number of epochs and the total number of training steps. This prepares for the actual model training process.

```python
#Start training
import tqdm
epochs = 2
t_total = len(train_dataloader) * epochs

```

--------------------------------

### EMDDistiller Usage Example

Source: https://github.com/airaria/textbrewer/blob/master/examples/mnli_example/README.md

This Python snippet shows the basic instantiation and usage of the EMDDistiller class for knowledge distillation.

```python
from distiller_emd import EMDDistiller
distiller = EMDDistiller(...)
with distiller:
    distiller.train(...)
```

--------------------------------

### Initialize DistillationConfig with Default Values

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Create a DistillationConfig instance using default parameters or specify a custom temperature.

```python
from textbrewer import DistillationConfig

# basic configuration: use default values, or try different temperatures 
distill_config = DistillationConfig(temperature=8)
```

--------------------------------

### BasicDistiller

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Performs basic single-model, single-task distillation without intermediate feature matching. Suitable for debugging or testing distillation setups.

```APIDOC
## BasicDistiller

### Description
Performs basic single-model, single-task distillation. This class does not support intermediate feature matching and is recommended for debugging or testing purposes.

### Class Signature
`BasicDistiller(train_config, distill_config, model_T, model_S, adaptor_T, adaptor_S)`

### Parameters
* **train_config** (`TrainingConfig`): Configuration for the training process.
* **distill_config** (`DistillationConfig`): Configuration for the distillation process.
* **model_T** (`torch.nn.Module`): The teacher model.
* **model_S** (`torch.nn.Module`): The student model.
* **adaptor_T** (`Callable`): A function to adapt the teacher model's outputs.
* **adaptor_S** (`Callable`): A function to adapt the student model's outputs.

### Train Method Signature
`train(optimizer, scheduler, dataloader, num_epochs, num_steps=None, callback=None, batch_postprocessor=None, **args)`

### Note
The `train` method has the same interface as `GeneralDistiller.train`.
```

--------------------------------

### Long-Range Disordering Example

Source: https://github.com/airaria/textbrewer/wiki/Classes-and-Functions

Demonstrates long-range disordering of tokens, showing how to swap halves of spans based on a specified length or relative length.

```python
long_disorder([0,1,2,3,4,5,6,7,8,9,10], p=1, length=0.4)
# [2, 3, 0, 1, 6, 7, 4, 5, 8, 9]
```

--------------------------------

### Prepare Optimizer and Schedule

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sqaudv1.1.ipynb

Initializes the AdamW optimizer and a linear learning rate scheduler with warmup steps for model training.

```python
optimizer = AdamW(model.parameters(), lr=3e-5, eps=1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=t_total)
```

--------------------------------

### Set Device to CUDA or CPU

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/sst2.ipynb

Sets the computation device to CUDA if available, otherwise defaults to CPU. This is a common setup for PyTorch models.

```python
import torch
device='cuda' if torch.cuda.is_available() else 'cpu'
```

--------------------------------

### Initialize Trainer

Source: https://github.com/airaria/textbrewer/blob/master/examples/notebook_examples/msra_ner.ipynb

Sets up the Trainer object with the model, training arguments, datasets, data collator, tokenizer, and compute_metrics function.

```python
trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
```