### Setup Fairseq and Dependencies

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/paraphraser/README.md

Clone the fairseq repository and install necessary dependencies, including sacremoses and sentencepiece, for the paraphrasing functionality.

```bash
git clone https://github.com/pytorch/fairseq.git
cd fairseq
pip install --editable .
pip install sacremoses sentencepiece
```

--------------------------------

### Install SOFA from Source

Source: https://github.com/alibaba/alicemind/blob/main/sofa/README.md

Clone the AliceMind repository and install SOFA using the setup script. Ensure Python version is 3.6+ and transformers version is 4.10.0+.

```shell
git clone https://github.com/alibaba/AliceMind.git
cd AliceMind/sofa
python3 setup.py install
```

--------------------------------

### Install wav2letter Bindings

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/speech_recognition/README.md

Commands to install prerequisites and wav2letter python bindings.

```bash
# additional prerequisites - use equivalents for your distro
sudo apt-get install build-essential cmake libatlas-base-dev libfftw3-dev liblzma-dev libbz2-dev libzstd-dev
# install KenLM from source
git clone https://github.com/kpu/kenlm.git
cd kenlm
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j16
cd ..
export KENLM_ROOT_DIR=$(pwd)
cd ..
# install wav2letter python bindings
git clone https://github.com/facebookresearch/wav2letter.git
cd wav2letter/bindings/python
# make sure your python environment is active at this point
pip install torch packaging
pip install -e .
# try some examples to verify installation succeeded
python ./examples/criterion_example.py
python ./examples/decoder_example.py ../../src/decoder/test
python ./examples/feature_example.py ../../src/feature/test/data
```

--------------------------------

### Set up Fairseq Environment Variables

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/roberta/wsc/README.md

Before running the training script, set the FAIRSEQ_PATH and FAIRSEQ_USER_DIR environment variables to point to your Fairseq installation and the WSC example directory, respectively.

```bash
FAIRSEQ_PATH=/path/to/fairseq
FAIRSEQ_USER_DIR=${FAIRSEQ_PATH}/examples/roberta/wsc
```

--------------------------------

### Install Apex for Faster Training

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/README.md

Install NVIDIA's Apex library for optimized training performance. This involves cloning the repository and running a pip install command with specific global options.

```bash
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" \
  --global-option="--deprecated_fused_adam" --global-option="--xentropy" \
  --global-option="--fast_multihead_attn" ./
```

--------------------------------

### Pronoun disambiguation setup

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/roberta/README.md

Installation commands for spaCy dependencies required for the WSC task.

```bash
pip install spacy
python -m spacy download en_core_web_lg
```

--------------------------------

### Install Requirements

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/simultaneous_translation/docs/baseline.md

Commands to clone the specific fairseq branch and install dependencies.

```bash
git clone --branch simulastsharedtask git@github.com:pytorch/fairseq.git
cd fairseq
pip install -e .
```

```bash
conda install -c powerai sentencepiece
```

--------------------------------

### Execute Model Run

Source: https://github.com/alibaba/alicemind/blob/main/PALM/text2image/notebook/vqganclip_palm.ipynb

Initializes the device and loads the model to start the generation process.

```python
#@title Actually do the run...
from urllib.request import urlopen
import numpy as np


device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)
if texts:
    print('Using texts:', texts)
if target_images:
    print('Using image prompts:', target_images)
if args.seed is None:
    seed = torch.seed()
else:
    seed = args.seed
torch.manual_seed(seed)
print('Using seed:', seed)


#palm_encoder=load_alice_model()
palm_encoder=encoder

model = load_vqgan_model(args.vqgan_config, args.vqgan_checkpoint).to(device)
```

--------------------------------

### Initialize Environment and Dependencies

Source: https://github.com/alibaba/alicemind/blob/main/PALM/text2image/notebook/vqganclip_palm.ipynb

Clones necessary repositories and installs required Python packages for the PALM environment.

```bash
!git clone https://github.com/openai/CLIP
!git clone https://github.com/alibaba/AliceMind.git
!git clone https://github.com/CompVis/taming-transformers.git
!pip install ftfy regex tqdm omegaconf pytorch-lightning
!pip install kornia
!pip install imageio-ffmpeg   
!pip install einops          
!mkdir steps


!wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1mSp-4KfBwGKUAdWiW-ctOR9Qgi0a-w9B' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1mSp-4KfBwGKUAdWiW-ctOR9Qgi0a-w9B" -O palm_model_and_data.tar.gz && rm -rf /tmp/cookies.txt)
!tar -zxvf palm_model_and_data.tar.gz
```

--------------------------------

### Install Dependencies

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/backtranslation/README.md

Install required Python packages for preprocessing.

```bash
pip install subword_nmt sacremoses
```

--------------------------------

### Install Fairseq Locally

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/README.md

Clone the fairseq repository and install it in editable mode for local development. Ensure PyTorch and Python versions meet the requirements.

```bash
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./
```

--------------------------------

### Install PST dependencies

Source: https://github.com/alibaba/alicemind/blob/main/S4/PST/README.md

Install the required packages for the PST repository.

```bash
pip3 install -r requirement.txt
```

--------------------------------

### Install Libraries with pip

Source: https://github.com/alibaba/alicemind/blob/main/PALM/README.md

Installs necessary libraries for the project. Ensure PyTorch version 1.1.0 is installed separately.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Dependencies for Preprocessing

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/translation/README.md

Install necessary Python packages for text preprocessing before using pre-trained translation models.

```bash
pip install fastBPE sacremoses subword_nmt
```

--------------------------------

### Setup and execute a Fairseq task

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/docs/tasks.md

Initializes a task, builds the model and criterion, loads datasets, and iterates through batches to compute loss.

```python
# setup the task (e.g., load dictionaries)
task = fairseq.tasks.setup_task(args)

# build model and criterion
model = task.build_model(args)
criterion = task.build_criterion(args)

# load datasets
task.load_dataset('train')
task.load_dataset('valid')

# iterate over mini-batches of data
batch_itr = task.get_batch_iterator(
    task.dataset('train'), max_tokens=4096,
)
for batch in batch_itr:
    # compute the loss
    loss, sample_size, logging_output = task.get_loss(
        model, criterion, batch,
    )
    loss.backward()
```

--------------------------------

### Install Dependencies for Preprocessing

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/language_model/README.md

Installs necessary Python packages for data preprocessing. Ensure these are installed before proceeding with data preparation.

```bash
pip install fastBPE sacremoses
```

--------------------------------

### Run Sparse Training with PST

Source: https://context7.com/alibaba/alicemind/llms.txt

Setup and execution commands for sparse-aware training on NLU tasks.

```bash
pip3 install -r requirement.txt

# Add PST to Python path
export PYTHONPATH=$(pwd):$PYTHONPATH

# Run sparse training on NLU tasks (BERT/RoBERTa)
cd NLU
python run_glue.py \
  --model_name_or_path bert-base-uncased \
  --task_name sst2 \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir ./output
```

--------------------------------

### Install Dependencies

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLU/README.md

Run this script to install required Python packages including transformers, seqeval, and various text segmentation tools.

```bash
bash install_tools.sh
```

--------------------------------

### Run Evaluation Server

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/simultaneous_translation/docs/baseline.md

Commands to start the evaluation server for text or speech translation models.

```bash
python ./eval/server.py \
    --hostname localhost \
    --port 12321 \
    --src-file $DATA_ROOT/data/dev/txt/dev.en \
    --ref-file $DATA_ROOT/data/dev/txt/dev.de
```

```bash
python ./eval/server.py \
    --hostname localhost \
    --port 12321 \
    --ref-file $DATA_ROOT \
    --data-type speech
```

--------------------------------

### Install PyArrow for Large Datasets

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/README.md

Install the PyArrow library using pip, which is recommended for handling large datasets efficiently.

```bash
pip install pyarrow
```

--------------------------------

### Input Format Example

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/roberta/commonsense_qa/README.md

The expected input format for the model, prepending Q: to the question and A: to the answer.

```text
<s> Q: Where would I not want a fox? </s> A: hen house </s>
```

--------------------------------

### Monitor Parameter Count

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/layerdrop/README.md

Examples of parameter count output for full and pruned models.

```text
num. model params: 246933504
```

```text
num. model params: 146163712
```

--------------------------------

### Finetune RoBERTa on WinoGrande Dataset Configuration

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/roberta/wsc/README.md

Configuration variables for finetuning RoBERTa on WinoGrande datasets. This setup is for a more efficient implementation suitable for datasets with two candidates per instance.

```bash
TOTAL_NUM_UPDATES=23750 # Total number of training steps.
WARMUP_UPDATES=2375     # Linearly increase LR over this many steps.
LR=1e-05                # Peak LR for polynomial LR scheduler.
MAX_SENTENCES=32        # Batch size per GPU.
SEED=1                  # Random seed.
ROBERTA_PATH=/path/to/roberta/model.pt

# we use the --user-dir option to load the task and criterion

```

--------------------------------

### Implement Custom Architecture in __init__.py

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/docs/overview.md

Example of registering a custom model architecture within a user-defined module.

```python
from fairseq.models import register_model_architecture
from fairseq.models.transformer import transformer_vaswani_wmt_en_de_big

@register_model_architecture('transformer', 'my_transformer')
def transformer_mmt_big(args):
    transformer_vaswani_wmt_en_de_big(args)
```

--------------------------------

### Parenthetical Expression Start

Source: https://github.com/alibaba/alicemind/blob/main/StructuralLM/tokenizer_model/merges.txt

The beginning of a parenthetical expression.

```text
( .
```

--------------------------------

### Fine-tune VECO for NLU Tasks

Source: https://context7.com/alibaba/alicemind/llms.txt

Install dependencies and fine-tune VECO on cross-lingual NLI and QA datasets.

```bash
# Install dependencies
bash install_tools.sh

# Download multilingual datasets
mkdir -p download
bash scripts/download_data.sh

# Fine-tune on XNLI (Cross-lingual NLI)
bash scripts/train_xnli.sh /path/to/veco-large 0,1,2,3 2e-5 3 32

# Fine-tune on XQuAD (Cross-lingual QA)
bash scripts/train_squad.sh /path/to/veco-large 0,1,2,3 3e-5 2 16
```

--------------------------------

### Function or Block Start

Source: https://github.com/alibaba/alicemind/blob/main/StructuralLM/tokenizer_model/merges.txt

Syntax indicating the beginning of a code block or function scope.

```text
) {
```

--------------------------------

### Generate translations using Fairseq CLI

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/translation/README.md

Generate translations from binarized test sets using the Fairseq command-line interface. This example shows how to set up data and run generation with specific parameters.

```bash
mkdir -p data-bin
curl https://dl.fbaipublicfiles.com/fairseq/models/wmt14.v2.en-fr.fconv-py.tar.bz2 | tar xvjf - -C data-bin
curl https://dl.fbaipublicfiles.com/fairseq/data/wmt14.v2.en-fr.newstest2014.tar.bz2 | tar xvjf - -C data-bin
fairseq-generate data-bin/wmt14.en-fr.newstest2014  \
    --path data-bin/wmt14.en-fr.fconv-py/model.pt \
    --beam 5 --batch-size 128 --remove-bpe | tee /tmp/gen.out
```

--------------------------------

### Load and Translate Models via PyTorch Hub

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/pay_less_attention_paper/README.md

Example of listing, loading, and using a pre-trained model from PyTorch Hub.

```python
import torch

# List available models
torch.hub.list('pytorch/fairseq')  # [..., 'lightconv.glu.wmt17.zh-en', ... ]

# Load a transformer trained on WMT'16 En-De
zh2en = torch.hub.load('pytorch/fairseq', 'lightconv.glu.wmt17.zh-en', tokenizer='moses', bpe='subword_nmt')

# The underlying model is available under the *models* attribute
assert isinstance(zh2en.models[0], fairseq.models.lightconv.LightConvModel)

# Translate a sentence
zh2en.translate('你好 世界')
# 'Hello World'
```

--------------------------------

### Fine-tune on FUNSD dataset

Source: https://github.com/alibaba/alicemind/blob/main/StructuralLM/README.md

Execute the shell script to start the fine-tuning process on the processed FUNSD dataset.

```bash
sh finetune_funsd_dataset.sh
```

--------------------------------

### Integrating Quantization with Fairseq

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/quant_noise/README.md

Example code for integrating Quant-Noise and iPQ into your own models using Fairseq's quantization utilities. This process involves wrapping modules and then quantizing the trained model, potentially without modifying the training loop.

```python
from fairseq.modules.quantization.pq import quantize_model_, SizeTracker

# get configuration parameters
n_centroids_config = config["n_centroids"]
block_sizes_config = config["block_sizes"]
layers_to_quantize = config["layers_to_quantize"]

# size tracker for keeping track of assignments, centroids and non-compressed sizes
size_tracker = SizeTracker(model)

# Quantize model by stages
for step in range(len(layers_to_quantize)):

    # quantize model in-place
    quantized_layers = quantize_model_(
        model,
        size_tracker,
        layers_to_quantize,
        block_sizes_config,
        n_centroids_config,
        step=step,
    )
    logger.info(f"Finetuning stage {step}, quantized layers: {quantized_layers}")
    logger.info(f"{size_tracker}")

    # Don't forget to re-create/update trainer/optimizer since model parameters have changed
    optimizer = ...

    # Finetune the centroids with your usual training loop for a few epochs
    trainer.train_epoch()
```

--------------------------------

### Initialize Environment and Dependencies

Source: https://github.com/alibaba/alicemind/blob/main/S4/PST/NLG/README.md

Run these scripts to download necessary pretrained checkpoints and datasets before training.

```bash
bash download_pretrained_checkpoints.sh
bash create_datasets.sh
cd ./eval
bash download_evalscript.sh
cd ..
```

--------------------------------

### Download Datasets

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLU/README.md

Create a download directory and execute the script to fetch the necessary datasets for fine-tuning.

```bash
mkdir -p download
bash scripts/download_data.sh
```

--------------------------------

### Install CUDA Kernels

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/pay_less_attention_paper/README.md

Commands to compile and install the memory-efficient CUDA kernels for light and dynamic convolution layers.

```sh
# to install lightconv
cd fairseq/modules/lightconv_layer
python cuda_function_gen.py
python setup.py install

# to install dynamicconv
cd fairseq/modules/dynamicconv_layer
python cuda_function_gen.py
python setup.py install
```

--------------------------------

### Initialize CLIP and VQGAN components

Source: https://github.com/alibaba/alicemind/blob/main/PALM/text2image/notebook/vqganclip_palm.ipynb

Sets up the CLIP perceptor and defines the latent space dimensions and constraints for the VQGAN model.

```python
perceptor = clip.load(args.clip_model, jit=False)[0].eval().requires_grad_(False).to(device)


cut_size = perceptor.visual.input_resolution

f = 2**(model.decoder.num_resolutions - 1)
make_cutouts = MakeCutouts(cut_size, args.cutn, cut_pow=args.cut_pow)

toksX, toksY = args.size[0] // f, args.size[1] // f
sideX, sideY = toksX * f, toksY * f


e_dim = model.quantize.e_dim
n_toks = model.quantize.n_e
z_min = model.quantize.embedding.weight.min(dim=0).values[None, :, None, None]
z_max = model.quantize.embedding.weight.max(dim=0).values[None, :, None, None]
```

--------------------------------

### Run CAP-soft on QQP/MNLI

Source: https://github.com/alibaba/alicemind/blob/main/ContrastivePruning/UnstructuredPruning/README.md

Execute the CAP-soft model on the QQP and MNLI datasets. Ensure the TEACHER_PATH is correctly configured.

```bash
bash run_glue_soft_kd.sh
```

--------------------------------

### Run CAP-m on QQP/MNLI

Source: https://github.com/alibaba/alicemind/blob/main/ContrastivePruning/UnstructuredPruning/README.md

Execute the CAP-m model on the QQP and MNLI datasets. Ensure the TEACHER_PATH is correctly configured.

```bash
bash run_glue_topk_kd.sh
```

--------------------------------

### Initialize Sofa Environment

Source: https://github.com/alibaba/alicemind/blob/main/sofa/Tutorial.md

Set up the environment for the sofa toolkit before loading models.

```python
import sofa
sofa.environ("huggingface")
```

--------------------------------

### Prepare Training Data Manifest for VQ-Wav2Vec

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/wav2vec/README.md

Generate a data manifest file for training vq-wav2vec models. This is similar to wav2vec manifest preparation, specifying the audio directory and output destination.

```bash
python examples/wav2vec/wav2vec_manifest.py /path/to/waves --dest /manifest/path --ext wav
```

--------------------------------

### Register Custom Model

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/docs/overview.md

Example of using the @register_model decorator to define a new model architecture.

```python
@register_model('my_lstm')
class MyLSTM(FairseqEncoderDecoderModel):
    (...)
```

--------------------------------

### Download WritingPrompts Dataset

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/stories/README.md

Commands to navigate to the stories directory and download the WritingPrompts dataset.

```bash
cd examples/stories
curl https://dl.fbaipublicfiles.com/fairseq/data/writingPrompts.tar.gz | tar xvzf -
```

--------------------------------

### Run VECO Fine-tuning Script

Source: https://github.com/alibaba/alicemind/blob/main/sofa/Tutorial.md

Execute the provided shell script to start VECO NLU fine-tuning.

```shell
# NLU finetune
sh examples/finetune_veco.sh
# veco for NLG is comming soon! :)
```

--------------------------------

### Run PALM Fine-tuning Script

Source: https://github.com/alibaba/alicemind/blob/main/sofa/Tutorial.md

Execute the provided shell script to start PALM NLG fine-tuning.

```shell
# palm-NLG finetune
sh examples/finetune_palm.sh
```

--------------------------------

### Fine-tune VECO Models

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLU/README.md

Commands to initiate training for different NLU tasks. Replace placeholders with your specific model path, GPU ID, learning rate, epoch count, and batch size.

```bash
bash scripts/train_xnli.sh [MODEL_PATH] [GPU] [LR] [EPOCH] [BATCH_SIZE]
```

```bash
bash scripts/train_squad.sh [MODEL_PATH] [GPU] [LR] [EPOCH] [BATCH_SIZE]
```

```bash
bash scripts/run_tatoeba.sh [MODEL_PATH] [GPU] [LAYER]
```

--------------------------------

### XNLI Fine-tuning Evaluation Results

Source: https://github.com/alibaba/alicemind/blob/main/StructuralLM/multilingual.md

Example evaluation results after fine-tuning on the XNLI dataset with the Chinese-only model.

```text
***** Eval results *****
eval_accuracy = 0.774116
eval_loss = 0.83554
global_step = 24543
loss = 0.74603
```

--------------------------------

### Train Wav2Vec Model using CLI

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/wav2vec/README.md

Initiate the training process for a wav2vec model using the command-line interface. This command includes various hyperparameters for model architecture, optimization, and training schedule.

```bash
python train.py /manifest/path --save-dir /model/path --num-workers 6 --fp16 --max-update 400000 --save-interval 1 --no-epoch-checkpoints \
--arch wav2vec --task audio_pretraining --lr 1e-06 --min-lr 1e-09 --optimizer adam --max-lr 0.005 --lr-scheduler cosine \
--conv-feature-layers [(512, 10, 5), (512, 8, 4), (512, 4, 2), (512, 4, 2), (512, 4, 2), (512, 1, 1), (512, 1, 1)] \
--conv-aggregator-layers [(512, 2, 1), (512, 3, 1), (512, 4, 1), (512, 5, 1), (512, 6, 1), (512, 7, 1), (512, 8, 1), (512, 9, 1), (512, 10, 1), (512, 11, 1), (512, 12, 1), (512, 13, 1)] \
--skip-connections-agg --residual-scale 0.5 --log-compression --warmup-updates 500 --warmup-init-lr 1e-07 --criterion binary_cross_entropy --num-negatives 10 \
--max-sample-size 150000 --max-tokens 1500000 ---skip-invalid-size-inputs-valid-test
```

--------------------------------

### Download and Extract StructBERT Model

Source: https://github.com/alibaba/alicemind/blob/main/sofa/README.md

Download the StructBERT-large model files and extract them to a specified directory. Ensure you have wget installed.

```shell
wget https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/english_sbert-large-std-512.tar.gz -P /tmp
mkdir -p /tmp/english_sbert-large-std-512
tar xvf /tmp/english_sbert-large-std-512.tar.gz -C /tmp/english_sbert-large-std-512
```

--------------------------------

### Run CAP-soft on SQuAD v1.1

Source: https://github.com/alibaba/alicemind/blob/main/ContrastivePruning/UnstructuredPruning/README.md

Execute the CAP-soft model on the SQuAD v1.1 dataset. Ensure the TEACHER_PATH is correctly configured.

```bash
bash run_squad_soft_kd.sh
```

--------------------------------

### Clone PLUG Repository and Run Inference

Source: https://github.com/alibaba/alicemind/blob/main/PLUG/README.md

Clone the AliceMind repository, navigate to the PLUG directory, and execute the inference script. Ensure you have the necessary hardware and have applied for model download.

```shell
git clone https://github.com/alibaba/AliceMind.git
cd AliceMind/PLUG

# Download model: About 55 GB after unzip.
[Apply for download](https://www.alice-mind.com/portal#/)

# inference
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=8 predict.py
```

--------------------------------

### Download and Extract MBART Model

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/mbart/README.md

Download the pre-trained MBART CC25 model and extract its contents. Ensure you have `wget` and `tar` installed.

```bash
wget https://dl.fbaipublicfiles.com/fairseq/models/mbart/mbart.CC25.tar.gz
tar -xzvf mbart.CC25.tar.gz
```

--------------------------------

### Run CAP-m on SQuAD v1.1

Source: https://github.com/alibaba/alicemind/blob/main/ContrastivePruning/UnstructuredPruning/README.md

Execute the CAP-m model on the SQuAD v1.1 dataset. Ensure the TEACHER_PATH is correctly configured.

```bash
bash run_squad_topk_kd.sh
```

--------------------------------

### Evaluate with Client

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/simultaneous_translation/docs/baseline.md

Commands to run the client for evaluating translation quality and latency for text and speech models.

```bash
# TEXT
python $fairseq_dir/examples/simultaneous_translation/evaluate.py \
    data-bin/mustc_en_de \
    --user-dir $FAIRSEQ/examples/simultaneous_translation \
    --src-spm unigram-en-10000-3000/spm.model\
    --tgt-spm unigram-de-10000-3000/spm.model\
    -s en -t de \
    --path checkpoints/checkpoint_best.pt

# SPEECH
python $fairseq_dir/examples/simultaneous_translation/evaluate.py \
    data-bin/mustc_en_de \
    --user-dir $FAIRSEQ/examples/simultaneous_translation \
    --data-type speech \
    --tgt-spm unigram-de-10000-3000/spm.model\
    -s en -t de \
    --path checkpoints/checkpoint_best.pt
```

--------------------------------

### Perplexity After Renormalization Example

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/megatron_11b/README.md

Calculates the perplexity after renormalization using the provided token counts and an unnormalized perplexity of 8.46. The result is 10.54.

```text
2 ^ ( log_2(8.46) * (270847 / 245566)) = 10.54
```

--------------------------------

### Reproduce GLUE Results

Source: https://github.com/alibaba/alicemind/blob/main/sofa/Tutorial.md

Executes the shell script to reproduce GLUE benchmark results.

```shell
# reproduce all of the glue results
sh examples/finetune_sbert_glue.sh
```

--------------------------------

### Client Protocol Actions

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/simultaneous_translation/docs/evaluation.md

Defines the actions the client can take to interact with the server. Use 'Get' to request new data and 'SEND' to submit predictions.

```json
{key: "Get", value: None}
```

```json
{key: "SEND", value: "W"}
```

--------------------------------

### SentencePiece Tokenization for Training Data

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/mbart/README.md

Tokenizes training and validation data using SentencePiece for MBART. Requires SentencePiece to be installed and `spm_encode` to be in your PATH. Replace `/path/to/sentencepiece/build/src/spm_encode` with the actual path.

```bash
SPM=/path/to/sentencepiece/build/src/spm_encode
MODEL=sentence.bpe.model
${SPM} --model=${MODEL} < ${DATA}/${TRAIN}.${SRC} > ${DATA}/${TRAIN}.spm.${SRC} &
${SPM} --model=${MODEL} < ${DATA}/${TRAIN}.${TGT} > ${DATA}/${TRAIN}.spm.${TGT} &
${SPM} --model=${MODEL} < ${DATA}/${VALID}.${SRC} > ${DATA}/${VALID}.spm.${SRC} &
${SPM} --model=${MODEL} < ${DATA}/${VALID}.${TGT} > ${DATA}/${VALID}.spm.${TGT} &
${SPM} --model=${MODEL} < ${DATA}/${TEST}.${SRC} > ${DATA}/${TEST}.spm.${SRC} &
${SPM} --model=${MODEL} < ${DATA}/${TEST}.${TGT} > ${DATA}/${TEST}.spm.${TGT} &
```

--------------------------------

### Download Fairseq Dictionary

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/roberta/README.custom_classification.md

Download the fairseq dictionary file (dict.txt) which is required for preprocessing the data.

```bash
# Download fairseq dictionary.
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt'
```

--------------------------------

### Prepare Training Data Manifest for Wav2Vec

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/wav2vec/README.md

Generate a data manifest file required for training wav2vec models. Specify the directory containing audio files and the destination path for the manifest.

```bash
python scripts/wav2vec_manifest.py /path/to/waves --dest /manifest/path --ext wav
```

--------------------------------

### Fine-tune BART on RTE Task

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/bart/README.glue.md

Example command for fine-tuning the BART model on the RTE GLUE task. Adjust hyperparameters and paths as needed. Ensure `BART_PATH` points to your pre-trained BART model.

```bash
TOTAL_NUM_UPDATES=2036
WARMUP_UPDATES=61
LR=1e-05
NUM_CLASSES=2
MAX_SENTENCES=16
BART_PATH=/path/to/bart/model.pt

CUDA_VISIBLE_DEVICES=0,1 python train.py RTE-bin/ \
    --restore-file $BART_PATH \
    --max-sentences $MAX_SENTENCES \
    --max-tokens 4400 \
    --task sentence_prediction \
    --add-prev-output-tokens \
    --layernorm-embedding \
    --share-all-embeddings \
    --share-decoder-input-output-embed \
    --reset-optimizer --reset-dataloader --reset-meters \
    --required-batch-size-multiple 1 \
    --init-token 0 \
    --arch bart_large \
    --criterion sentence_prediction \
    --num-classes $NUM_CLASSES \
    --dropout 0.1 --attention-dropout 0.1 \
    --weight-decay 0.01 --optimizer adam --adam-betas "(0.9, 0.98)" --adam-eps 1e-08 \
    --clip-norm 0.0 \
    --lr-scheduler polynomial_decay --lr $LR --total-num-update $TOTAL_NUM_UPDATES --warmup-updates $WARMUP_UPDATES \
    --fp16 --fp16-init-scale 4 --threshold-loss-scale 1 --fp16-scale-window 128 \
    --max-epoch 10 \
    --find-unused-parameters \
    --best-checkpoint-metric accuracy --maximize-best-checkpoint-metric;
```

--------------------------------

### Fine-tune StructuralLM

Source: https://context7.com/alibaba/alicemind/llms.txt

Download the StructuralLM model and FUNSD dataset, then run the fine-tuning script.

```bash
# Download StructuralLM model
wget https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructuralLM/model/structural_lm_models.tar.gz

# Download FUNSD dataset
wget https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructuralLM/data/funsd_dataset_structurallm.tar.gz

# Fine-tune on FUNSD document understanding
sh finetune_funsd_dataset.sh

# Requires TensorFlow 1.14.0
pip install -r requirements.txt
```

--------------------------------

### Download GLUE Data

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/bart/README.glue.md

Use this script to download all necessary data for GLUE benchmark tasks from the official website.

```bash
wget https://gist.githubusercontent.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e/raw/17b8dd0d724281ed7c3b2aeeda662b92809aadd5/download_glue_data.py
python download_glue_data.py --data_dir glue_data --tasks all
```

--------------------------------

### BPE Preprocessing Script

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/bart/README.summarization.md

Downloads necessary BPE files and preprocesses the dataset using multiprocessing. Ensure the input files are correctly specified.

```bash
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/encoder.json'
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/vocab.bpe'
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt'

TASK=cnn_dm
for SPLIT in train val
do
  for LANG in source target
  do
    python -m examples.roberta.multiprocessing_bpe_encoder \
    --encoder-json encoder.json \
    --vocab-bpe vocab.bpe \
    --inputs "$TASK/$SPLIT.$LANG" \
    --outputs "$TASK/$SPLIT.bpe.$LANG" \
    --workers 60 \
    --keep-empty; 
  done
done
```

--------------------------------

### Binarize, Train, and Generate with Fairseq

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/stories/README.md

Commands for binarizing the dataset, training the convolutional model, and generating text.

```bash
# Binarize the dataset:
export TEXT=examples/stories/writingPrompts
fairseq-preprocess --source-lang wp_source --target-lang wp_target \
    --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \
    --destdir data-bin/writingPrompts --padding-factor 1 --thresholdtgt 10 --thresholdsrc 10

# Train the model:
fairseq-train data-bin/writingPrompts -a fconv_self_att_wp --lr 0.25 --clip-norm 0.1 --max-tokens 1500 --lr-scheduler reduce_lr_on_plateau --decoder-attention True --encoder-attention False --criterion label_smoothed_cross_entropy --weight-decay .0000001 --label-smoothing 0 --source-lang wp_source --target-lang wp_target --gated-attention True --self-attention True --project-input True --pretrained False

# Train a fusion model:
# add the arguments: --pretrained True --pretrained-checkpoint path/to/checkpoint

# Generate:
# Note: to load the pretrained model at generation time, you need to pass in a model-override argument to communicate to the fusion model at generation time where you have placed the pretrained checkpoint. By default, it will load the exact path of the fusion model's pretrained model from training time. You should use model-override if you have moved the pretrained model (or are using our provided models). If you are generating from a non-fusion model, the model-override argument is not necessary.

fairseq-generate data-bin/writingPrompts --path /path/to/trained/model/checkpoint_best.pt --batch-size 32 --beam 1 --sampling --sampling-topk 10 --temperature 0.8 --nbest 1 --model-overrides "{'pretrained_checkpoint':'/path/to/pretrained/model/checkpoint'}"
```

--------------------------------

### Launch Distributed Training

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/docs/getting_started.md

Use torch.distributed.launch to scale training across multiple nodes and GPUs.

```console
> python -m torch.distributed.launch --nproc_per_node=8 \
    --nnodes=2 --node_rank=0 --master_addr="192.168.1.1" \
    --master_port=1234 \
    $(which fairseq-train) data-bin/wmt16_en_de_bpe32k \
    --arch transformer_vaswani_wmt_en_de_big --share-all-embeddings \
    --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
    --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \
    --lr 0.0005 --min-lr 1e-09 \
    --dropout 0.3 --weight-decay 0.0 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \
    --max-tokens 3584 \
    --fp16  --distributed-no-spawn
```

--------------------------------

### Implement Fairseq Training Loop

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/docs/overview.md

High-level training flow demonstrating the iteration over batches and the execution of training steps.

```python
for epoch in range(num_epochs):
    itr = task.get_batch_iterator(task.dataset('train'))
    for num_updates, batch in enumerate(itr):
        task.train_step(batch, model, criterion, optimizer)
        average_and_clip_gradients()
        optimizer.step()
        lr_scheduler.step_update(num_updates)
    lr_scheduler.step(epoch)
```

--------------------------------

### Load RoBERTa model from checkpoint

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/roberta/README.custom_classification.md

Initializes the model using the fairseq hub interface. Ensure the checkpoint file and data path are correctly specified.

```python
from fairseq.models.roberta import RobertaModel
roberta = RobertaModel.from_pretrained(
    'checkpoints',
    checkpoint_file='checkpoint_best.pt',
    data_name_or_path='IMDB-bin'
)
roberta.eval()  # disable dropout
```

--------------------------------

### Load and Use Pre-trained English LM with PyTorch Hub

Source: https://github.com/alibaba/alicemind/blob/main/VECO/NLG/examples/language_model/README.md

Demonstrates loading a pre-trained English language model from PyTorch Hub, setting it to evaluation mode, moving it to a GPU, and then sampling text or computing perplexity.

```python
import torch

# List available models
torch.hub.list('pytorch/fairseq')  # [..., 'transformer_lm.wmt19.en', ...]

# Load an English LM trained on WMT'19 News Crawl data
en_lm = torch.hub.load('pytorch/fairseq', 'transformer_lm.wmt19.en', tokenizer='moses', bpe='fastbpe')
en_lm.eval()  # disable dropout

# Move model to GPU
en_lm.cuda()

# Sample from the language model
en_lm.sample('Barack Obama', beam=1, sampling=True, sampling_topk=10, temperature=0.8)
# "Barack Obama is coming to Sydney and New Zealand (...)"

# Compute perplexity for a sequence
en_lm.score('Barack Obama is coming to Sydney and New Zealand')['positional_scores'].mean().neg().exp()
# tensor(15.1474)
```

--------------------------------

### Run LatticeBERT Pre-training

Source: https://github.com/alibaba/alicemind/blob/main/LatticeBERT/README.md

Execute the pre-training process for LatticeBERT using the newly created TFRecords and domain-specific checkpoint. Multi-GPU training can be enabled using Horovod.

```shell
python run_pretraining_labert.py \
--init_checkpoint=/path/to/new/checkpoint/ \
--input_file=data/sample.tfrecord \
--eval_file=data/sample.tfrecord \
--labert_config_file=/path/to/new/checkpoint/labert_config.json \
--output_dir=/tmp/labert_pretrain \
--do_train
```