### Install Project Dependencies using Pip

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/multilingual/data_scripts/README.md

Installs all necessary Python packages listed in the 'requirement.txt' file. This is a standard step to ensure all project dependencies are met before running the code.

```bash
pip install -r requirement.txt

```

--------------------------------

### Initiate Fairseq Distributed Training

Source: https://context7.com/facebookresearch/fairseq/llms.txt

Provides a bash command example to start distributed training in Fairseq. It specifies the data directory, task, and architecture. Requires Fairseq installation and a configured distributed environment.

```bash
# Example command to start distributed training
# Assumes data is preprocessed into data-bin/custom
# --task custom_translation specifies the custom task defined previously
# --arch transformer specifies the model architecture
# Other arguments like --distributed-world-size, --distributed-backend would be needed for actual distributed setup.
# fairseq-train data-bin/custom \
#     --task custom_translation \
#     --arch transformer \
#     ...
```

--------------------------------

### Example Usage of Fairseq Tasks

Source: https://github.com/facebookresearch/fairseq/blob/main/docs/tasks.rst

Demonstrates the typical workflow for using Fairseq tasks, including setup, model/criterion building, dataset loading, and iterating over mini-batches to compute loss. This example assumes 'args' is a pre-configured argument object.

```python
import fairseq.tasks

# setup the task (e.g., load dictionaries)
task = fairseq.tasks.setup_task(args)

# build model and criterion
model = task.build_model(args)
criterion = task.build_criterion(args)

# load datasets
task.load_dataset('train')
task.load_dataset('valid')

# iterate over mini-batches of data
batch_itr = task.get_batch_iterator(
    task.dataset('train'), max_tokens=4096,
)
for batch in batch_itr:
    # compute the loss
    loss, sample_size, logging_output = task.get_loss(
        model, criterion, batch,
    )
    loss.backward()
```

--------------------------------

### Install Flashlight Library in Fairseq Project

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/asr/tutorial/MMS_ASR_Inference_Colab.ipynb

This Python script installs the Flashlight library, a dependency for the fairseq project. It clones the Flashlight repository, checks out a specific commit, and then installs it using setup.py. Finally, it navigates back to the fairseq directory.

```python
! rm -rf flashlight
! git clone --recursive https://github.com/flashlight/flashlight.git
%cd flashlight
! git checkout 035ead6efefb82b47c8c2e643603e87d38850076 
%cd bindings/python 
! python3 setup.py install

%cd /content/fairseq 

```

--------------------------------

### Install fairseq and MMPT Toolkit

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/MMPT/README.md

Installs the fairseq library and the MMPT toolkit. It requires cloning the fairseq repository, installing it locally, setting an environment variable, and then installing the MMPT toolkit from its directory. Optional apex installation for FP16 training is mentioned.

```bash
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install -e .  # also optionally follow fairseq README for apex installation for fp16 training.
export MKL_THREADING_LAYER=GNU  # fairseq may need this for numpy.

cd examples/MMPT  # MMPT can be in any folder, not necessarily under fairseq/examples.
pip install -e .
```

--------------------------------

### Install Fairseq Locally (Bash)

Source: https://github.com/facebookresearch/fairseq/blob/main/README.md

Provides bash commands to clone the fairseq repository and install it locally using pip. This is useful for development and making changes to the library. It includes a specific command for macOS users with CFLAGS. Also shows how to install the latest stable release.

```bash
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./ 

# on MacOS:
# CFLAGS="-stdlib=libc++" pip install --editable ./

# to install the latest stable release (0.10.x)
# pip install fairseq
```

--------------------------------

### Install Apex for Faster Training (Bash)

Source: https://github.com/facebookresearch/fairseq/blob/main/README.md

Commands to clone and install the NVIDIA Apex library, which is recommended for faster training with fairseq, especially when using mixed precision. It involves cloning the repository, navigating into it, and running a pip install command with specific global options for C++ and CUDA extensions.

```bash
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" \
  --global-option="--deprecated_fused_adam" --global-option="--xentropy" \
  --global-option="--fast_multihead_attn" ./
```

--------------------------------

### Install Intel MKL 2020 using apt

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/asr/tutorial/MMS_ASR_Inference_Colab.ipynb

Installs Intel MKL 2020 by downloading GPG keys, adding Intel's MKL repository to apt sources, updating package lists, and installing the intel-mkl-64bit package. This process requires root privileges and is configured for non-interactive installation.

```bash
! cd /tmp && wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB && \
    apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
! sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' && \
    apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
```

--------------------------------

### Install soundfile Library

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/wav2vec/README.md

Installs the soundfile library, which is a dependency for preparing audio training data manifests. This is a prerequisite step before running the data preparation script.

```shell
pip install soundfile
```

--------------------------------

### Setup Fairseq and Dependencies

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/paraphraser/README.md

Installs the Fairseq library and essential dependencies like sacremoses and sentencepiece, which are required for natural language processing tasks such as translation and tokenization.

```bash
git clone https://github.com/pytorch/fairseq.git
cd fairseq
pip install --editable .
pip install sacremoses sentencepiece
```

--------------------------------

### Multi-node Training with DDP

Source: https://context7.com/facebookresearch/fairseq/llms.txt

This shows how to configure multi-node training using Distributed Data Parallel (DDP). Each node requires specifying its rank, world size, and the initialization method, typically a TCP-based connection. The example illustrates the setup for Node 0 and Node 1.

```bash
# Node 0:
fairseq-train data-bin/wmt14.en-de \
    --distributed-world-size 8 \
    --distributed-rank 0 \
    --distributed-port 12345 \
    --distributed-init-method 'tcp://node0.example.com:12345' \
    ...
```

```bash
# Node 1:
fairseq-train data-bin/wmt14.en-de \
    --distributed-world-size 8 \
    --distributed-rank 4 \
    --distributed-port 12345 \
    --distributed-init-method 'tcp://node0.example.com:12345' \
    ...
```

--------------------------------

### Install Dependencies for LM Decoding with Flashlight

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/asr/tutorial/MMS_ASR_Inference_Colab.ipynb

This snippet installs necessary system dependencies for speech recognition decoding using the Flashlight library. It includes FFTW3, libsndfile, Google Glog, OpenMPI, and Boost. It also compiles and installs KenLM from source, which is required for language model integration.

```bash
# Taken from https://github.com/flashlight/flashlight/blob/main/scripts/colab/colab_install_deps.sh 
# Install dependencies from apt
! sudo apt-get install -y libfftw3-dev libsndfile1-dev libgoogle-glog-dev libopenmpi-dev libboost-all-dev
# Install Kenlm
! cd /tmp && git clone https://github.com/kpu/kenlm && cd kenlm && mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release && make install -j$(nproc)
```

--------------------------------

### Example manifest.json Structure

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/data_prep/README.md

Displays the beginning of a manifest.json file generated by the alignment script. Each entry contains information about a segmented audio file, including start time, filepath, duration, original text, normalized text, and uroman tokens.

```json
> head /path/to/output/manifest.json 

 {"audio_start_sec": 0.0, "audio_filepath": "/path/to/output/segment1.flac", "duration": 6.8, "text": "she wondered afterwards how she could have spoken with that hard serenity how she could have", "normalized_text": "she wondered afterwards how she could have spoken with that hard serenity how she could have", "uroman_tokens": "s h e w o n d e r e d a f t e r w a r d s h o w s h e c o u l d h a v e s p o k e n w i t h t h a t h a r d s e r e n i t y h o w s h e c o u l d h a v e"}
{"audio_start_sec": 6.8, "audio_filepath": "/path/to/output/segment2.flac", "duration": 5.3, "text": "gone steadily on with story after story poem after poem till", "normalized_text": "gone steadily on with story after story poem after poem till", "uroman_tokens": "g o n e s t e a d i l y o n w i t h s t o r y a f t e r s t o r y p o e m a f t e r p o e m t i l l"}
{"audio_start_sec": 12.1, "audio_filepath": "/path/to/output/segment3.flac", "duration": 5.9, "text": "allan's grip on her hands relaxed and he fell into a heavy tired sleep", "normalized_text": "allan's grip on her hands relaxed and he fell into a heavy tired sleep", "uroman_tokens": "a l l a n ' s g r i p o n h e r h a n d s r e l a x e d a n d h e f e l l i n t o a h e a v y t i r e d s l e e p"}
```

--------------------------------

### Fairseq Python Training API Setup

Source: https://context7.com/facebookresearch/fairseq/llms.txt

Demonstrates the setup for training models programmatically using Fairseq's Python API. It covers parsing arguments, setting up the task, building the model and criterion, initializing the trainer, and loading checkpoints for resuming training.

```python
#!/usr/bin/env python3
import logging
import torch
from fairseq import checkpoint_utils, options, tasks, utils
from fairseq.dataclass.utils import convert_namespace_to_omegaconf
from fairseq.trainer import Trainer
from omegaconf import DictConfig

# Parse command-line arguments or create config
parser = options.get_training_parser()
args = options.parse_args_and_arch(parser)
cfg = convert_namespace_to_omegaconf(args)

# Set random seed for reproducibility
utils.set_torch_seed(cfg.common.seed)

# Setup task (translation, language modeling, etc.)
task = tasks.setup_task(cfg.task)

# Build model and criterion
model = task.build_model(cfg.model)
criterion = task.build_criterion(cfg.criterion)

print(f"Model: {model.__class__.__name__}")
print(f"Number of parameters: {sum(p.numel() for p in model.parameters()):,}")

# Initialize trainer
trainer = Trainer(cfg, task, model, criterion)

# Load checkpoint if resuming
extra_state, epoch_itr = checkpoint_utils.load_checkpoint(
    cfg.checkpoint,
    trainer,
    disable_iterator_cache=False
)

```

--------------------------------

### Install Preprocessing Dependencies

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/pay_less_attention_paper/README.md

Installs necessary Python packages for data preprocessing, such as sacremoses and subword_nmt.

```bash
pip install sacremoses subword_nmt
```

--------------------------------

### Configure Multiple Waveform Augmentations

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md

This example illustrates how to configure multiple audio waveform augmentation transforms simultaneously. It includes `musicaugment`, `backgroundnoiseaugment`, and `sporadicnoiseaugment`, each with their specific parameters, and lists them under `waveform_transforms` for training.

```yaml
musicaugment:
  samples_path: ${MUSIC_PATH}
  snr_min: 5 
  snr_max: 20
  rate: 0.25
backgroundnoiseaugment:
  samples_path: ${NOISES_PATH}
  snr_min: 10
  snr_max: 20
  rate: 0.1sporadicnoiseaugment:
  samples_path: ${NOISES_PATH}
  snr_min: 5
  snr_max: 15
  rate: 0.1
  noise_rate: 0.25
waveform_transforms:
  _train:
  - musicaugment
  - backgroundnoiseaugment
  - sporadicnoiseaugment
```

--------------------------------

### Download and Set Up Dataset Directory

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/multilingual/data_scripts/README.md

Sets an environment variable 'WORKDIR_ROOT' to define the main directory for all working files, including the downloaded dataset. The dataset will be located at '$WORKDIR_ROOT/ML50'.

```bash
export WORKDIR_ROOT=<a directory which will hold all working files>

```

--------------------------------

### Run MMS Inference (LM Decoding)

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/asr/tutorial/MMS_ASR_Inference_Colab.ipynb

This section demonstrates how to perform MMS inference with Language Model (LM) decoding. It's noted that lmweight and wordscore parameters often require tuning for optimal performance with specific LMs. The setup involves similar environment variable configurations as the greedy decoding example.

```shell
# Note that the lmweight, wordscore needs to tuned for each LM 
print("\n\n\n======= WITH LM DECODING=======")

# Assuming the command for LM decoding would follow a similar pattern to greedy decoding, 
# potentially with additional arguments for LM integration. 
# Example placeholder (actual command might differ based on fairseq's infer script): 
# !python examples/mms/asr/infer/mms_infer.py --model "/content/fairseq/models_new/mms1b_fl102.pt" --lang "eng" --audio "/content/fairseq/audio_samples/audio.wav" "/content/fairseq/audio_samples/audio_noisy.wav" --use-lm --lm-weight <weight> --word-score <score>
```

--------------------------------

### Install Dependencies for Preprocessing

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/language_model/README.md

Installs necessary Python packages for data preprocessing, including fastBPE and sacremoses, which are often required for text tokenization and BPE encoding in language modeling tasks.

```bash
pip install fastBPE sacremoses
```

--------------------------------

### Install FAISS for Retrieval Model Pretraining

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/MMPT/pretraining.md

This command installs the FAISS library, which is a dependency for the retrieval model pretraining described in the documentation. FAISS is used for efficient similarity search and clustering of dense vectors. It is recommended to use conda for installation.

```shell
conda install faiss-cpu -c pytorch
```

--------------------------------

### Console: Fairseq Generation Example

Source: https://github.com/facebookresearch/fairseq/blob/main/docs/tutorial_simple_lstm.rst

Demonstrates a typical command-line invocation of the fairseq-generate script for translating sentences. It shows input parameters like the data bin path, model checkpoint, beam size, and BPE removal, along with example output metrics like BLEU score and generation speed.

```console
> fairseq-generate data-bin/iwslt14.tokenized.de-en \
    --path checkpoints/checkpoint_best.pt \
    --beam 5 \
    --remove-bpe
(...)
| Translated 6750 sentences (153132 tokens) in 17.3s (389.12 sentences/s, 8827.68 tokens/s)
| Generate test with beam=5: BLEU4 = 8.18, 38.8/12.1/4.7/2.0 (BP=1.000, ratio=1.066, syslen=139865, reflen=131146)
```

--------------------------------

### Wav2vec 2.0 Inference and Fine-tuning with Transformers (Python)

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/wav2vec/README.md

Example demonstrating how to load a pre-trained Wav2vec 2.0 model from Hugging Face Transformers, perform inference on an audio file, and compute the loss for fine-tuning.

```python
# !pip install transformers
# !pip install datasets
import soundfile as sf
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

# load pretrained model
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")


librispeech_samples_ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")

# load audio
audio_input, sample_rate = sf.read(librispeech_samples_ds[0]["file"])

# pad input values and return pt tensor
input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values

# INFERENCE

# retrieve logits & take argmax
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)

# transcribe
transcription = processor.decode(predicted_ids[0])

# FINE-TUNE

target_transcription = "A MAN SAID TO THE UNIVERSE I EXIST"

# encode labels
with processor.as_target_processor():
  labels = processor(target_transcription, return_tensors="pt").input_ids

# compute loss by passing labels
loss = model(input_values, labels=labels).loss
loss.backward()
```

--------------------------------

### Install Fairseq Dependencies

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/translation/README.md

Installs necessary Python packages for preprocessing text data when using Fairseq models, such as fastBPE, sacremoses, and subword_nmt.

```bash
pip install fastBPE sacremoses subword_nmt
```

--------------------------------

### Install spaCy and Download Language Model

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/roberta/README.md

Provides the bash commands to install the spaCy library and download the large English language model, which is a prerequisite for using the pronoun disambiguation feature.

```bash
pip install spacy
python -m spacy download en_core_web_lg
```

--------------------------------

### Install PyArrow for Large Datasets (Bash)

Source: https://github.com/facebookresearch/fairseq/blob/main/README.md

A simple pip command to install the PyArrow library, which is recommended for handling large datasets with fairseq. This facilitates efficient data loading and processing.

```bash
pip install pyarrow
```

--------------------------------

### Install Required Packages for fairseq

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/textless_nlp/pgslm/README.md

Installs essential Python packages required for the project, including AMFM-decompy, SoundFile, scipy, sklearn, torchaudio, and npy-append-array, in addition to the base fairseq library.

```bash
pip install AMFM-decompy SoundFile scipy sklearn torchaudio npy-append-array
```

--------------------------------

### Download and Process Audio Sample

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/asr/tutorial/MMS_ASR_Inference_Colab.ipynb

This snippet downloads an audio file using wget and then processes it using FFmpeg to resample it to 16000 Hz. It saves the processed audio to './audio_samples/audio_noisy.wav'. Ensure FFmpeg is installed and accessible in your environment.

```shell
!wget -O ./audio_samples/tmp.wav 'https://datasets-server.huggingface.co/assets/MLCommons/peoples_speech/--/dirty/train/0/audio/audio.wav'
!ffmpeg -y -i ./audio_samples/tmp.wav -ar 16000 ./audio_samples/audio_noisy.wav
```

--------------------------------

### Distributed Training on SLURM Clusters

Source: https://github.com/facebookresearch/fairseq/blob/main/docs/getting_started.rst

Example commands for launching distributed training on SLURM clusters. Fairseq automatically detects nodes and GPUs. A distributed port must be provided.

```console
> salloc --gpus=16 --nodes 2 (...)
> srun fairseq-train --distributed-port 12345 (...).
```

--------------------------------

### Load Translation Model with torch.hub (Python)

Source: https://github.com/facebookresearch/fairseq/blob/main/README.md

Demonstrates how to load a pre-trained English to German translation model from PyTorch Hub using the fairseq library. It shows a simple translation of a given text. This requires PyTorch and the fairseq library to be installed.

```python
en2de = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-de.single_model')
en2de.translate('Hello world', beam=5)
# 'Hallo Welt'
```

--------------------------------

### Mine Pseudo-Parallel Data between Kazakh and English

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/criss/README.md

Mines pseudo-parallel data between Kazakh and English. This process is crucial for iterative self-supervised training and relies on the faiss library for efficient nearest neighbor search. Refer to faiss installation guide for setup.

```bash
bash mining/mine_example.sh
```

--------------------------------

### Install Python Packages for S2T Data Processing and Training

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_text/docs/librispeech_example.md

Installs necessary Python packages such as pandas, torchaudio, and sentencepiece, which are required for speech-to-text data processing and model training. This step ensures all dependencies are met before proceeding with data preparation.

```bash
# additional Python packages for S2T data processing/model training
pip install pandas torchaudio sentencepiece
```

--------------------------------

### Download and Prepare Models for Reranking Example

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/noisychannel/README.md

This snippet demonstrates how to download and extract pre-trained models and test data required for the reranking example. It uses `curl` to fetch compressed archives and `tar` to extract them into a specified directory. Ensure `curl` and `tar` are installed.

```bash
mkdir rerank_example
curl https://dl.fbaipublicfiles.com/fairseq/models/noisychannel/forward_de2en.tar.bz2 | tar xvjf - -C rerank_example
curl https://dl.fbaipublicfiles.com/fairseq/models/noisychannel/backward_en2de.tar.bz2 | tar xvjf - -C rerank_example
curl https://dl.fbaipublicfiles.com/fairseq/models/noisychannel/reranking_en_lm.tar.bz2 | tar xvjf - -C rerank_example
curl https://dl.fbaipublicfiles.com/fairseq/models/noisychannel/wmt17test.tar.bz2 | tar xvjf - -C rerank_example
```

--------------------------------

### Initializing and Configuring a Fairseq Transform

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md

Shows the implementation of the from_config_dict class method to instantiate a transform from configuration settings and the __init__ method to set up instance attributes, including default hyperparameter values.

```python
    @classmethod
    def from_config_dict(cls, config=None):
        _config = {} if config is None else config
        return ConcatAugment(
            _config.get("rate", _DEFAULTS["rate"]),
            _config.get("max_tokens", _DEFAULTS["max_tokens"]),
            _config.get("attempts", _DEFAULTS["attempts"]),
        )

    def __init__(
        self,
        rate=_DEFAULTS["rate"],
        max_tokens=_DEFAULTS["max_tokens"],
        attempts=_DEFAULTS["attempts"],
    ):
        self.rate, self.max_tokens, self.attempts = rate, max_tokens, attempts
```

--------------------------------

### Train RoBERTa Base Model

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/roberta/README.pretraining.md

Initiates the pretraining of a RoBERTa base model using Fairseq's Hydra configuration system. It specifies the data directory and the pretraining configuration. The example assumes a distributed training setup on multiple GPUs.

```bash
DATA_DIR=data-bin/wikitext-103

fairseq-hydra-train -m --config-dir examples/roberta/config/pretraining \
--config-name base task.data=$DATA_DIR
```

--------------------------------

### Train Wav2Vec2 with Command Line Arguments

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/wav2vec/README.md

This command initiates the training of a Wav2Vec2 model using command-line arguments. It allows fine-grained control over various training parameters such as data path, save directory, optimizer, learning rate, and model architecture. Note that this method has a known issue referenced in the provided link.

```bash
OMP_NUM_THREADS=1 python train.py /manifest/path --save-dir /model/path --num-workers 6 --fp16 --max-update 400000 --save-interval 1 --no-epoch-checkpoints \
--arch wav2vec2 --task audio_pretraining --min-lr 1e-06 --stop-min-lr 1e-09 --optimizer adam --lr 0.005 --lr-scheduler cosine \
--conv-feature-layers [(512, 10, 5), (512, 8, 4), (512, 4, 2), (512, 4, 2), (512, 4, 2), (512, 1, 1), (512, 1, 1)] \
--conv-aggregator-layers [(512, 2, 1), (512, 3, 1), (512, 4, 1), (512, 5, 1), (512, 6, 1), (512, 7, 1), (512, 8, 1), (512, 9, 1), (512, 10, 1), (512, 11, 1), (512, 12, 1), (512, 13, 1)] \
--skip-connections-agg --residual-scale 0.5 --log-compression --warmup-updates 500 --warmup-init-lr 1e-07 --criterion wav2vec --num-negatives 10 \
--max-sample-size 150000 --max-tokens 1500000 --skip-invalid-size-inputs-valid-test \
--tpu --distributed-world-size 8 --num-batch-buckets 3 --enable-padding \
--encoder-layerdrop 0 --mask-channel-prob 0.1
```

--------------------------------

### Load and Use Pre-trained English-German Translation Model with Torch Hub

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/backtranslation/README.md

This snippet demonstrates how to load a pre-trained English-to-German translation model ensemble from Fairseq using PyTorch Hub. It requires installing 'subword_nmt' and 'sacremoses'. The example shows listing available models, loading the 'transformer.wmt18.en-de' ensemble, checking the number of models in the ensemble, and performing a sample translation.

```python
import torch

# List available models
torch.hub.list('pytorch/fairseq')  # [..., 'transformer.wmt18.en-de', ... ]

# Load the WMT'18 En-De ensemble
en2de_ensemble = torch.hub.load(
    'pytorch/fairseq', 'transformer.wmt18.en-de',
    checkpoint_file='wmt18.model1.pt:wmt18.model2.pt:wmt18.model3.pt:wmt18.model4.pt:wmt18.model5.pt',
    tokenizer='moses', bpe='subword_nmt')

# The ensemble contains 5 models
len(en2de_ensemble.models)
# 5

# Translate
en2de_ensemble.translate('Hello world!')
# 'Hallo Welt!'
```

--------------------------------

### Integrate Quant-Noise and iPQ with Your Model (Python)

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/quant_noise/README.md

Example Python code demonstrating how to integrate Quant-Noise and iPQ into your own models. This involves wrapping modules with `quant_noise` and then using `quantize_model_` to quantize the trained model. The process allows quantization without modifying the training loop, although the optimizer may need to be re-created.

```python
from fairseq.modules.quantization.pq import quantize_model_, SizeTracker

# get configuration parameters
n_centroids_config = config["n_centroids"]
block_sizes_config = config["block_sizes"]
layers_to_quantize = config["layers_to_quantize"]

# size tracker for keeping track of assignments, centroids and non-compressed sizes
size_tracker = SizeTracker(model)

# Quantize model by stages
for step in range(len(layers_to_quantize)):

    # quantize model in-place
    quantized_layers = quantize_model_(
        model,
        size_tracker,
        layers_to_quantize,
        block_sizes_config,
        n_centroids_config,
        step=step,
    )
    logger.info(f"Finetuning stage {step}, quantized layers: {quantized_layers}")
    logger.info(f"{size_tracker}")

    # Don't forget to re-create/update trainer/optimizer since model parameters have changed
    optimizer = ...

    # Finetune the centroids with your usual training loop for a few epochs
    trainer.train_epoch()
```

--------------------------------

### Install Light/Dynamic Convolution CUDA Kernels

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/pay_less_attention_paper/README.md

Commands to install the memory-efficient CUDA kernels for Light and Dynamic convolutions. After installation, these kernels will be automatically used by Fairseq.

```shell
# to install lightconv
cd fairseq/modules/lightconv_layer
python cuda_function_gen.py
python setup.py install

# to install dynamicconv
cd fairseq/modules/dynamicconv_layer
python cuda_function_gen.py
python setup.py install
```

--------------------------------

### Download and Load RoBERTa Model in Fairseq (Python)

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/shuffled_word_order/README.md

Demonstrates how to download a pre-trained RoBERTa model checkpoint and its associated dictionary files, then load it using Fairseq's RobertaModel class for evaluation or fine-tuning. This requires the 'wget' command-line tool and the fairseq library.

```python
from fairseq.models.roberta import RobertaModel

# Download roberta.base.shuffle.n1 model
wget https://dl.fbaipublicfiles.com/unnatural_pretraining/roberta.base.shuffle.n1.tar.gz
tar -xzvf roberta.base.shuffle.n1.tar.gz
# Copy the dictionary files
cd roberta.base.shuffle.n1.tar.gz
wget -O dict.txt https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt && wget -O encoder.json https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/encoder.json && wget -O vocab.bpe https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/vocab.bpe
cd ..

# Load the model in fairseq
roberta = RobertaModel.from_pretrained('/path/to/roberta.base.shuffle.n1', checkpoint_file='model.pt')
roberta.eval()  # disable dropout (or leave in train mode to finetune)
```

--------------------------------

### Train 13B GPT-3 LM on 8 V100 GPUs with FSDP and CPU Offload

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/fully_sharded_data_parallel/README.md

This bash script demonstrates how to train a 13B parameter GPT-3 language model using Fairseq. It leverages FSDP with full sharding and CPU offloading across 8 V100 GPUs, optimizing for both memory and speed. Key parameters include data input, distributed training backend, mixed precision, checkpointing, model architecture, optimizer settings, and learning rate scheduling.

```bash
OMP_NUM_THREADS=20 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
    fairseq-train data-bin/wikitext-103-roberta-bpe-bin \
    --ddp-backend fully_sharded --fp16 --fp16-init-scale 4 \
    --cpu-offload --checkpoint-activations \
    --task language_modeling --tokens-per-sample 2048 --batch-size 8 \
    --arch transformer_lm_gpt3_13 \
    --optimizer cpu_adam --adam-betas "(0.9,0.98)" \
    --lr 0.0001 --lr-scheduler polynomial_decay --warmup-updates 5 --total-num-update 10 \
    --max-update 10 --no-save --log-format json --log-interval 1
```

--------------------------------

### Prepare Wav2vec Training Data Manifest (Shell)

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/wav2vec/README.md

Generates a manifest file for Wav2vec pre-training from a directory of WAV audio files. Specifies the destination directory and file extension.

```shell
python examples/wav2vec/wav2vec_manifest.py /path/to/waves --dest /manifest/path --ext wav
```

--------------------------------

### Install VITS Dependencies

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/tts/tutorial/MMS_TTS_Inference_Colab.ipynb

Installs essential Python packages for the VITS project using pip. This includes specific versions of Cython, librosa, and phonemizer, along with general packages like scipy, numpy, torch, torchvision, matplotlib, and Unidecode. Ensure these packages are installed before proceeding with other VITS functionalities.

```bash
%pwd
!git clone https://github.com/jaywalnut310/vits.git
!python --version
%cd vits/

!pip install Cython==0.29.21
!pip install librosa==0.8.0
!pip install phonemizer==2.2.1
!pip install scipy
!pip install numpy
!pip install torch
!pip install torchvision
!pip install matplotlib
!pip install Unidecode==1.1.1

%cd monotonic_align/
%mkdir monotonic_align
!python3 setup.py build_ext --inplace
%cd ../
%pwd
```

--------------------------------

### Change Directory and Install Fairseq

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/mms/lid/tutorial/MMS_LID_Inference_Colab.ipynb

This snippet demonstrates how to change the current working directory to the Fairseq project directory and then install the project in editable mode using pip. It also installs the tensorboardX library, which is often used for visualization in machine learning projects.

```bash
!pwd
%cd "/content/fairseq"
!pip install --editable ./ 
!pip install tensorboardX
```

--------------------------------

### Loading Custom Architectures with --user-dir

Source: https://github.com/facebookresearch/fairseq/blob/main/docs/overview.rst

Shows how to load custom model architectures defined in a separate directory using the --user-dir flag. This requires an __init__.py file in the custom module that registers the new architecture.

```python
from fairseq.models import register_model_architecture
from fairseq.models.transformer import transformer_vaswani_wmt_en_de_big

@register_model_architecture('transformer', 'my_transformer')
def transformer_mmt_big(args):
    transformer_vaswani_wmt_en_de_big(args)
```

--------------------------------

### Train Wav2vec Model with CLI Tools (Shell)

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/wav2vec/README.md

Command to initiate Wav2vec model training using Fairseq's CLI tools. Configures various training parameters including optimizer, learning rate schedule, and model architecture.

```shell
$ python train.py /manifest/path --save-dir /model/path --num-workers 6 --fp16 --max-update 400000 --save-interval 1 --no-epoch-checkpoints \
--arch wav2vec --task audio_pretraining --min-lr 1e-06 --stop-min-lr 1e-09 --optimizer adam --lr 0.005 --lr-scheduler cosine \
--conv-feature-layers [(512, 10, 5), (512, 8, 4), (512, 4, 2), (512, 4, 2), (512, 4, 2), (512, 1, 1), (512, 1, 1)] \
--conv-aggregator-layers [(512, 2, 1), (512, 3, 1), (512, 4, 1), (512, 5, 1), (512, 6, 1), (512, 7, 1), (512, 8, 1), (512, 9, 1), (512, 10, 1), (512, 11, 1), (512, 12, 1), (512, 13, 1)] \
--skip-connections-agg --residual-scale 0.5 --log-compression --warmup-updates 500 --warmup-init-lr 1e-07 --criterion wav2vec --num-negatives 10 \
--max-sample-size 150000 --max-tokens 1500000 --skip-invalid-size-inputs-valid-test
```

--------------------------------

### Train 13B GPT-3 Model with CPU Offloading on Fairseq

Source: https://github.com/facebookresearch/fairseq/blob/main/examples/fully_sharded_data_parallel/README.md

This bash command initiates the training of a 13 billion parameter GPT-3 model using fairseq. It utilizes CPU offloading for parameters and optimizer states, gradient checkpointing, and mixed precision training. The command specifies data sources, model architecture, optimizer settings, and training parameters like learning rate, batch size, and update frequency. It requires fairseq and deepspeed to be installed.

```bash
OMP_NUM_THREADS=20 CUDA_VISIBLE_DEVICES=0 \
    fairseq-train data-bin/wikitext-103-roberta-bpe-bin \
    --ddp-backend fully_sharded --fp16 --fp16-init-scale 4 \
    --cpu-offload --checkpoint-activations \
    --task language_modeling --tokens-per-sample 2048 --batch-size 8 \
    --arch transformer_lm_gpt3_13 \
    --optimizer cpu_adam --adam-betas "(0.9,0.98)" \
    --lr 0.0001 --lr-scheduler polynomial_decay --warmup-updates 5 --total-num-update 10 \
    --max-update 10 --no-save --log-format json --log-interval 1
```