### Install TorchScale

Source: https://context7.com/microsoft/torchscale/llms.txt

Install TorchScale from PyPI or from source for local development. Optional installations for Flash Attention and xFormers are also provided for optimized performance on compatible GPUs.

```bash
pip install torchscale
```

```bash
git clone https://github.com/microsoft/torchscale.git
cd torchscale
pip install -e .
```

```bash
pip install flash-attn
```

```bash
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118
```

--------------------------------

### Install TorchScale

Source: https://github.com/microsoft/torchscale/blob/main/README.md

Install the torchscale library using pip. For development, clone the repository and install in editable mode.

```bash
pip install torchscale
```

```bash
git clone https://github.com/microsoft/torchscale.git
cd torchscale
pip install -e .
```

--------------------------------

### TorchScale BERT Pretraining Setup with FairSeq

Source: https://context7.com/microsoft/torchscale/llms.txt

Installs TorchScale and necessary dependencies, including FairSeq and Infinibatch, for training BERT-style masked language models.

```bash
# Setup
git clone https://github.com/microsoft/torchscale.git
cd torchscale && pip install -e .
pip install git+https://github.com/shumingma/fairseq.git@moe
pip install git+https://github.com/shumingma/infinibatch.git
pip install iopath numpy==1.23.0
```

--------------------------------

### Install TorchScale and FairSeq Dependencies

Source: https://github.com/microsoft/torchscale/blob/main/examples/fairseq/README.md

Install the torchscale repository and necessary dependencies, including a specific version of fairseq and infinibatch, for integration.

```bash
git clone https://github.com/microsoft/torchscale.git
cd torchscale
pip install -e .
pip install git+https://github.com/shumingma/fairseq.git@moe
pip install git+https://github.com/shumingma/infinibatch.git
pip install iopath
pip install numpy==1.23.0
```

--------------------------------

### Install Flash Attention or xFormers

Source: https://github.com/microsoft/torchscale/blob/main/README.md

For faster training on compatible GPUs, install Flash Attention or xFormers. Ensure you select the correct version for your CUDA installation.

```bash
pip install flash-attn
```

```bash
# cuda 11.8 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118
# cuda 12.1 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121
```

--------------------------------

### Install Dependencies for LongViT

Source: https://github.com/microsoft/torchscale/blob/main/examples/longvit/README.md

Installs necessary packages including fairseq, xformers, and project-specific requirements.

```bash
pip install -r requirements.txt
pip install git+https://github.com/shumingma/fairseq.git@moe
pip install -v -U git+https://github.com/facebookresearch/xformers.git@v0.0.20#egg=xformers
```

--------------------------------

### EncoderDecoder training loop example

Source: https://context7.com/microsoft/torchscale/llms.txt

An example of a typical machine translation training loop using the EncoderDecoder model. It demonstrates calculating cross-entropy loss and backpropagation.

```python
import torch.nn.functional as F
optimizer = torch.optim.Adam(model.parameters(), lr=5e-4)

for src, tgt in dataloader:
    # tgt_input: all tokens except last
    # tgt_output: all tokens except first (labels)
    tgt_input = tgt[:, :-1]
    tgt_output = tgt[:, 1:]

    optimizer.zero_grad()
    logits, _ = model(src, tgt_input)
    loss = F.cross_entropy(logits.view(-1, 64000), tgt_output.view(-1))
    loss.backward()
    optimizer.step()
```

--------------------------------

### Fine-tune LongViT on TCGA Subtyping (Standard)

Source: https://github.com/microsoft/torchscale/blob/main/examples/longvit/get_started/get_started_for_tcga_subtyping.md

Command to fine-tune LongViT on TCGA subtyping for images up to 16,384x16,384. Adjust parameters like `IMAGE_SIZE`, `TASK`, `K_FOLD`, and paths according to your setup. The `--finetune` argument points to pretrained model weights.

```bash
# IMAGE_SIZE - {1024, 4096, 8192, 16384}
# TASK - {"brca", "kidney", "lung"}  
# K_FOLD - {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}   
python -m torch.distributed.launch --nproc_per_node=8 run_longvit_finetuning.py \
        --input_size ${IMAGE_SIZE} \
        --model longvit_small_patch32_${IMAGE_SIZE} \
        --task tcga_${TASK}_subtyping \
        --batch_size 1 \
        --layer_decay 1.0 \
        --lr 5e-5 \
        --update_freq 1 \
        --epochs 10 \
        --warmup_epochs 1 \
        --drop_path 0.1 \
        --finetune /your_longvit_model_path/longvit_small_patch32_1024.pth
        --data_path ./subtyping_split_index/tcga_${TASK} \
        --image_dir /path/to/your_resized_WSIs  \
        --output_dir /path/to/save/your_model \
        --log_dir /path/to/save/your_model/log \
        --weight_decay 0.05 \
        --seed 42 \
        --save_ckpt_freq 5 \
        --k_fold ${K_FOLD} \
        --num_workers 1 \
        --enable_deepspeed \
        --model_key teacher \
        --randaug
```

--------------------------------

### Configure LongNet for Extremely Long Sequences

Source: https://context7.com/microsoft/torchscale/llms.txt

Example configuration for a LongNet decoder designed to handle sequences up to 1 million tokens, utilizing increased layers and segment lengths with corresponding dilation ratios.

```python
# Example configuration for extremely long sequences (1M tokens)
config_1m = DecoderConfig(
    vocab_size=64000,
    decoder_embed_dim=768,
    decoder_attention_heads=12,
    decoder_layers=24,
    flash_attention=True,
    segment_length='[2048,4096,8192,16384,32768]',
    dilated_ratio='[1,2,4,8,16]'
)
```

--------------------------------

### Run LongViT Finetuning

Source: https://github.com/microsoft/torchscale/blob/main/examples/longvit/get_started/get_started_for_tcga_survival_prediction.md

Use this command to initiate the finetuning process for LongViT models. Ensure all paths and parameters are correctly set for your specific task and data.

```bash
python -m torch.distributed.launch --nproc_per_node=1 run_longvit_finetuning.py \
        --input_size ${IMAGE_SIZE} \
        --model longvit_small_patch32_${IMAGE_SIZE} \
        --task tcga_${TASK}_survival \
        --batch_size 1 \
        --layer_decay 1.0 \
        --lr 5e-5 \
        --update_freq 1 \
        --epochs 10 \
        --warmup_epochs 1 \
        --drop_path 0.1 \
        --finetune /path/to/save/your_model/checkpoint-best/mp_rank_00_model_states.pt \
        --data_path ./survival_split_index/tcga_${TASK} \
        --image_dir /path/to/your_resized_WSIs \
        --output_dir /path/to/save/your_model \
        --log_dir /path/to/save/your_model/log \
        --weight_decay 0.05 \
        --seed 42 \
        --save_ckpt_freq 5 \
        --k_fold ${K_FOLD} \
        --num_workers 1 \
        --enable_deepspeed \
        --model_key module \
        --eval \
        --no_auto_resume
```

--------------------------------

### Initialize and Use BEiT3 Model

Source: https://context7.com/microsoft/torchscale/llms.txt

Configure and initialize the BEiT3 multimodal model. Ensure 'multiway=True' is set in the configuration for its specific architecture.

```python
import torch
from torchscale.architecture.config import EncoderConfig
from torchscale.model.BEiT3 import BEiT3

# BEiT3 configuration (requires multiway=True)
config = EncoderConfig(
    vocab_size=64000,
    encoder_embed_dim=768,
    encoder_attention_heads=12,
    encoder_ffn_embed_dim=3072,
    encoder_layers=12,
    multiway=True,  # Required for BEiT3
    img_size=224,
    patch_size=16,
    in_chans=3,
    max_source_positions=512
)

# Initialize BEiT3 model
model = BEiT3(config)

batch_size = 2
```

--------------------------------

### Configure and Initialize an Encoder Model

Source: https://github.com/microsoft/torchscale/blob/main/README.md

Demonstrates how to configure an Encoder with specific features like DeepNorm and Multiway, and then initialize the model. Ensure necessary imports are present.

```python
>>> from torchscale.architecture.config import EncoderConfig
>>> from torchscale.architecture.encoder import Encoder

>>> config = EncoderConfig(vocab_size=64000, deepnorm=True, multiway=True)
>>> model = Encoder(config)

>>> print(model)
```

--------------------------------

### Get features only from EncoderDecoder

Source: https://context7.com/microsoft/torchscale/llms.txt

Extracts only the features from the EncoderDecoder model without applying the output projection to the vocabulary. Useful for intermediate feature extraction.

```python
output, extra = model(prev_output_tokens, features_only=True)
features = output  # Shape: (batch_size, seq_len, embed_dim)
```

--------------------------------

### Get Activation Functions

Source: https://context7.com/microsoft/torchscale/llms.txt

Retrieves pre-defined activation functions like ReLU, GELU, and Swish (SiLU) for use in neural network components.

```python
# Available activation functions
relu_fn = get_activation_fn("relu")
gelu_fn = get_activation_fn("gelu")
swish_fn = get_activation_fn("swish")  # Also known as SiLU
```

--------------------------------

### Launch BERT Pretraining with TorchScale and FairSeq

Source: https://github.com/microsoft/torchscale/blob/main/examples/fairseq/README.md

Execute the BERT pretraining script using torch.distributed.launch, specifying data paths, model architecture, and training parameters.

```bash
cd examples/fairseq/
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=8 train.py ${PATH_TO_DATA} \
        --task pretraining  \
        --tokens-per-sample 512  \
        --mask-prob 0.15  \
        --span-length 3.0  \
        --leave-unmasked-prob 0.0  \
        --random-token-prob 0.0 \
        --criterion masked_lm  \
        --arch mlm_base  \
        --share-encoder-input-output-embed \
        --required-batch-size-multiple 8 \
        --spm-model ${PATH_TO_DATA}/sentencepiece.bpe.model \
        --dict-file ${PATH_TO_DATA}/dict.txt \
        --optimizer adam  \
        --adam-betas '(0.9,0.98)'  \
        --adam-eps 1e-6  \
        --clip-norm 2.0 \
        --lr-scheduler polynomial_decay  \
        --lr 0.0005  \
        --warmup-updates 10000  \
        --total-num-update 125000 \
        --max-update 125000 \
        --max-sentences 32  \
        --update-freq 1 \
        --log-format simple  \
        --log-interval 100 \
        --disable-validation \
        --save-interval-updates 5000 \
        --no-epoch-checkpoints \
        --fp16 \
        --fp16-init-scale 4 \
        --fp16-scale-window 256 \
        --min-loss-scale 0.0001 \
        --seed 1 \
        --save-dir ${PATH_TO_CKPT} \
        --ddp-backend=no_c10d \
        --distributed-no-spawn \
        --reset-dataloader \
        --batch-read-ahead 10000 \
        --rel-pos-buckets 32 \
        --max-rel-pos 128 \
        --deepnorm
```

--------------------------------

### Pretrain GPT with X-MoE

Source: https://context7.com/microsoft/torchscale/llms.txt

Command to pretrain a GPT model using X-MoE for sparse expert utilization. Includes MoE-specific parameters.

```bash
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 train.py ${PATH_TO_DATA} \
    --arch lm_base \
    --task language_modeling \
    --tokens-per-sample 128 \
    --optimizer adam --adam-betas "(0.9, 0.98)" \
    --lr 5e-4 --lr-scheduler polynomial_decay \
    --warmup-updates 750 \
    --batch-size 4 --max-update 50000 \
    --memory-efficient-fp16 \
    --moe-expert-count 2 --moe-freq 2 \
    --use-xmoe \
    --criterion moe_cross_entropy --moe-gate-loss-wt 0.01
```

--------------------------------

### Initialize MultiScaleRetention

Source: https://context7.com/microsoft/torchscale/llms.txt

Sets up the MultiScaleRetention component with a given configuration, defining dimensions, number of heads, and gate function. The `args` parameter should be a RetNetConfig object.

```python
# Create config
config = RetNetConfig(
    decoder_embed_dim=768,
    decoder_value_embed_dim=1280,
    decoder_retention_heads=4,
    layernorm_eps=1e-6,
    multiway=False,
    recurrent_chunk_size=512
)

# Initialize multi-scale retention
retention = MultiScaleRetention(
    args=config,
    embed_dim=768,
    value_dim=1280,
    num_heads=4,
    gate_fn="swish"
)
```

--------------------------------

### Initialize MultiheadAttention

Source: https://context7.com/microsoft/torchscale/llms.txt

Initialize the MultiheadAttention component with support for Flash Attention, relative position bias, and Xpos. SubLN is enabled for stability.

```python
import torch
from torchscale.architecture.config import EncoderConfig
from torchscale.component.multihead_attention import MultiheadAttention

# Create a config object
config = EncoderConfig(
    encoder_embed_dim=768,
    encoder_attention_heads=12,
    layernorm_eps=1e-5,
    multiway=False,
    xpos_rel_pos=False,
    flash_attention=False
)

# Initialize multi-head attention
mha = MultiheadAttention(
    args=config,
    embed_dim=768,
    num_heads=12,
    dropout=0.1,
    self_attention=True,
    encoder_decoder_attention=False,
    subln=True  # Enable SubLN for stability
)

# Self-attention forward pass
batch_size, seq_len, embed_dim = 2, 128, 768
x = torch.randn(batch_size, seq_len, embed_dim)

# Self-attention (query = key = value)
output, attn_weights = mha(
    query=x,
    key=x,
    value=x
)
# output shape: (batch_size, seq_len, embed_dim)
```

--------------------------------

### Sequence Parallel Training for Large Images

Source: https://github.com/microsoft/torchscale/blob/main/examples/longvit/get_started/get_started_for_tcga_subtyping.md

Placeholder for the command to perform sequence parallel training on 32,768x32,768 images. Specific parameters for `TASK` need to be defined.

```bash
# TASK - {"brca", "kidney", "lung"}  

```

--------------------------------

### Train Dense Model for GPT Pretraining

Source: https://github.com/microsoft/torchscale/blob/main/examples/fairseq/README.md

Command to train a dense language model for GPT pretraining. Requires data preprocessing and setting the data path.

```bash
cd examples/fairseq/\npython -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 train.py \     ${PATH_TO_DATA} \     --num-workers 2 \     --activation-fn gelu \     --share-decoder-input-output-embed \     --validate-interval-updates 1000 \     --save-interval-updates 1000 \     --no-epoch-checkpoints \     --memory-efficient-fp16 \     --fp16-init-scale 4 \     --arch lm_base \     --task language_modeling \     --sample-break-mode none \     --tokens-per-sample 128 \     --optimizer adam --adam-betas "(0.9, 0.98)" \     --adam-eps 1e-08 \     --clip-norm 0.0 \     --lr 5e-4 \     --lr-scheduler polynomial_decay \     --warmup-updates 750 \     --dropout 0.1 \     --attention-dropout 0.1 \     --weight-decay 0.01 \     --batch-size 4 \     --update-freq 1 \     --required-batch-size-multiple 1 \     --total-num-update 50000 \     --max-update 50000 \     --seed 1 \     --ddp-backend=c10d
```

--------------------------------

### Incremental decoding with RetNetDecoder (first step)

Source: https://context7.com/microsoft/torchscale/llms.txt

Demonstrates the first step of incremental decoding with the RetNetDecoder model in recurrent mode. This is the initial call for O(1) inference.

```python
# Incremental decoding (recurrent mode for inference with O(1) complexity)
incremental_state = {"is_first_step": True}

# First step
first_token = torch.randint(0, 64000, (batch_size, 1))
output, extra = model(first_token, incremental_state=incremental_state)
```

--------------------------------

### Initialize EncoderDecoderConfig

Source: https://context7.com/microsoft/torchscale/llms.txt

Use EncoderDecoderConfig for sequence-to-sequence models. Supports basic, large, and MoE configurations.

```python
from torchscale.architecture.config import EncoderDecoderConfig

# Basic encoder-decoder configuration
config = EncoderDecoderConfig(
    vocab_size=64000,
    encoder_embed_dim=512,
    encoder_attention_heads=8,
    encoder_ffn_embed_dim=2048,
    encoder_layers=6,
    decoder_embed_dim=512,
    decoder_attention_heads=8,
    decoder_ffn_embed_dim=2048,
    decoder_layers=6,
    share_all_embeddings=True,
    share_decoder_input_output_embed=True
)
```

```python
# Large encoder-decoder with DeepNorm
config_large = EncoderDecoderConfig(
    vocab_size=64000,
    encoder_layers=24,
    decoder_layers=24,
    encoder_embed_dim=1024,
    decoder_embed_dim=1024,
    deepnorm=True
)
```

```python
# Encoder-decoder with MoE
config_moe = EncoderDecoderConfig(
    vocab_size=64000,
    encoder_layers=12,
    decoder_layers=12,
    moe_freq=2,
    moe_expert_count=16,
    use_xmoe=True
)
```

--------------------------------

### Incremental decoding with EncoderDecoder

Source: https://context7.com/microsoft/torchscale/llms.txt

Demonstrates incremental decoding for generation tasks using the EncoderDecoder model. It shows the first step and subsequent steps for generating tokens.

```python
incremental_state = {"is_first_step": True}
# First step
first_token = torch.randint(0, 64000, (batch_size, 1))
output, extra = model(first_token, incremental_state=incremental_state)

# Subsequent steps (incremental)
incremental_state["is_first_step"] = False
for _ in range(10):
    next_token_logits = output[:, -1, :]  # Get last token logits
    next_token = torch.argmax(next_token_logits, dim=-1, keepdim=True)
    all_tokens = torch.cat([first_token, next_token], dim=1)
    output, extra = model(all_tokens, incremental_state=incremental_state)
    first_token = all_tokens
```

--------------------------------

### Pretrain Dense GPT

Source: https://context7.com/microsoft/torchscale/llms.txt

Command to pretrain a dense GPT-style causal language model. Uses standard language modeling task parameters.

```bash
cd examples/fairseq/
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 train.py ${PATH_TO_DATA} \
    --arch lm_base \
    --task language_modeling \
    --sample-break-mode none \
    --tokens-per-sample 128 \
    --share-decoder-input-output-embed \
    --optimizer adam --adam-betas "(0.9, 0.98)" --adam-eps 1e-08 \
    --lr 5e-4 --lr-scheduler polynomial_decay \
    --warmup-updates 750 \
    --dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
    --batch-size 4 \
    --total-num-update 50000 --max-update 50000 \
    --memory-efficient-fp16
```

--------------------------------

### Initialize FeedForwardNetwork with SubLN

Source: https://context7.com/microsoft/torchscale/llms.txt

Initializes a FeedForwardNetwork with SubLayer Normalization (SubLN) enabled for improved training stability. Common practice is to set ffn_dim to 4x embed_dim.

```python
ffn = FeedForwardNetwork(
    embed_dim=768,
    ffn_dim=3072,  # Usually 4x embed_dim
    activation_fn="gelu",  # Options: "relu", "gelu", "swish"
    dropout=0.1,
    activation_dropout=0.1,
    layernorm_eps=1e-5,
    subln=True  # Enable SubLN for improved stability
)

# Forward pass
batch_size, seq_len, embed_dim = 2, 128, 768
x = torch.randn(batch_size, seq_len, embed_dim)
output = ffn(x)  # Shape: (batch_size, seq_len, embed_dim)
```

--------------------------------

### Train Sparse (MoE) Model for GPT Pretraining

Source: https://github.com/microsoft/torchscale/blob/main/examples/fairseq/README.md

Command to train a sparse Mixture-of-Experts model for GPT pretraining. Note the MoE-specific arguments.

```bash
cd examples/fairseq/\npython -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 train.py \     ${PATH_TO_DATA} \     --num-workers 2 \     --activation-fn gelu \     --share-decoder-input-output-embed \     --validate-interval-updates 1000 \     --save-interval-updates 1000 \     --no-epoch-checkpoints \     --memory-efficient-fp16 \     --fp16-init-scale 4 \     --arch lm_base \     --task language_modeling \     --sample-break-mode none \     --tokens-per-sample 128 \     --optimizer adam --adam-betas "(0.9, 0.98)" \     --adam-eps 1e-08 \     --clip-norm 0.0 \     --lr 5e-4 \     --lr-scheduler polynomial_decay \     --warmup-updates 750 \     --dropout 0.1 \     --attention-dropout 0.1 \     --weight-decay 0.01 \     --batch-size 4 \     --update-freq 1 \     --required-batch-size-multiple 1 \     --total-num-update 50000 \     --max-update 50000 \     --seed 1 \     --ddp-backend=no_c10d \     --moe-expert-count 2 --moe-freq 2 \     --moe-gating-use-fp32 --moe-second-expert-policy random --moe-normalize-gate-prob-before-dropping \     --moe-eval-capacity-token-fraction -1.0 \     --criterion moe_cross_entropy --moe-gate-loss-wt 0.01 --moe-gate-loss-combine-method sum \     --use-xmoe
```

--------------------------------

### Decoder Configuration with X-MoE

Source: https://context7.com/microsoft/torchscale/llms.txt

Configures a decoder model with X-MoE for sparse scaling. Parameters include `moe_freq`, `moe_expert_count`, `use_xmoe`, and `moe_gating_use_fp32` for gating precision.

```python
from torchscale.architecture.config import DecoderConfig

# Decoder with X-MoE for sparse scaling
config_moe = DecoderConfig(
    vocab_size=64000,
    decoder_layers=12,
    moe_freq=2,
    moe_expert_count=32,
    use_xmoe=True,
    moe_gating_use_fp32=True
)
```

--------------------------------

### Initialize RetNetConfig

Source: https://context7.com/microsoft/torchscale/llms.txt

Use RetNetConfig for Retentive Networks. Supports basic, chunkwise recurrent, large, and MoE configurations.

```python
from torchscale.architecture.config import RetNetConfig

# Basic RetNet configuration
config = RetNetConfig(
    vocab_size=64000,
    decoder_embed_dim=768,
    decoder_value_embed_dim=1280,
    decoder_retention_heads=3,
    decoder_ffn_embed_dim=1280,
    decoder_layers=12,
    dropout=0.1
)
```

```python
# RetNet with chunkwise recurrent mode for efficient training
config_chunkwise = RetNetConfig(
    vocab_size=64000,
    decoder_embed_dim=768,
    decoder_retention_heads=4,
    decoder_layers=12,
    chunkwise_recurrent=True,
    recurrent_chunk_size=512
)
```

```python
# Large RetNet configuration
config_large = RetNetConfig(
    vocab_size=64000,
    decoder_embed_dim=2048,
    decoder_value_embed_dim=3456,
    decoder_retention_heads=8,
    decoder_ffn_embed_dim=3456,
    decoder_layers=24,
    subln=True
)
```

```python
# RetNet with MoE
config_moe = RetNetConfig(
    vocab_size=64000,
    decoder_layers=12,
    moe_freq=2,
    moe_expert_count=32,
    use_xmoe=True
)
```

--------------------------------

### Initialize and use EncoderDecoder model

Source: https://context7.com/microsoft/torchscale/llms.txt

Initializes an EncoderDecoder model with a specified configuration and performs a forward pass. Includes preparation of source and target sequences.

```python
import torch
from torchscale.architecture.config import EncoderDecoderConfig
from torchscale.architecture.encoder_decoder import EncoderDecoder

# Create encoder-decoder configuration
config = EncoderDecoderConfig(
    vocab_size=64000,
    encoder_embed_dim=512,
    encoder_attention_heads=8,
    encoder_ffn_embed_dim=2048,
    encoder_layers=6,
    decoder_embed_dim=512,
    decoder_attention_heads=8,
    decoder_ffn_embed_dim=2048,
    decoder_layers=6,
    share_all_embeddings=True
)

# Initialize encoder-decoder model
model = EncoderDecoder(config)

# Prepare inputs
batch_size = 2
src_len, tgt_len = 64, 32
src_tokens = torch.randint(0, 64000, (batch_size, src_len))  # Source sequence
prev_output_tokens = torch.randint(0, 64000, (batch_size, tgt_len))  # Target sequence (shifted right)

# Forward pass
decoder_out = model(src_tokens, prev_output_tokens)
logits = decoder_out[0]  # Shape: (batch_size, tgt_len, vocab_size)
extra = decoder_out[1]
```

--------------------------------

### Create Decoder and Encoder-Decoder Models

Source: https://github.com/microsoft/torchscale/blob/main/README.md

Instantiate Decoder and EncoderDecoder models using their respective configuration classes and model implementations from torchscale.architecture.

```python
# Creating a decoder model
from torchscale.architecture.config import DecoderConfig
from torchscale.architecture.decoder import Decoder

config = DecoderConfig(vocab_size=64000)
decoder = Decoder(config)
print(decoder)

# Creating a encoder-decoder model
from torchscale.architecture.config import EncoderDecoderConfig
from torchscale.architecture.encoder_decoder import EncoderDecoder

config = EncoderDecoderConfig(vocab_size=64000)
encdec = EncoderDecoder(config)
print(encdec)
```

--------------------------------

### Initialize a RetNetDecoder Model

Source: https://github.com/microsoft/torchscale/blob/main/README.md

Shows the basic initialization of a RetNetDecoder model using RetNetConfig. This is useful for setting up a Transformer successor for large language models.

```python
config = RetNetConfig(vocab_size=64000)
retnet = RetNetDecoder(config)
```

--------------------------------

### Create LongNet Encoder and Decoder Models

Source: https://github.com/microsoft/torchscale/blob/main/README.md

Instantiate LongNet encoder and decoder models with specified segment lengths and dilation ratios. Requires Flash Attention. Ensure segment_length and dilated_ratio are passed as strings representing lists.

```python
import torch
from torchscale.architecture.config import EncoderConfig, DecoderConfig
from torchscale.model.longnet import LongNetEncoder, LongNetDecoder

# Creating a LongNet encoder with the dilated pattern of segment_length=[2048,4096] and dilated_ratio=[1,2]
config = EncoderConfig(vocab_size=64000, segment_length='[2048,4096]', dilated_ratio='[1,2]', flash_attention=True)
longnet = LongNetEncoder(config)

# Creating a LongNet decoder with the dilated pattern of segment_length=[2048,4096] and dilated_ratio=[1,2]
config = DecoderConfig(vocab_size=64000, segment_length='[2048,4096]', dilated_ratio='[1,2]', flash_attention=True)
longnet = LongNetDecoder(config)
```

--------------------------------

### Initialize and Use LongNet Decoder

Source: https://context7.com/microsoft/torchscale/llms.txt

Configure and use the LongNetDecoder for sequence generation tasks. Supports dilated attention and requires Flash Attention.

```python
# LongNet Decoder configuration
decoder_config = DecoderConfig(
    vocab_size=64000,
    decoder_embed_dim=768,
    decoder_attention_heads=12,
    decoder_layers=12,
    flash_attention=True,
    segment_length='[2048,4096]',
    dilated_ratio='[1,2]'
)

# Initialize LongNet decoder
longnet_decoder = LongNetDecoder(decoder_config)

# Forward pass
prev_output_tokens = torch.randint(0, 64000, (batch_size, long_seq_len))
output, extra = longnet_decoder(prev_output_tokens)
```

--------------------------------

### Create a RetNet Model

Source: https://github.com/microsoft/torchscale/blob/main/README.md

Instantiate a RetNetDecoder model using RetNetConfig and the RetNetDecoder class from torchscale.architecture.retnet.

```python
# Creating a RetNet model
import torch
from torchscale.architecture.config import RetNetConfig
from torchscale.architecture.retnet import RetNetDecoder

config = RetNetConfig(vocab_size=64000)
retnet = RetNetDecoder(config)

print(retnet)
```

--------------------------------

### Create a BERT-like Encoder Model

Source: https://github.com/microsoft/torchscale/blob/main/README.md

Instantiate a BERT-like encoder model using EncoderConfig and the Encoder class from torchscale.architecture.

```python
from torchscale.architecture.config import EncoderConfig
from torchscale.architecture.encoder import Encoder

config = EncoderConfig(vocab_size=64000)
model = Encoder(config)

print(model)
```

--------------------------------

### Initialize FeedForwardNetwork without SubLN

Source: https://context7.com/microsoft/torchscale/llms.txt

Initializes a standard FeedForwardNetwork without SubLayer Normalization. This is a basic FFN configuration.

```python
ffn_basic = FeedForwardNetwork(
    embed_dim=768,
    ffn_dim=3072,
    activation_fn="gelu",
    dropout=0.1,
    activation_dropout=0.0,
    layernorm_eps=1e-5,
    subln=False
)
```

--------------------------------

### Train Sparse (MoE) Model

Source: https://github.com/microsoft/torchscale/blob/main/examples/fairseq/README.md

Use this command to train a sparse Mixture-of-Experts model. Ensure data and checkpoint paths are set.

```bash
cd examples/fairseq/\npython -m torch.distributed.launch --nproc_per_node=8 --nnodes=8 train.py ${PATH_TO_DATA} \         --task pretraining  \         --tokens-per-sample 512  \         --mask-prob 0.15  \         --span-length 3.0  \         --leave-unmasked-prob 0.0  \         --random-token-prob 0.0 \         --arch mlm_base  \         --share-encoder-input-output-embed \         --required-batch-size-multiple 8 \         --spm-model ${PATH_TO_DATA}/sentencepiece.bpe.model \         --dict-file ${PATH_TO_DATA}/dict.txt \         --optimizer adam  \         --adam-betas '(0.9,0.98)'  \         --adam-eps 1e-6  \         --clip-norm 2.0 \         --lr-scheduler polynomial_decay  \         --lr 0.0005  \         --warmup-updates 10000  \         --total-num-update 125000 \         --max-update 125000 \         --max-sentences 32  \         --update-freq 1 \         --log-format simple  \         --log-interval 100 \         --disable-validation \         --save-interval-updates 5000 \         --no-epoch-checkpoints \         --fp16 \         --fp16-init-scale 4 \         --fp16-scale-window 256 \         --min-loss-scale 0.0001 \         --seed 1 \         --save-dir ${PATH_TO_CKPT} \         --ddp-backend=no_c10d \         --distributed-no-spawn \         --reset-dataloader \         --batch-read-ahead 10000 \         --rel-pos-buckets 32 \         --max-rel-pos 128 \         --deepnorm \         --moe-expert-count 64 --moe-freq 2 \         --moe-gating-use-fp32 --moe-second-expert-policy random --moe-normalize-gate-prob-before-dropping \         --moe-eval-capacity-token-fraction -1.0 \         --criterion masked_lm_moe_cross_entropy --moe-gate-loss-wt 0.01 --moe-gate-loss-combine-method sum \         --use-xmoe --pad-to-max-length
```

--------------------------------

### Convert Whole Slide Images

Source: https://github.com/microsoft/torchscale/blob/main/examples/longvit/get_started/get_started_for_tcga_subtyping.md

Resize whole slide images to a specified size for fine-tuning. Replace `/path/to/your_WSIs` and `/path/to/your_resized_WSIs` with your actual directory paths, and set `${target_size}` and `${wsi_level}`.

```python
python data_preprocessing/convert_wsi_to_images.py /path/to/your_WSIs /path/to/your_resized_WSIs ${target_size} ${wsi_level}
```

--------------------------------

### Initialize and Use LongNet Encoder

Source: https://context7.com/microsoft/torchscale/llms.txt

Configure and use the LongNetEncoder for processing long sequences with dilated attention. Ensure Flash Attention is enabled for efficiency.

```python
import torch
from torchscale.architecture.config import EncoderConfig, DecoderConfig
from torchscale.model.LongNet import LongNetEncoder, LongNetDecoder

# LongNet Encoder configuration with dilated attention
encoder_config = EncoderConfig(
    vocab_size=64000,
    encoder_embed_dim=768,
    encoder_attention_heads=12,
    encoder_layers=12,
    flash_attention=True,  # Required for LongNet
    segment_length='[2048,4096]',  # Segment lengths for dilated attention
    dilated_ratio='[1,2]'  # Dilation ratios
)

# Initialize LongNet encoder
longnet_encoder = LongNetEncoder(encoder_config)

# Forward pass with long sequence
batch_size = 2
long_seq_len = 8192  # Can handle very long sequences
src_tokens = torch.randint(0, 64000, (batch_size, long_seq_len))
output = longnet_encoder(src_tokens)
```

--------------------------------

### Train LongNet Model

Source: https://github.com/microsoft/torchscale/blob/main/examples/fairseq/README.md

Command to train a LongNet language model. Ensure data path is set and adjust parameters as needed.

```bash
cd examples/fairseq/ 
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 train.py 
    ${PATH_TO_DATA} 
    --num-workers 2 
    --activation-fn gelu 
    --share-decoder-input-output-embed 
    --validate-interval-updates 1000 
    --save-interval-updates 1000 
    --no-epoch-checkpoints 
    --memory-efficient-fp16 
    --fp16-init-scale 4 
    --arch lm_base 
    --task language_modeling 
    --sample-break-mode none 
    --tokens-per-sample 4096 
    --optimizer adam --adam-betas "(0.9, 0.98)" 
    --adam-eps 1e-08 
    --clip-norm 0.0 
    --lr 5e-4 
    --lr-scheduler polynomial_decay 
    --warmup-updates 750 
    --dropout 0.1 
    --attention-dropout 0.1 
    --weight-decay 0.01 
    --batch-size 4 
    --update-freq 1 
    --required-batch-size-multiple 1 
    --total-num-update 50000 
    --max-update 50000 
    --seed 1 
    --ddp-backend=c10d 
    --flash-attention 
    --segment-length [2048,4096] 
    --dilated-ratio [1,2]
```

--------------------------------

### Fine-tune LongViT on TCGA Survival (Large Images)

Source: https://github.com/microsoft/torchscale/blob/main/examples/longvit/get_started/get_started_for_tcga_survival_prediction.md

Command for fine-tuning LongViT on very large images (32,768x32,768) using sequence parallelism. It requires specifying the input size, model, task, and paths to pre-trained models and data. `--seq_parallel` enables sequence parallelism for large images.

```bash
# TASK - {"brca", "kidney", "lung"}  
# K_FOLD - {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}   
python -m torch.distributed.launch --nproc_per_node=8 run_longvit_finetuning.py \
        --input_size 32768 \
        --model longvit_small_patch32_32768 \
        --task tcga_${TASK}_survival \
        --batch_size 2 \
        --layer_decay 1.0 \
        --lr 5e-5 \
        --update_freq 4 \
        --epochs 10 \
        --warmup_epochs 1 \
        --drop_path 0.1 \
        --finetune /your_longvit_model_path/longvit_small_patch32_1024.pth
        --data_path ./subtyping_split_index/tcga_${TASK} \
        --image_dir /path/to/your_splited_WSIs  \
        --output_dir /path/to/save/your_model \
        --log_dir /path/to/save/your_model/log \
        --weight_decay 0.05 \
        --seed 42 \
        --save_ckpt_freq 5 \
        --k_fold ${K_FOLD} \
        --num_workers 1 \
        --enable_deepspeed \
        --model_key teacher \
        --seq_parallel
```

--------------------------------

### Initialize and use RetNetDecoder model

Source: https://context7.com/microsoft/torchscale/llms.txt

Initializes a RetNetDecoder model with a specified configuration and performs a forward pass in parallel mode, suitable for training.

```python
import torch
from torchscale.architecture.config import RetNetConfig
from torchscale.architecture.retnet import RetNetDecoder

# Create RetNet configuration
config = RetNetConfig(
    vocab_size=64000,
    decoder_embed_dim=768,
    decoder_value_embed_dim=1280,
    decoder_retention_heads=4,
    decoder_ffn_embed_dim=1280,
    decoder_layers=12
)

# Initialize RetNet decoder
model = RetNetDecoder(config)

# Prepare input tokens
batch_size, seq_len = 2, 128
prev_output_tokens = torch.randint(0, 64000, (batch_size, seq_len))

# Forward pass (parallel mode for training)
output, extra = model(prev_output_tokens)
logits = output  # Shape: (batch_size, seq_len, vocab_size)
```

--------------------------------

### Pretrain Sparse MoE BERT

Source: https://context7.com/microsoft/torchscale/llms.txt

Command to pretrain a sparse Mixture-of-Experts (MoE) BERT model. This configuration includes MoE-specific parameters for expert count and gating.

```bash
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=8 train.py ${PATH_TO_DATA} \
    --task pretraining \
    --tokens-per-sample 512 \
    --mask-prob 0.15 \
    --arch mlm_base \
    --share-encoder-input-output-embed \
    --optimizer adam --adam-betas '(0.9,0.98)' \
    --lr 0.0005 --lr-scheduler polynomial_decay \
    --warmup-updates 10000 --max-update 125000 \
    --max-sentences 32 --fp16 \
    --deepnorm \
    --moe-expert-count 64 --moe-freq 2 \
    --moe-gating-use-fp32 --use-xmoe \
    --criterion masked_lm_moe_cross_entropy --moe-gate-loss-wt 0.01
```

--------------------------------

### Initialize and Use Decoder Model

Source: https://context7.com/microsoft/torchscale/llms.txt

Implement a causal transformer decoder for language modeling. Requires DecoderConfig. Supports forward passes with padding masks and returns logits and inner states.

```python
import torch
from torchscale.architecture.config import DecoderConfig
from torchscale.architecture.decoder import Decoder

# Create decoder configuration
config = DecoderConfig(
    vocab_size=64000,
    decoder_embed_dim=768,
    decoder_attention_heads=12,
    decoder_ffn_embed_dim=3072,
    decoder_layers=12
)

# Initialize decoder model
model = Decoder(config)

# Prepare input tokens (batch_size=2, seq_len=128)
batch_size, seq_len = 2, 128
prev_output_tokens = torch.randint(0, 64000, (batch_size, seq_len))
```

```python
# Forward pass (training mode)
output, extra = model(prev_output_tokens)
logits = output  # Shape: (batch_size, seq_len, vocab_size)
inner_states = extra["inner_states"]  # Hidden states from each layer
```

```python
# Forward pass with padding mask
padding_mask = torch.zeros(batch_size, seq_len).bool()
output, extra = model(prev_output_tokens, self_attn_padding_mask=padding_mask)
```

--------------------------------

### Basic Decoder Configuration

Source: https://context7.com/microsoft/torchscale/llms.txt

Defines a basic configuration for decoder-only transformer models like GPT. Includes standard parameters and `max_target_positions` for autoregressive generation.

```python
from torchscale.architecture.config import DecoderConfig

# Basic decoder configuration for GPT-like models
config = DecoderConfig(
    vocab_size=64000,
    decoder_embed_dim=768,
    decoder_attention_heads=12,
    decoder_ffn_embed_dim=3072,
    decoder_layers=12,
    dropout=0.1,
    activation_fn="gelu",
    max_target_positions=1024
)
```

--------------------------------

### BEiT3 Vision-Language Input

Source: https://context7.com/microsoft/torchscale/llms.txt

Process combined visual and textual input with BEiT3 for multimodal tasks like VQA or image captioning. The output includes multimodal features and the split position.

```python
# Vision-language input (VQA, image captioning, etc.)
visual_tokens = torch.randn(batch_size, 3, 224, 224)
textual_tokens = torch.randint(0, 64000, (batch_size, 64))
text_padding = torch.zeros(batch_size, 64).bool()

output = model(
    visual_tokens=visual_tokens,
    textual_tokens=textual_tokens,
    text_padding_position=text_padding
)
multimodal_features = output["encoder_out"]
split_position = output["multiway_split_position"]  # Position where vision ends and text begins

# Extract vision and text features separately
vision_part = multimodal_features[:, :split_position, :]
text_part = multimodal_features[:, split_position:, :]
```

--------------------------------

### Fine-tune LongViT on TCGA Survival (Standard Images)

Source: https://github.com/microsoft/torchscale/blob/main/examples/longvit/get_started/get_started_for_tcga_survival_prediction.md

Command to fine-tune LongViT for TCGA survival prediction on images up to 16,384x16,384. It utilizes distributed training with 8 GPUs and includes options for model path, data paths, and training parameters. `--randaug` enables image augmentation.

```bash
# IMAGE_SIZE - {1024, 4096, 8192, 16384}
# TASK - {"brca", "kidney", "lung"}  
# K_FOLD - {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}   
python -m torch.distributed.launch --nproc_per_node=8 run_longvit_finetuning.py \
        --input_size ${IMAGE_SIZE} \
        --model longvit_small_patch32_${IMAGE_SIZE} \
        --task tcga_${TASK}_survival \
        --batch_size 1 \
        --layer_decay 1.0 \
        --lr 5e-5 \
        --update_freq 1 \
        --epochs 10 \
        --warmup_epochs 1 \
        --drop_path 0.1 \
        --finetune /your_longvit_model_path/longvit_small_patch32_1024.pth
        --data_path ./survival_split_index/tcga_${TASK} \
        --image_dir /path/to/your_resized_WSIs  \
        --output_dir /path/to/save/your_model \
        --log_dir /path/to/save/your_model/log \
        --weight_decay 0.05 \
        --seed 42 \
        --save_ckpt_freq 5 \
        --k_fold ${K_FOLD} \
        --num_workers 1 \
        --enable_deepspeed \
        --model_key teacher \
        --randaug
```

--------------------------------

### BEiT3 Vision-Only Input

Source: https://context7.com/microsoft/torchscale/llms.txt

Process visual input with the BEiT3 model for tasks like image classification. The model expects RGB images of size 224x224 with a patch size of 16.

```python
# Vision-only input (image classification, etc.)
visual_tokens = torch.randn(batch_size, 3, 224, 224)  # RGB images
output = model(visual_tokens=visual_tokens)
vision_features = output["encoder_out"]
```

--------------------------------

### Export SentencePiece Vocabulary to Dictionary

Source: https://github.com/microsoft/torchscale/blob/main/examples/fairseq/README.md

Extract the vocabulary from a SentencePiece model file to create a dictionary file required for training.

```bash
spm_export_vocab --model=sentencepiece.bpe.model | sed 's/\t/ /g' | tail -n +4 > dict.txt
```