### Install Open MPI

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Horovod-Installation

Download, extract, configure, and install Open MPI version 4.1.0. Ensure the installation completes successfully before proceeding.

```bash
wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.0.tar.gz
tar -xf openmpi-4.1.0.tar.gz
cd openmpi-4.1.0
gunzip -c openmpi-4.1.0.tar.gz | tar xf - 
cd openmpi-4.1.0
./configure --prefix=/usr/local
# <...lots of output...>
make all install
```

--------------------------------

### Install MPI Prerequisites

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Horovod

Commands to download, configure, and install OpenMPI from source.

```bash
# install prerequisites
sudo make install -y g++
# 
wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.0.tar.gz
tar -xf openmpi-4.1.0.tar.gz
cd openmpi-4.1.0
./configure --prefix=/usr/local
# <...lots of output...>
make all && sudo make install
```

--------------------------------

### Install YouTokenToMe

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Install the YouTokenToMe library to create custom BPE models.

```bash
$ pip install youtokentome
```

--------------------------------

### Setup DALL-E Environment with Pip

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed---Installation

Create and configure a virtual environment using Pip.

```bash
#!/bin/bash

python -m pip install virtualenv
python -m virtualenv -p=python3.7 ~/.virtualenvs/dalle_env
source ~/.virtualenvs/dalle_env/bin/activate
# Make sure your terminal shows that you're inside the virtual environment - and then run:
pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
pip install "git+https://github.com:lucidrains/DALLE-pytorch.git"
```

--------------------------------

### Install DALL-E PyTorch

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Install the dalle-pytorch library using pip. This is the primary step to begin using the library.

```bash
pip install dalle-pytorch
```

--------------------------------

### Install Weights & Biases

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Install the Weights & Biases library for experiment tracking. This is a common tool used for monitoring training progress and results.

```bash
pip install wandb
```

--------------------------------

### Install Horovod

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Horovod-Installation

Install the Horovod library using pip after successfully installing MPI. This enables distributed training capabilities.

```bash
pip install horovod
```

--------------------------------

### Install Apex for DeepSpeed

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Install Apex from source to enable automatic mixed precision support via DeepSpeed.

```sh
sh install_apex.sh
```

--------------------------------

### Install Cairo Dependencies

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Install the necessary Cairo graphics library development files and the pycairo Python binding.

```bash
!apt install libcairo2-dev
!pip install pycairo
```

--------------------------------

### Install CLIP and Load Model

Source: https://github.com/lucidrains/dalle-pytorch/wiki/CLIP-Rerank

Installs the CLIP library from GitHub and loads a pre-trained ViT-B/32 model. It automatically selects the CUDA device if available, otherwise defaults to CPU.

```python
%pip install "git+https://github.com/openai/CLIP.git"
import clip
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
```

--------------------------------

### Install NVIDIA Apex with CUDA Extensions

Source: https://github.com/lucidrains/dalle-pytorch/wiki/DeepSpeed---Apex-Automatic-Mixed-Precision

Clone the NVIDIA Apex repository and install it with C++ and CUDA extensions enabled. Ensure you are in the 'apex' directory before running the pip install command.

```sh
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```

--------------------------------

### Initialize and Train DALL-E

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Setup the DiscreteVAE and DALLE components, then perform a training step with random tensors.

```python
vae = DiscreteVAE(
    image_size = 256,
    num_layers = 3,
    num_tokens = 8192,
    codebook_dim = 1024,
    hidden_dim = 64,
    num_resnet_blocks = 1,
    temperature = 0.9
)

# Initialize DALL-E
dalle = DALLE(
    dim = 1024,                     # Model dimension
    vae = vae,                      # VAE for image encoding/decoding
    num_text_tokens = 10000,        # Vocabulary size for text
    text_seq_len = 256,             # Max text sequence length
    depth = 12,                     # Number of transformer layers (aim for 64)
    heads = 16,                     # Number of attention heads
    dim_head = 64,                  # Dimension per attention head
    reversible = False,             # Enable reversible networks for memory efficiency
    attn_dropout = 0.1,             # Attention dropout rate
    ff_dropout = 0.1,               # Feed-forward dropout rate
    loss_img_weight = 7,            # Weight for image loss vs text loss
    attn_types = ('full',),         # Attention types: full, axial_row, axial_col, conv_like, sparse
    stable = False,                 # Stable softmax for fp16 training
    shift_tokens = True,            # Use token shifting
    rotary_emb = True               # Use rotary positional embeddings
)

# Training
text = torch.randint(0, 10000, (4, 256))
images = torch.randn(4, 3, 256, 256)

loss = dalle(text, images, return_loss=True)
loss.backward()

# Generation
with torch.no_grad():
    generated_images = dalle.generate_images(
        text,
        filter_thres = 0.9,         # Top-k filtering threshold
        temperature = 1.0           # Sampling temperature
    )
    # generated_images.shape: (4, 3, 256, 256)
```

--------------------------------

### Install Deepspeed Sparse Attention Dependencies

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed---Installation

Initial system-level installation of required libraries and cloning the Deepspeed repository.

```sh
sudo apt-get -y install llvm-9-dev cmake
git clone https://github.com/microsoft/DeepSpeed.git /tmp/Deepspeed
cd /tmp/Deepspeed && DS_BUILD_SPARSE_ATTN=1 ./install.sh -s # Change this to -r if you need to run as root
pip install triton
cd ~
```

--------------------------------

### Install Triton Package

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Install the Triton package, ensuring it is a version less than 1.0, as required by Microsoft's implementation for Sparse Attention.

```bash
pip install triton==0.4.2
```

--------------------------------

### Setup DALL-E Environment with Conda

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed---Installation

Create and configure a Conda environment for DALL-E PyTorch.

```bash
#!/bin/bash

conda create -n dalle_env python=3.7
conda activate dalle_env
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch
pip install "git+https://github.com:lucidrains/DALLE-pytorch.git"
```

--------------------------------

### Configure CUDA on Pop!_OS

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed-ZeRO-Infinity

Commands to install system-specific CUDA packages and manage versions using update-alternatives.

```bash
sudo apt install system76-cuda-latest
sudo apt install system76-cudnn-latest
sudo update-alternatives --config cuda
# Choose the most recent version of cuda-toolkit-you see here. 

# After you're done - to switch back to your original cuda-toolkit version, just run:
sudo update-alternatives --config cuda
```

--------------------------------

### Install Dependencies on Debian

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed-ZeRO-Infinity

Commands to install system dependencies and the required Python packages for DeepSpeed and dalle-pytorch.

```bash
apt install -y libaio-dev gcc cmake llvm-9-dev
python -V # Check your version

# For CUDA 11.1 - change if you have a different version. CUDA 11.2 not supported.
pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
pip3 install deepspeed
pip3 install dalle-pytorch
```

--------------------------------

### Install Deepspeed Sparse Attention

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Install Deepspeed with Sparse Attention support by running the provided shell script. This is a prerequisite for using sparse attention features.

```bash
sh install_deepspeed.sh
```

--------------------------------

### Generate a Grid of Images

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Initializes an empty list to store generated images and begins a loop to create a grid of 20 rows and 40 columns. This snippet is the start of a larger image generation process.

```python
pics = []
for _ in range(20):
    row = []
    for _ in range(40):

```

--------------------------------

### Configure Sparse Attention for Latest Nvidia GPUs

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed---Installation

Commands to install drivers, compatible torch versions, and specific DeepSpeed branches for RTX 30XX and A100 GPUs.

```bash
sudo apt install nvidia-driver-460
```

```bash
pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
```

```bash
git clone https://github.com/ptillet/triton.git;
cd triton;
git checkout a598fba0
pip install ./ --no-cache-dir
```

```bash
git clone --single-branch --branch sparse_triton_support https://github.com/afiaka87/DeepSpeed.git
```

```bash
nano requirements/requirements-sparse_attn.txt
```

```bash
cd DeepSpeed;
DS_BUILD_SPARSE_ATTN=1 ./install.sh -s;
```

```bash
DS_BUILD_SPARSE_ATTN=1 pip install ./ --no-cache-dir
```

--------------------------------

### CLIP Training and Inference

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Train the CLIP model and perform inference to get similarity scores between text and images. Requires torch.

```python
# Training
text = torch.randint(0, 10000, (4, 256))
images = torch.randn(4, 3, 256, 256)
mask = torch.ones_like(text).bool()

loss = clip(text, images, text_mask=mask, return_loss=True)
loss.backward()
```

```python
# Inference: get similarity scores
with torch.no_grad():
    scores = clip(text, images, text_mask=mask, return_loss=False)
    # scores.shape: (4,) - cosine similarity scores
```

```python
# Use with DALL-E for ranking generations
dalle_images, ranking_scores = dalle.generate_images(text, clip=clip)
top_indices = ranking_scores.argsort(descending=True)
```

--------------------------------

### Initialize DALL-E Model

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Configure the DALL-E model architecture and set up the optimizer and learning rate scheduler.

```python
dalle = DALLE(
    dim = 1024,
    vae = vae,                  # automatically infer (1) image sequence length and (2) number of image tokens
    num_text_tokens = len(word_tokens) + 1,    # vocab size for text
    text_seq_len = longest_caption,         # text sequence length
    depth = 12,                 # should aim to be 64
    heads = 16,                 # attention heads
    dim_head = 64,              # attention head dimension
    attn_dropout = 0.1,         # attention dropout
    ff_dropout = 0.1            # feedforward dropout
).to(device)

opt = torch.optim.Adam(dalle.parameters(), lr=0.001, weight_decay=0.0)
scheduler = torch.optim.lr_scheduler.ExponentialLR(opt, 0.98)
```

--------------------------------

### Configure Classifier-Free Guidance

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Use null_cond_prob during training and cond_scale during generation to implement classifier-free guidance.

```python
import torch
from dalle_pytorch import DiscreteVAE, DALLE

vae = DiscreteVAE(
    image_size = 256,
    num_layers = 3,
    num_tokens = 8192,
    codebook_dim = 1024,
    hidden_dim = 64,
    num_resnet_blocks = 1,
    temperature = 0.9
)

dalle = DALLE(
    dim = 1024,
    vae = vae,
    num_text_tokens = 10000,
    text_seq_len = 256,
    depth = 12,
    heads = 16,
    dim_head = 64,
    attn_dropout = 0.1,
    ff_dropout = 0.1
)

text = torch.randint(0, 10000, (4, 256))
images = torch.randn(4, 3, 256, 256)

loss = dalle(
    text,
    images,
    return_loss = True,
    null_cond_prob = 0.2  # firstly, set this to the probability of dropping out the condition, 20% is recommended as a default
)

loss.backward()

# do the above for a long time with a lot of data ... then

images = dalle.generate_images(
    text,
    cond_scale = 3. # secondly, set this to a value greater than 1 to increase the conditioning beyond average
)

images.shape # (4, 3, 256, 256)
```

--------------------------------

### Initialize CLIP

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Basic import for the CLIP contrastive model.

```python
import torch
from dalle_pytorch import CLIP
```

--------------------------------

### Initialize DALL-E with Discrete VAE

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Initialize DALL-E with a Discrete VAE. The VAE handles image tokenization. Requires torch.

```python
import torch
from dalle_pytorch import DiscreteVAE, DALLE

vae = DiscreteVAE(image_size=256, num_layers=3, num_tokens=8192, codebook_dim=1024, hidden_dim=64)
dalle = DALLE(dim=1024, vae=vae, num_text_tokens=10000, text_seq_len=256, depth=12, heads=16)
```

--------------------------------

### Load OpenAI Pretrained VAE

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Instantiate DALL-E using OpenAI's pretrained Discrete VAE. The model is automatically downloaded and cached.

```python
import torch
from dalle_pytorch import OpenAIDiscreteVAE, DALLE

vae = OpenAIDiscreteVAE()       # loads pretrained OpenAI VAE

dalle = DALLE(
    dim = 1024,
    vae = vae,                  # automatically infer (1) image sequence length and (2) number of image tokens
    num_text_tokens = 10000,    # vocab size for text
    text_seq_len = 256,         # text sequence length
    depth = 1,                  # should aim to be 64
    heads = 16,                 # attention heads
    dim_head = 64,              # attention head dimension
    attn_dropout = 0.1,         # attention dropout
    ff_dropout = 0.1            # feedforward dropout
)

text = torch.randint(0, 10000, (4, 256))
images = torch.randn(4, 3, 256, 256)

loss = dalle(text, images, return_loss = True)
loss.backward()
```

--------------------------------

### Initialize DALL-E with Reversible Networks and Sparse Attention

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Initialize DALL-E with reversible networks for memory efficiency and various sparse attention patterns. Requires torch.

```python
import torch
from dalle_pytorch import DiscreteVAE, DALLE

vae = DiscreteVAE(image_size=256, num_layers=3, num_tokens=8192, codebook_dim=512, hidden_dim=64)

# Sparse attention patterns
dalle = DALLE(
    dim = 1024,
    vae = vae,
    num_text_tokens = 10000,
    text_seq_len = 256,
    depth = 64,
    heads = 16,
    dim_head = 64,
    reversible = True,              # Enable reversible networks (memory efficient)
    attn_types = (
        'full',
        'axial_row',
        'axial_col',
        'conv_like'
    )
)
```

--------------------------------

### Manage Docker Containers

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Build and run the DALL-E environment using Docker.

```bash
docker build -t dalle docker
```

```bash
docker run --gpus all -it --mount src="$(pwd)",target=/workspace/dalle,type=bind dalle:latest bash
```

--------------------------------

### Train DALL-E with WebDataset

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Use WebDataset .tar files for training by specifying column keys and the data path.

```bash
$ python train_dalle.py --wds img,cap --image_text_folder /path/to/data.tar(.gz)
```

```bash
$ deepspeed train_dalle.py --wds img,cap --image_text_folder /path/to/data.tar(.gz) --fp16 --deepspeed
```

```bash
$ deepspeed train_dalle.py --wds img,cap --image_text_folder /path/to/shardfolder --fp16 --deepspeed
```

```bash
$ deepspeed train_dalle.py --image_text_folder "http://storage.googleapis.com/nvdata-openimages/openimages-train-{000000..000554}.tar" --wds jpg,json --taming --truncate_captions --random_resize_crop_lower_ratio=0.8 --attn_types=full --epochs=2 --fp16 --deepspeed
```

--------------------------------

### Get and Decode Image Codes

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Retrieves the codebook indices for all images using the trained VAE and then decodes these indices back into image format. It also samples and displays 10 random codebook index sets.

```python
with torch.no_grad():
    all_image_codes = vae.get_codebook_indices(images)
    all_images_decoded = vae.decode(all_image_codes)
    
all_image_codes[np.random.choice(images.shape[0], 10), ...]
```

--------------------------------

### DALLE.generate_images - Image Generation

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

The generate_images method performs autoregressive image generation from text prompts. It supports image priming (starting from a partial image), CLIP-based ranking, and classifier-free guidance for improved text conditioning.

```APIDOC
## DALLE.generate_images - Image Generation

### Description
The `generate_images` method performs autoregressive image generation from text prompts. It supports image priming (starting from a partial image), CLIP-based ranking, and classifier-free guidance for improved text conditioning.

### Method
`POST` (or similar, depending on framework)

### Endpoint
`/dalle/generate_images`

### Parameters
#### Query Parameters
- **text** (torch.Tensor) - Required - Text tokens to condition image generation.
- **filter_thres** (float) - Optional - Top-k filtering threshold for sampling.
- **temperature** (float) - Optional - Sampling temperature.
- **img** (torch.Tensor) - Optional - Image to prime the generation from.
- **num_init_img_tokens** (int) - Optional - Number of initial image tokens to use for priming.
- **clip** (CLIP) - Optional - CLIP model instance for ranking generated images.
- **cond_scale** (float) - Optional - Conditioning scale for classifier-free guidance.

### Request Example
```python
import torch
from dalle_pytorch import DiscreteVAE, DALLE, CLIP

vae = DiscreteVAE(image_size=256, num_layers=3, num_tokens=8192, codebook_dim=512, hidden_dim=64)
dalle = DALLE(dim=1024, vae=vae, num_text_tokens=10000, text_seq_len=256, depth=12, heads=16)

text = torch.randint(0, 10000, (2, 256))

# Basic generation
images = dalle.generate_images(text, filter_thres=0.9, temperature=1.0)

# Generation with image priming (start from partial image)
img_prime = torch.randn(2, 3, 256, 256)
images = dalle.generate_images(
    text,
    img = img_prime,
    num_init_img_tokens = 14 * 32   # ~43.75% of image tokens as primer
)

# Generation with CLIP ranking
clip = CLIP(
    dim_text=512, dim_image=512, dim_latent=512,
    num_text_tokens=10000, text_enc_depth=6, text_seq_len=256, text_heads=8,
    num_visual_tokens=512, visual_enc_depth=6, visual_image_size=256, visual_patch_size=32, visual_heads=8
)
images, scores = dalle.generate_images(text, clip=clip)

# Generation with classifier-free guidance (requires training with null_cond_prob)
images = dalle.generate_images(
    text,
    cond_scale = 3.0,               # Conditioning scale > 1 increases text adherence
    filter_thres = 0.9
)
```

### Response
#### Success Response (200)
- **images** (torch.Tensor) - Generated images.
- **scores** (torch.Tensor) - CLIP similarity scores (if CLIP is used).

#### Response Example
```json
{
  "images": "[tensor of shape (batch_size, 3, height, width)]",
  "scores": "[tensor of shape (batch_size,)]"
}
```
```

--------------------------------

### DALL-E Training with Classifier-Free Guidance

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Train DALL-E using classifier-free guidance by randomly dropping text conditions. Requires torch.

```python
text = torch.randint(0, 10000, (4, 256))
images = torch.randn(4, 3, 256, 256)

# Training with null conditioning probability
# Randomly drops text condition with 20% probability
loss = dalle(
    text,
    images,
    return_loss = True,
    null_cond_prob = 0.2            # Probability of dropping text condition
)
loss.backward()
```

--------------------------------

### Train DALL-E with Pretrained VAE

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Set up and train a DALL-E model using a previously trained Discrete VAE. Requires significant training time and data.

```python
import torch
from dalle_pytorch import DiscreteVAE, DALLE

vae = DiscreteVAE(
    image_size = 256,
    num_layers = 3,
    num_tokens = 8192,
    codebook_dim = 1024,
    hidden_dim = 64,
    num_resnet_blocks = 1,
    temperature = 0.9
)

dalle = DALLE(
    dim = 1024,
    vae = vae,                  # automatically infer (1) image sequence length and (2) number of image tokens
    num_text_tokens = 10000,    # vocab size for text
    text_seq_len = 256,         # text sequence length
    depth = 12,                 # should aim to be 64
    heads = 16,                 # attention heads
    dim_head = 64,              # attention head dimension
    attn_dropout = 0.1,         # attention dropout
    ff_dropout = 0.1            # feedforward dropout
)

text = torch.randint(0, 10000, (4, 256))
images = torch.randn(4, 3, 256, 256)

loss = dalle(text, images, return_loss = True)
loss.backward()

# do the above for a long time with a lot of data ... then

images = dalle.generate_images(text)
images.shape # (4, 3, 256, 256)
```

--------------------------------

### Train DALL-E with Taming Transformer VQVAE

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Enable the Taming Transformer VQVAE by adding the --taming flag.

```bash
$ python train_dalle.py --image_text_folder /path/to/coco/dataset --taming
```

--------------------------------

### Distributed Training Commands

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed---Usage

Command-line instructions for executing training sessions using the DeepSpeed backend.

```bash
deepspeed train_dalle.py <...> --distributed_backend deepspeed
```

```bash
deepspeed train_dalle.py <...> --distributed_backend deepspeed --fp16
```

```bash
deepspeed --num_gpus 1 train_dalle.py <...> --distributed_backend deepspeed
```

```bash
deepspeed train_dalle.py --image_text_folder=/path/to/your/dataset --distributed_backend --deepspeed --fp16
```

--------------------------------

### Generate BPE Model

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Create a BPE model file using the youtokentome command-line tool.

```bash
$ yttm bpe --vocab_size 8000 --data ./path/to/big/text/file.txt --model ./path/to/bpe.model
```

--------------------------------

### Configure full attention

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Attention-Layers

Use full attention when VRAM and runtime savings are not required.

```python
attn_types = ('full')
```

--------------------------------

### Train DALL-E Model (Resume Training)

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Resume DALL-E training from a checkpoint by running the `train_dalle.py` script. Specify the path to your DALL-E checkpoint file and the image-text data folder.

```python
python train_dalle.py --dalle_path ./dalle.pt --image_text_folder /path/to/data
```

--------------------------------

### Run Training with DeepSpeed

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Execute the training script using the deepspeed launcher with AMP enabled.

```sh
deepspeed train_dalle.py \
    --taming \
    --image_text_folder 'DatasetsDir' \
    --distr_backend 'deepspeed' \
    --amp
```

--------------------------------

### HuggingFace Tokenizer Initialization and Usage

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Initialize and use a HuggingFace tokenizer from a BPE JSON file. Requires torch.

```python
from dalle_pytorch.tokenizer import HugTokenizer

# HuggingFace tokenizer (from BPE json file)
hug_tokenizer = HugTokenizer(bpe_path='./tokenizer.json')
tokens = hug_tokenizer.tokenize(["hello world"], context_length=256)
```

--------------------------------

### Train DALL-E with Custom Tokenizer

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Hugging-Faces-Tokenizer

Use the --bpe_path argument to specify the path to a BPE JSON file when training the model.

```bash
$ python train_dalle.py --image_text_folder ./path/to/data --bpe_path ./path/to/bpe.json
```

--------------------------------

### YouTokenToMe BPE Tokenizer Initialization and Usage

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Initialize and use a YouTokenToMe BPE tokenizer from a model file. Requires torch.

```python
from dalle_pytorch.tokenizer import YttmTokenizer

# YouTokenToMe BPE tokenizer
yttm_tokenizer = YttmTokenizer(bpe_path='./bpe.model')
tokens = yttm_tokenizer.tokenize(["custom vocabulary text"], context_length=256)
```

--------------------------------

### Create WebDataset Shards

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

This Python script demonstrates how to create WebDataset shards for training. It uses the `webdataset` library to write image data and captions into tar files, specifying column names for image and text data.

```python
import webdataset as wds

with wds.TarWriter("dataset-000000.tar") as sink:
    for idx, (image_path, caption) in enumerate(dataset):
        with open(image_path, "rb") as f:
            image_data = f.read()
        sink.write({
            "__key__": f"sample{idx:06d}",
            "jpg": image_data,
            "txt": caption.encode("utf-8")
        })
```

--------------------------------

### Login to Weights & Biases

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Log in to your Weights & Biases account. This command is necessary to authenticate and associate your training runs with your account.

```bash
wandb login
```

--------------------------------

### Visualize random VAE reconstructions

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Selects 10 random images from the dataset and displays them alongside their VAE reconstructions.

```python
idx = np.random.choice(images.shape[0], 10)
orig_pics = images[idx, ...].cpu().permute(0, 2, 3, 1).reshape(320, 32, 3).permute(1, 0, 2).numpy()
decoded_pics = all_images_decoded[idx, ...].cpu().permute(0, 2, 3, 1).reshape(320, 32, 3).permute(1, 0, 2).numpy()
both = np.concatenate((orig_pics, decoded_pics), axis=0)
plt.imshow(both)
plt.axis("off")
```

--------------------------------

### Import Libraries

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Import all necessary libraries for DALL-E, image manipulation, and plotting. Includes numpy, cairo, matplotlib, PIL, and torch.

```python
import math
import itertools
import os
import glob

import numpy as np
import cairo
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
%config InlineBackend.figure_format = 'retina'
from PIL import Image
from tqdm.autonotebook import tqdm, trange
import torch
from dalle_pytorch import DiscreteVAE, DALLE
```

--------------------------------

### Load Custom VQGAN VAE

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Load a custom VQGAN VAE by specifying the model and config paths. Useful for using different pretrained VQGAN models.

```python
from dalle_pytorch import VQGanVAE

vae = VQGanVAE(vqgan_model_path = 'path/to/your/model.ckpt', vqgan_config_path = 'path/to/your/config.yaml')
```

--------------------------------

### Run Distributed Training on Multiple Machines

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Horovod-Installation

Launch a DALL-E training job across multiple machines, with each machine utilizing 4 GPUs. The '-H' flag specifies the hostnames and the number of GPUs per host.

```bash
$ horovodrun -np 16 -H server1:4,server2:4,server3:4,server4:4 python train_dalle.py --image_text_folder=/path/to/your/dataset --distributed_backend horovod
```

--------------------------------

### DALL-E Generation with Classifier-Free Guidance

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Generate images with DALL-E, strengthening text conditioning by adjusting the conditioning scale. Requires torch.

```python
# Generation with increased conditioning
# cond_scale > 1 strengthens text conditioning
generated = dalle.generate_images(
    text,
    cond_scale = 3.0,               # Conditioning scale (1.0 = normal, >1 = stronger)
    filter_thres = 0.9,
    temperature = 1.0
)
```

--------------------------------

### Initialize CLIP Model

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Initialize the CLIP model with specified dimensions and vocabulary size. Used for text-image similarity.

```python
clip = CLIP(
    dim_text = 512,                 # Text encoder dimension
    dim_image = 512,                # Image encoder dimension
    dim_latent = 512,               # Latent space dimension
    num_text_tokens = 10000,        # Text vocabulary size
    text_enc_depth = 6,             # Text transformer depth
    text_seq_len = 256,             # Text sequence length
    text_heads = 8,                 # Text attention heads
    num_visual_tokens = 512,        # Visual tokens (not used, for compatibility)
    visual_enc_depth = 6,           # Vision transformer depth
    visual_image_size = 256,        # Image size
    visual_patch_size = 32,         # Patch size for ViT
    visual_heads = 8                # Vision attention heads
)
```

--------------------------------

### Inspect Captions

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Display the first 10 captions from the dataset.

```python
print(captions[:10])
```

--------------------------------

### Use Pretrained OpenAI VAE with DALL-E

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Load OpenAI's pretrained discrete VAE, which automatically downloads weights. Use this VAE with the DALLE model for text-to-image generation without training a custom VAE.

```python
import torch
from dalle_pytorch import OpenAIDiscreteVAE, DALLE

# Load pretrained OpenAI VAE (downloads automatically)
vae = OpenAIDiscreteVAE()

# Use with DALL-E
dalle = DALLE(
    dim = 1024,
    vae = vae,
    num_text_tokens = 10000,
    text_seq_len = 256,
    depth = 12,
    heads = 16,
    dim_head = 64,
    attn_dropout = 0.1,
    ff_dropout = 0.1
)

# Get codebook indices from images
images = torch.randn(4, 3, 256, 256)
with torch.no_grad():
    indices = vae.get_codebook_indices(images)

# Decode indices back to images
decoded = vae.decode(indices)
```

--------------------------------

### Train DALL-E with OpenAI VAE

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Train DALL-E without a custom VAE path by omitting the --vae_path argument.

```bash
$ python train_dalle.py --image_text_folder /path/to/coco/dataset
```

--------------------------------

### Prepare Captions Array

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Pad and convert tokenized captions into a NumPy array and then to a PyTorch tensor.

```python
longest_caption = max(len(c) for c in captions)
captions_array = np.zeros((len(caption_tokens), longest_caption), dtype=np.int64)
for i in range(len(caption_tokens)):
    captions_array[i, :len(caption_tokens[i])] = caption_tokens[i]
    
captions_array = torch.from_numpy(captions_array).to(device)
```

--------------------------------

### DALL-E with DeepSpeed Sparse Attention

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Initialize DALL-E with DeepSpeed sparse attention for further memory efficiency. Requires torch and triton.

```python
# DeepSpeed sparse attention (requires installation)
# pip install triton==0.4.2
dalle_sparse = DALLE(
    dim = 512,
    vae = vae,
    num_text_tokens = 10000,
    text_seq_len = 256,
    depth = 64,
    heads = 8,
    attn_types = ('full', 'sparse') # Interleave full and sparse attention
)
```

--------------------------------

### Run Distributed Training on Single Machine

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Horovod-Installation

Execute a DALL-E training script on a single machine using 4 GPUs with Horovod. Specify the dataset path and the distributed backend.

```bash
$ horovodrun -np 4 python train_dalle.py --image_text_folder=/path/to/your/dataset --distributed_backend horovod
```

--------------------------------

### Train Discrete VAE

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Use this script to train the Discrete VAE. Specify the image folder and optionally tune parameters for advanced training or enable distributed training with DeepSpeed or Horovod.

```bash
python train_vae.py --image_folder /path/to/images
```

```bash
python train_vae.py \
    --image_folder /path/to/images \
    --image_size 256 \
    --epochs 20 \
    --batch_size 8 \
    --learning_rate 1e-3 \
    --num_tokens 8192 \
    --num_layers 3 \
    --num_resnet_blocks 2 \
    --emb_dim 512 \
    --hidden_dim 256 \
    --starting_temp 1.0 \
    --temp_min 0.5 \
    --anneal_rate 1e-6
```

```bash
deepspeed train_vae.py \
    --image_folder /path/to/images \
    --deepspeed
```

```bash
horovodrun -np 4 train_vae.py \
    --image_folder /path/to/images \
    --distributed_backend horovod
```

--------------------------------

### Generate and Save Images with VAE

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

This script generates variations of an image based on provided parameters, saves them to disk, and uses a VAE for processing. It creates directories if they don't exist and shows progress using tqdm.

```python
all_variations = list(itertools.product(*[v.items() for v in variations.values()]))
if not os.path.exists("data"):
    os.mkdir("data")

if not os.path.exists("data/rainbow"):
    os.mkdir("data/rainbow")
    for vars in tqdm(all_variations):
        name = " ".join(n for n, v in vars if n != "").replace(" ", "_")
        params = dict(zip(variations, [v for n, v in vars]))
        pic = make_pic(32, **params)[:, :, :3]
        im = Image.fromarray(pic)
        im.save(f"data/rainbow/{name}.png")
```

--------------------------------

### Enable Horovod Autotuning

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Horovod-Installation

Run the DALL-E training script with Horovod autotuning enabled. This feature optimizes communication parameters. The autotune log will be saved to the specified path.

```bash
$ mpirun -x HOROVOD_AUTOTUNE=1 -x HOROVOD_AUTOTUNE_LOG=/tmp/autotune_log.csv ... train_dalle.py --image_text_folder=/path/to/your/dataset --distributed_backend horovod
```

--------------------------------

### Initialize Discrete VAE Model

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Initializes a Discrete Variational Autoencoder (VAE) model with specified parameters for image size, layers, tokens, and dimensions. It also sets up an Adam optimizer and an exponential learning rate scheduler.

```python
vae = DiscreteVAE(
    image_size = images.shape[2],
    num_layers = 3,          # number of downsamples - ex. 256 / (2 ** 3) = (32 x 32 feature map)
    num_tokens = 256,       # number of visual tokens. in the paper, they used 8192, but could be smaller for downsized projects
    codebook_dim = 512,      # codebook dimension
    hidden_dim = 64,         # hidden dimension
    num_resnet_blocks = 2,   # number of resnet blocks
    temperature = 0.9,       # gumbel softmax temperature, the lower this is, the harder the discretization
    straight_through = False # straight-through for gumbel softmax. unclear if it is better one way or the other
).to(device)

opt = torch.optim.Adam(vae.parameters(), lr=0.001, weight_decay=0.0)
scheduler = torch.optim.lr_scheduler.ExponentialLR(opt, 0.99)
```

--------------------------------

### Chinese Tokenizer Initialization and Usage

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Initialize and use the Chinese tokenizer, which is based on BERT. Requires torch.

```python
from dalle_pytorch.tokenizer import ChineseTokenizer

# Chinese tokenizer (uses BERT)
chinese_tokenizer = ChineseTokenizer()
tokens = chinese_tokenizer.tokenize([""], context_length=256)
```

--------------------------------

### Generate Images with DALLE-pytorch

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Generate-From-Pretrained-Model

Use this script to generate images. Specify the number of images, DALL-E model path, batch size, output directory, and text prompts. This process can take a significant amount of time.

```bash
#!/bin/bash

# This can take awhile
python generate.py \
    --num_images=512 \
    --dalle_path=dalle.pt \
    --taming \
    --batch_size=32 \
    --outputs_dir="/path" \
    --text "an armchair in the shape of an avocado, an armchair imitating an avocado"
```

--------------------------------

### Run Training with Deepspeed

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed---Installation

Command to execute the training script using Deepspeed with sparse attention arguments.

```bash
deepspeed train_dalle.py --image_text_folder my-text-image-folder --truncate_captions=True --random_resize_crop_lower_ratio 0.9 --taming --attn_types=sparse --fp16 --deepspeed
```

--------------------------------

### DeepSpeed Stage 3 Configuration Template

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Deepspeed-ZeRO-Infinity

A dictionary template for configuring DeepSpeed ZeRO Stage 3, including parameter and optimizer offloading settings.

```python
deepspeed_config = {
    "zero_optimization": {
        "stage": 3,
        # Offload the model parameters If you have an nvme drive - you should use the nvme option.
        # Otherwise, use 'cpu' and remove the `nvme_path` line
        "offload_param": {
            "device": "nvme",
            "nvme_path": "/path/to/nvme/folder",
        },
        # Offload the optimizer of choice. If you have an nvme drive - you should use the nvme option.
        # Otherwise, use 'cpu' and remove the `nvme_path` line
        "offload_optimizer": {
            "device": "nvme", # options are 'none', 'cpu', 'nvme'
            "nvme_path": "/path/to/nvme/folder",
        },
    },
    # Override pytorch's Adam optim with `FusedAdam` (just called Adam here). Can 
    "optimizer": {
        "type": "Adam",  # You can also use AdamW here
        "params": {
            "lr": LEARNING_RATE,
        },
    },
    'train_batch_size': BATCH_SIZE,
    'gradient_clipping': GRAD_CLIP_NORM,
    'fp16': {
        'enabled': args.fp16,
    },
}
```

--------------------------------

### Train with Custom BPE Tokenizer

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Pass the path to a BPE model file when training with a custom tokenizer.

```sh
$ python train_dalle.py --image_text_folder ./path/to/data --bpe_path ./path/to/bpe.model
```

--------------------------------

### Run Distributed Training with Horovod

Source: https://github.com/lucidrains/dalle-pytorch/wiki/Horovod

Commands to execute training on single or multiple nodes using the Horovod backend.

```sh
$ horovodrun -np 4 python train_dalle.py --image_text_folder=/path/to/your/dataset --distributed_backend horovod
```

```sh
$ horovodrun -np 16 -H server1:4,server2:4,server3:4,server4:4 python train_dalle.py --image_text_folder=/path/to/your/dataset --distributed_backend horovod
```

```sh
$ mpirun -x HOROVOD_AUTOTUNE=1 -x HOROVOD_AUTOTUNE_LOG=/tmp/autotune_log.csv ... train_dalle.py --image_text_folder=/path/to/your/dataset --distributed_backend horovod
```

--------------------------------

### Train Discrete VAE

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Instantiate and train a Discrete VAE. Ensure sufficient data for effective codebook learning.

```python
import torch
from dalle_pytorch import DiscreteVAE

vae = DiscreteVAE(
    image_size = 256,
    num_layers = 3,           # number of downsamples - ex. 256 / (2 ** 3) = (32 x 32 feature map)
    num_tokens = 8192,        # number of visual tokens. in the paper, they used 8192, but could be smaller for downsized projects
    codebook_dim = 512,       # codebook dimension
    hidden_dim = 64,          # hidden dimension
    num_resnet_blocks = 1,    # number of resnet blocks
    temperature = 0.9,        # gumbel softmax temperature, the lower this is, the harder the discretization
    straight_through = False, # straight-through for gumbel softmax. unclear if it is better one way or the other
)

images = torch.randn(4, 3, 256, 256)

loss = vae(images, return_loss = True)
loss.backward()

# train with a lot of data to learn a good codebook
```

--------------------------------

### Run Distributed Training with DeepSpeed

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Execute training scripts using the DeepSpeed library for distributed performance.

```bash
$ deepspeed <file>.py [args...] --deepspeed
```

--------------------------------

### Train DALL-E Model (New Training)

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Train the DALL-E model using the `train_dalle.py` script. Provide the path to your trained VAE model and the folder containing image-text data.

```python
python train_dalle.py --vae_path ./vae.pt --image_text_folder /path/to/data
```

--------------------------------

### Integrate Taming Transformers VQGAN with DALL-E

Source: https://context7.com/lucidrains/dalle-pytorch/llms.txt

Wrap pretrained VQGAN models from Taming Transformers for use with DALL-E. This offers reduced training costs due to shorter image sequence lengths. Load the default VQGAN or a custom checkpoint.

```python
import torch
from dalle_pytorch import VQGanVAE, DALLE

# Load default VQGAN (1024 codebook, trained on ImageNet)
vae = VQGanVAE()

# Or load custom VQGAN checkpoint
vae = VQGanVAE(
    vqgan_model_path = '/path/to/model.ckpt',
    vqgan_config_path = '/path/to/config.yaml'
)

# Use with DALL-E for faster training
dalle = DALLE(
    dim = 1024,
    vae = vae,
    num_text_tokens = 10000,
    text_seq_len = 256,
    depth = 12,
    heads = 16
)

images = torch.randn(4, 3, 256, 256)
text = torch.randint(0, 10000, (4, 256))

loss = dalle(text, images, return_loss=True)
loss.backward()
```

--------------------------------

### Fit DALL-E Model

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Train the model or load existing weights from a file.

```python
dalle_model_file = "data/rainbow_dalle.model"
if not os.path.exists(dalle_model_file):
    dalle, loss_history = fit(dalle, opt, None, scheduler, 
                              (captions_array[train_idx, ...], all_image_codes[train_idx, ...]), None, 200, 256, 
                              dalle_model_file, train_dalle_batch, 
                              n_train_samples=len(train_idx))

    plt.plot(loss_history)
else:
    dalle.load_state_dict(torch.load(dalle_model_file))
```

--------------------------------

### Train and Generate with Chinese Tokenizer

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Use the --chinese flag to enable the Huggingface pretrained Chinese tokenizer.

```sh
$ python train_dalle.py --chinese --image_text_folder ./path/to/data
```

```sh
$ python generate.py --chinese --text '追老鼠的猫'
```

--------------------------------

### Load and Preprocess Images for PyTorch

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/examples/rainbow_dalle.ipynb

Loads PNG images from a specified directory, extracts captions, converts them to a NumPy array, normalizes pixel values, and then converts them into a PyTorch tensor on the selected device. The tensor is permuted for PyTorch's expected channel-first format.

```python
captions = []
images = []
for fn in glob.glob("data/rainbow/*.png"):
    captions.append(os.path.basename(fn).replace(".png", "").split("_"))    
    im = np.array(Image.open(fn))
    images.append(im)
    
images = np.stack(images).astype(float) / 255
images = torch.from_numpy(images).to(torch.float32).permute(0, 3, 1, 2).to(device)
```

--------------------------------

### Generate Images with Priming

Source: https://github.com/lucidrains/dalle-pytorch/blob/main/README.md

Generate images using DALL-E, optionally priming the generation with an initial image crop. The size of the initial crop can be controlled.

```python
img_prime = torch.randn(4, 3, 256, 256)

images = dalle.generate_images(
    text,
    img = img_prime,
    num_init_img_tokens = (14 * 32)  # you can set the size of the initial crop, defaults to a little less than ~1/2 of the tokens, as done in the paper
)

images.shape # (4, 3, 256, 256)
```