### Install Example Dependencies

Source: https://github.com/lucidrains/transfusion-pytorch/blob/main/README.md

Install the necessary dependencies to run the examples provided in the project.

```bash
$ pip install .[examples]
```

--------------------------------

### Install transfusion-pytorch

Source: https://github.com/lucidrains/transfusion-pytorch/blob/main/README.md

Install the transfusion-pytorch library using pip.

```bash
$ pip install transfusion-pytorch
```

--------------------------------

### Install Safetensors and Related Libraries

Source: https://github.com/lucidrains/transfusion-pytorch/blob/main/README.md

Install or upgrade specific libraries including safetensors, diffusers, transformers, accelerate, scipy, and ftfy to resolve potential errors.

```bash
$ pip install -U diffusers transformers accelerate scipy ftfy safetensors
```

--------------------------------

### Multi-Modal Dataset and Training Example

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Example of creating a custom multi-modal dataset and using it with the Transfusion model for training. Ensure the dataset returns tuples of (type_index, tensor) for different modalities.

```python
text_images_and_audio = [
    [randint(0, 256, (16,)), (0, randn(4, 384)), randint(0, 256, (8,)), (1, randn(6, 192))],
    [randint(0, 256, (16,)), randn(7, 384), (1, randn(2, 192)), randint(0, 256, (9,))]
]

class MultiModalDataset(Dataset):
    def __len__(self):
        return 1000

    def __getitem__(self, idx):
        # Return (text_label, image_tensor) - order determines generation order
        return torch.tensor([idx % 10]), randn(4, 384)

dataset = MultiModalDataset()
dataloader = model.create_dataloader(dataset, batch_size = 16, shuffle = True)

optimizer = Adam(model.parameters(), lr = 3e-4)

for batch in dataloader:
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
    optimizer.step()
    optimizer.zero_grad()
```

--------------------------------

### Sampling Modalities and Guidance

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Demonstrates various sampling configurations including modality prompts, classifier-free guidance, and raw modality retrieval.

```python
image_prompt = torch.randn(16, 384).cuda()
sample = model.sample(prompt = (0, image_prompt), max_length = 1024)

# Sample with classifier-free guidance
sample = model.sample(
    prompt = text_prompt,
    max_length = 1024,
    cfg_scale = 3.0  # guidance scale (1.0 = no guidance)
)

# Get raw modalities without decoder processing
raw_sample = model.sample(
    max_length = 1024,
    return_unprocessed_modalities = True
)
```

--------------------------------

### Multi-Modal Sampling with sample() Method

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Utilizes the `sample()` method for generating complete multi-modal sequences, supporting both text and modality prompting. It alternates between text and flow-based modality generation.

```python
import torch
from transfusion_pytorch import Transfusion, print_modality_sample

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    modality_default_shape = (16,),
    transformer = dict(dim = 512, depth = 8)
).cuda()

# Sample from scratch (no prompt)
sample = model.sample(
    max_length = 2048,        # maximum sequence length
    text_temperature = 1.5,   # temperature for text sampling
    text_min_p = 0.1,         # min-p filtering for text
    modality_steps = 16,      # ODE steps for flow sampling
    cache_kv = True           # cache key-values for faster generation
)

# Inspect the sample structure
print_modality_sample(sample)
# Output: [('text', torch.Size([...])), ('modality', torch.Size([...])), ...]

# Sample with text prompt
text_prompt = torch.tensor([72, 101, 108, 108, 111]).cuda()  # "Hello"
sample = model.sample(prompt = text_prompt, max_length = 1024)
```

--------------------------------

### Utilize Dataloading and Debugging Helpers

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Leverage built-in dataloader creation for mixed-modality datasets and use print_modality_sample to inspect generated multi-modal outputs.

```python
from torch import tensor
from torch.utils.data import Dataset
from transfusion_pytorch import Transfusion, print_modality_sample, create_dataloader

# Character tokenization helpers (built into model)
model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    transformer = dict(dim = 512, depth = 8)
)

# Create dataloader with proper collation for mixed modalities
class MyDataset(Dataset):
    def __len__(self):
        return 100

    def __getitem__(self, idx):
        text = tensor([72, 101, 108, 108, 111])  # "Hello"
        image = torch.randn(16, 384)
        return text, image

dataset = MyDataset()

# Use model's create_dataloader for proper collation
dataloader = model.create_dataloader(dataset, batch_size=8, shuffle=True)

# Or use create_dataloader directly
dataloader = create_dataloader(dataset, batch_size=8, shuffle=True)

# Debug multi-modal samples
sample = model.sample(max_length=256)
print_modality_sample(sample)
# Output example:
# [('text', torch.Size([42])), ('modality', torch.Size([16, 384])), ('text', torch.Size([8]))]

# Access individual components
for item in sample:
    if isinstance(item, tuple):
        modality_type, modality_tensor = item
        print(f"Modality type {modality_type}: {modality_tensor.shape}")
    else:
        print(f"Text tokens: {item.shape}")
```

--------------------------------

### Initialize Transfusion Models

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Create multi-modal models with varying configurations for text tokens, latent dimensions, and transformer architecture.

```python
from transfusion_pytorch import Transfusion

# Basic single-modality model (e.g., text + images)
model = Transfusion(
    num_text_tokens = 256,          # vocabulary size for text
    dim_latent = 384,               # dimension of modality latents
    modality_default_shape = (4,),  # fallback shape if LM doesn't produce valid dimensions
    transformer = dict(
        dim = 512,                  # model dimension
        depth = 8,                  # number of transformer layers
        dim_head = 64,              # dimension per attention head
        heads = 8                   # number of attention heads
    )
)

# Multi-modality model (e.g., text + images + audio)
model = Transfusion(
    num_text_tokens = 256,
    dim_latent = (384, 192),                # different latent dims per modality
    modality_default_shape = ((4,), (2,)),  # default shapes for each modality
    transformer = dict(
        dim = 512,
        depth = 8
    )
)

# Advanced configuration with encoder/decoder
from torch import nn

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    channel_first_latent = True,              # latent format: (batch, channels, ...)
    modality_default_shape = (14, 14),        # 2D modality (e.g., 14x14 patches)
    modality_encoder = nn.Conv2d(3, 384, 3),  # raw image -> latent
    modality_decoder = nn.Conv2d(384, 3, 3),  # latent -> raw image
    add_pos_emb = True,                       # add axial positional embeddings
    modality_num_dim = 2,                     # number of dimensions (2D for images)
    prob_uncond = 0.1,                        # probability of unconditional training for CFG
    flow_loss_weight = 1.,                    # weight for flow matching loss
    text_loss_weight = 1.,                    # weight for text prediction loss
    velocity_consistency_loss_weight = 0.1,   # weight for velocity consistency
    reconstruction_loss_weight = 0.,          # optional reconstruction loss
    transformer = dict(
        dim = 512,
        depth = 8,
        attn_laser = True,                    # use LASER attention
        unet_skips = True                     # use U-Net style skip connections
    )
)
```

--------------------------------

### Modality-Only Training and Generation with Custom Encoder/Decoder

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Shows how to set up Transfusion for modality-only tasks like image generation. Use `forward_modality` for training and `generate_modality_only` for sampling. Requires custom encoder and decoder modules.

```python
import torch
from torch import nn
from einops import rearrange
from transfusion_pytorch import Transfusion

# Custom encoder/decoder for images
class Encoder(nn.Module):
    def forward(self, x):
        # Patchify: (batch, 1, 28, 28) -> (batch, 14, 14, 4)
        x = rearrange(x, '... 1 (h p1) (w p2) -> ... h w (p1 p2)', p1=2, p2=2)
        return x * 2 - 1  # normalize to [-1, 1]

class Decoder(nn.Module):
    def forward(self, x):
        # Unpatchify: (batch, 14, 14, 4) -> (batch, 1, 28, 28)
        x = rearrange(x, '... h w (p1 p2) -> ... 1 (h p1) (w p2)', p1=2, p2=2)
        return ((x + 1) * 0.5).clamp(min=0., max=1.)

model = Transfusion(
    num_text_tokens = 0,              # no text tokens for image-only model
    dim_latent = 4,
    modality_default_shape = (14, 14),
    modality_encoder = Encoder(),
    modality_decoder = Decoder(),
    add_pos_emb = True,
    modality_num_dim = 2,
    transformer = dict(dim = 128, depth = 6)
).cuda()

# Training on images only (Float tensor)
images = torch.randn(8, 1, 28, 28).cuda()
loss = model(images)  # uses forward_modality internally
loss.backward()

# Generate images unconditionally
generated_images = model.generate_modality_only(
    batch_size = 16,
    modality_type = 0,               # which modality to generate
    fixed_modality_shape = (14, 14), # shape of generated modality
    modality_steps = 16              # number of ODE integration steps
)

print(f"Generated shape: {generated_images.shape}")  # (16, 1, 28, 28) after decoder
```

--------------------------------

### Transfusion Model - One Modality (Images)

Source: https://github.com/lucidrains/transfusion-pytorch/blob/main/README.md

Initialize and use the Transfusion model for a single modality, such as images. Ensure input tensors are of type torch.long for text and torch.float for modalities.

```python
from torch import randint, randn
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    modality_default_shape = (4,),  # fallback, in the case the language model did not produce a valid modality shape
    transformer = dict(
        dim = 512,
        depth = 8
    )
)

# any torch.long is text, torch.float is modalities

text_and_images = [
    [randint(0, 256, (16,)), randn(4, 384), randint(0, 256, (8,)), randn(6, 384)],
    [randint(0, 256, (16,)), randn(7, 384), randint(0, 256, (5,)), randn(2, 384), randint(0, 256, (9,))]
]

loss = model(text_and_images)

loss.backward()

# after much training

one_multimodal_sample = model.sample()
```

--------------------------------

### Monitor Training Loss Breakdown

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Use the return_breakdown flag during the forward pass to retrieve individual loss components for text, flow, velocity, and reconstruction.

```python
import torch
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    modality_default_shape = (16,),
    velocity_consistency_loss_weight = 0.1,
    reconstruction_loss_weight = 0.1,
    transformer = dict(dim = 512, depth = 8)
).cuda()

ema_model = model.create_ema()

data = [
    [torch.randint(0, 256, (16,)), torch.randn(16, 384)],
    [torch.randint(0, 256, (8,)), torch.randn(8, 384)]
]

# Get detailed loss breakdown
total_loss, breakdown = model(
    data,
    velocity_consistency_ema_model = ema_model,
    return_breakdown = True
)

print(f"Total loss: {breakdown.total.item():.4f}")
print(f"Text loss: {breakdown.text.item():.4f}")
print(f"Flow losses: {[l.item() for l in breakdown.flow]}")

if breakdown.velocity is not None:
    print(f"Velocity losses: {[l.item() for l in breakdown.velocity]}")

if breakdown.recon is not None:
    print(f"Recon losses: {[l.item() for l in breakdown.recon]}")
```

--------------------------------

### Text-Only Training and Generation with Transfusion

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Demonstrates how to configure and use the Transfusion model for text-only tasks. Pass data as a standard `Int['batch seq']` tensor for training and use `generate_text_only` for generation.

```python
import torch
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    transformer = dict(
        dim = 384,
        depth = 8,
        dim_head = 64,
        heads = 8,
        attn_laser = True  # LASER attention for better performance
    )
).cuda()

# Text-only training: pass Int['batch seq'] tensor directly
text = torch.randint(0, 256, (2, 1024)).cuda()

loss = model(text)  # automatically uses forward_text internally
loss.backward()

# Text-only generation
prompt = text[:, :64]  # use first 64 tokens as prompt

generated = model.generate_text_only(
    prompt = prompt,
    seq_len = 512,           # total length to generate
    temperature = 1.5,       # sampling temperature
    min_p = 0.1              # min-p filtering threshold
)

print(f"Generated shape: {generated.shape}")  # (batch, seq_len - prompt_len)
```

--------------------------------

### Transfusion Model - Text-Only Pretraining

Source: https://github.com/lucidrains/transfusion-pytorch/blob/main/README.md

Pretrain the Transfusion model on text data only by passing text tensors directly. Use `.cuda()` to move the model to the GPU. The `generate_text_only` method can be used for text generation after training.

```python
import torch
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    transformer = dict(
        dim = 512,
        depth = 8,
    )
).cuda()

text = torch.randint(0, 256, (2, 1024)).cuda()

loss = model(text)
loss.backward()

# after much training

sampled = model.generate_text_only(text[:, :1], 1024)
```

--------------------------------

### Train with Mixed Modalities

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Pass mixed text and modality data as a list of tensors to the model to compute the training loss.

```python
import torch
from torch import randint, randn
from torch.utils.data import Dataset, DataLoader
from torch.optim import Adam
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    modality_default_shape = (4,),
    transformer = dict(dim = 512, depth = 8)
).cuda()

# Data format: list of samples, each sample is a list of text/modality tensors
# torch.long = text tokens, torch.float = modality
text_and_images = [
    [randint(0, 256, (16,)), randn(4, 384), randint(0, 256, (8,)), randn(6, 384)],
    [randint(0, 256, (16,)), randn(7, 384), randint(0, 256, (5,)), randn(2, 384)]
]

# Forward pass returns loss
loss = model(text_and_images)
loss.backward()
```

--------------------------------

### EMA Model Training

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Implements an Exponential Moving Average model to track smoothed weights for improved sampling quality.

```python
import torch
from torch.optim import Adam
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    modality_default_shape = (16,),
    transformer = dict(dim = 512, depth = 8)
).cuda()

# Create EMA model
ema_model = model.create_ema(beta = 0.99)  # decay rate

optimizer = Adam(model.parameters(), lr = 3e-4)

# Training loop
for step in range(10000):
    data = [...]  # your training data

    loss = model(data)
    loss.backward()

    optimizer.step()
    optimizer.zero_grad()

    # Update EMA after each step
    ema_model.update()

    # Sample using EMA model for better quality
    if step % 100 == 0:
        sample = ema_model.sample(max_length = 512)

# EMA model has same interface as main model
generated_text = ema_model.generate_text_only(prompt, seq_len = 256)
generated_image = ema_model.generate_modality_only(batch_size = 4)
```

--------------------------------

### VAE Latent Space Integration

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Configures Transfusion to operate in compressed latent space using custom VAE encoder/decoder wrappers.

```python
import torch
from torch import nn
from transfusion_pytorch import Transfusion

# VAE encoder/decoder wrappers
class VAEEncoder(nn.Module):
    def __init__(self, vae):
        super().__init__()
        self.vae = vae

    def forward(self, image):
        with torch.no_grad():
            latent = self.vae.encode(image * 2 - 1)
        return 0.18215 * latent.latent_dist.sample()

class VAEDecoder(nn.Module):
    def __init__(self, vae):
        super().__init__()
        self.vae = vae

    def forward(self, latents):
        latents = (1 / 0.18215) * latents
        with torch.no_grad():
            image = self.vae.decode(latents).sample
        return (image / 2 + 0.5).clamp(0, 1)

# Load your VAE (e.g., from diffusers)
# vae = AutoencoderKL.from_pretrained(...)

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 4,                           # VAE latent channels
    channel_first_latent = True,              # VAE uses channel-first format
    modality_default_shape = (32, 32),        # latent spatial dimensions
    modality_encoder = VAEEncoder(vae),
    modality_decoder = VAEDecoder(vae),
    # Optional: learned down/up projections (like UNet in paper)
    pre_post_transformer_enc_dec = (
        nn.Conv2d(4, 128, 3, 2, 1),                              # downsample
        nn.ConvTranspose2d(128, 4, 3, 2, 1, output_padding=1),  # upsample
    ),
    add_pos_emb = True,
    modality_num_dim = 2,
    reconstruction_loss_weight = 0.1,  # optional: add reconstruction loss
    transformer = dict(
        dim = 512,
        depth = 12,
        dim_head = 64,
        heads = 8
    )
).cuda()

# Training with raw images (encoder handles VAE encoding)
images = torch.randn(4, 3, 256, 256).cuda()
text_labels = [torch.randint(0, 256, (10,)) for _ in range(4)]

data = [[text, img] for text, img in zip(text_labels, images)]
loss = model(data)
```

--------------------------------

### Velocity Consistency Training

Source: https://context7.com/lucidrains/transfusion-pytorch/llms.txt

Enables velocity consistency loss to straighten flow trajectories, requiring an EMA model for the loss calculation.

```python
import torch
from torch.optim import Adam
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = 384,
    modality_default_shape = (16,),
    velocity_consistency_loss_weight = 0.1,  # weight for velocity loss
    transformer = dict(dim = 512, depth = 8)
).cuda()

# EMA model is required for velocity consistency
ema_model = model.create_ema(beta = 0.99)

optimizer = Adam(model.parameters(), lr = 3e-4)

for step in range(10000):
    data = [...]  # training data

    # Pass EMA model for velocity consistency loss
    loss = model(
        data,
        velocity_consistency_ema_model = ema_model,
        velocity_consistency_delta_time = 1e-3  # time delta for consistency
    )

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

    ema_model.update()

# For modality-only training with velocity consistency
images = torch.randn(8, 16, 384).cuda()
loss = model.forward_modality(
    images,
    velocity_consistency_ema_model = ema_model,
    return_loss = True
)
```

--------------------------------

### Transfusion Model - Image Encoding/Decoding

Source: https://github.com/lucidrains/transfusion-pytorch/blob/main/README.md

Integrate custom image encoders and decoders with the Transfusion model. Ensure `channel_first_latent` is set to True if your latent representation is channel-first. Use `print_modality_sample` to visualize the output.

```python
import torch
from torch import nn, randint, randn
from transfusion_pytorch import Transfusion, print_modality_sample

mock_encoder = nn.Conv2d(3, 384, 3, padding = 1)
mock_decoder = nn.Conv2d(384, 3, 3, padding = 1)

model = Transfusion(
    num_text_tokens = 12,
    dim_latent = 384,
    channel_first_latent = True,
    modality_default_shape = (4, 4),
    modality_encoder = mock_encoder,
    modality_decoder = mock_decoder,
    transformer = dict(
        dim = 512,
        depth = 8
    )
)

text_and_images = [
    [
        randint(0, 12, (16,)),  # 16 text tokens
        randn(3, 8, 8),         # (8 x 8) 3 channeled image
        randint(0, 12, (8,)),   # 8 text tokens
        randn(3, 7, 7)          # (7 x 7) 3 channeled image
    ],
    [
        randint(0, 12, (16,)),  # 16 text tokens
        randn(3, 8, 5),         # (8 x 5) 3 channeled image
        randint(0, 12, (5,)),   # 5 text tokens
        randn(3, 2, 16),        # (2 x 16) 3 channeled image
        randint(0, 12, (9,))    # 9 text tokens
    ]
]

loss = model(text_and_images)

loss.backward()

# after much training

one_multimodal_sample = model.sample()

print_modality_sample(one_multimodal_sample)
```

--------------------------------

### Transfusion Model - Multiple Modalities

Source: https://github.com/lucidrains/transfusion-pytorch/blob/main/README.md

Configure and utilize the Transfusion model for multiple modalities, specifying distinct latent dimensions and default shapes. Modality index is passed as the first element of a tuple for float tensors.

```python
from torch import randint, randn
from transfusion_pytorch import Transfusion

model = Transfusion(
    num_text_tokens = 256,
    dim_latent = (384, 192),                 # specify multiple latent dimensions
    modality_default_shape = ((4,), (2,)),   # default shapes for first and second modality
    transformer = dict(
        dim = 512,
        depth = 8
    )
)

# then for the Tensors of type float, you can pass a tuple[int, Tensor] and specify the modality index in the first position

# any torch.long is text, torch.float is modalities

text_images_and_audio = [
    [randint(0, 256, (16,)), (0, randn(4, 384)), randint(0, 256, (8,)), (1, randn(6, 192))],
    [randint(0, 256, (16,)), randn(7, 384), randint(0, 256, (5,)), (1, randn(2, 192)), randint(0, 256, (9,))]
]

loss = model(text_images_and_audio)

loss.backward()

# after much training

one_multimodal_sample = model.sample()
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.