### Install iTransformer Package

Source: https://github.com/lucidrains/itransformer/blob/main/README.md

Install the iTransformer library using pip. This command fetches and installs the latest version of the package.

```bash
pip install iTransformer
```

--------------------------------

### Complete iTransformer Training Pipeline

Source: https://context7.com/lucidrains/itransformer/llms.txt

Provides a full training example for the iTransformer model, including model initialization with multiple prediction lengths, data preparation using DataLoader, optimizer and scheduler setup, and moving the model to the appropriate device.

```python
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from iTransformer import iTransformer

# Configuration
config = {
    'num_variates': 137,
    'lookback_len': 96,
    'pred_lengths': (12, 24, 48),
    'batch_size': 32,
    'learning_rate': 1e-4,
    'epochs': 100
}

# Initialize model
model = iTransformer(
    num_variates=config['num_variates'],
    lookback_len=config['lookback_len'],
    dim=256,
    depth=6,
    heads=8,
    dim_head=64,
    pred_length=config['pred_lengths'],
    use_reversible_instance_norm=True,
    flash_attn=True
)

# Move to GPU if available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)

# Create synthetic dataset (replace with real data)
num_samples = 1000
X = torch.randn(num_samples, config['lookback_len'], config['num_variates'])
Y = {length: torch.randn(num_samples, length, config['num_variates'])
     for length in config['pred_lengths']}

# DataLoader setup
dataset = TensorDataset(X, Y[12], Y[24], Y[48])
dataloader = DataLoader(dataset, batch_size=config['batch_size'], shuffle=True)

# Optimizer and scheduler
optimizer = torch.optim.AdamW(model.parameters(), lr=config['learning_rate'])
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=config['epochs'])
```

--------------------------------

### Get RevIN Statistics

Source: https://context7.com/lucidrains/itransformer/llms.txt

Extracts and prints statistics like mean, variance, and gamma from the RevIN layer. Ensures reversibility by reconstructing the input.

```python
normalized, reverse_fn, statistics = revin(x, return_statistics=True)
print(f"Mean shape: {statistics.mean.shape}")       # torch.Size([2, 512, 1])
print(f"Variance shape: {statistics.variance.shape}")  # torch.Size([2, 512, 1])
print(f"Gamma shape: {statistics.gamma.shape}")     # torch.Size([512, 1])

x_reconstructed = reverse_fn(normalized)
assert torch.allclose(x, x_reconstructed, atol=1e-5)
print("RevIN is perfectly reversible!")
```

--------------------------------

### iTransformer Single Prediction Length Usage

Source: https://context7.com/lucidrains/itransformer/llms.txt

Demonstrates initializing and using the iTransformer model for single-horizon forecasting. When `pred_length` is an integer, the model returns a tensor directly. Includes an example of training with a single target.

```python
import torch
from iTransformer import iTransformer

# Model with single prediction horizon
model = iTransformer(
    num_variates=50,
    lookback_len=48,
    dim=128,
    depth=4,
    heads=4,
    dim_head=32,
    pred_length=24,                      # Single prediction length (not tuple)
    use_reversible_instance_norm=True,
    flash_attn=True
)

# Input time series
time_series = torch.randn(4, 48, 50)  # (batch=4, lookback=48, variates=50)

# Returns tensor directly (not dict)
predictions = model(time_series)
print(f"Direct tensor output: {predictions.shape}")  # torch.Size([4, 24, 50])

# Training with single target
model.train()
target = torch.randn(4, 24, 50)
loss = model(time_series, targets=target)
print(f"Single-horizon loss: {loss.item():.4f}")
```

--------------------------------

### Initialize Attend Module with Flash Attention

Source: https://context7.com/lucidrains/itransformer/llms.txt

Initializes the Attend module, enabling Flash Attention for efficient computation on compatible GPUs. Falls back to other attention mechanisms if Flash Attention is not available or disabled.

```python
import torch
from iTransformer.attend import Attend

# Initialize attention module with Flash Attention
attend = Attend(
    dropout=0.1,           # Attention dropout rate
    flash=True,            # Enable Flash Attention (requires PyTorch 2.0+)
    causal=False,          # Non-causal attention for iTransformer
    scale=None             # Auto-compute scale as dim_head ** -0.5
)

# Simulated query, key, value tensors
# Shape: (batch, heads, sequence_length, head_dim)
batch_size, num_heads, seq_len, head_dim = 2, 8, 137, 64
q = torch.randn(batch_size, num_heads, seq_len, head_dim)
k = torch.randn(batch_size, num_heads, seq_len, head_dim)
v = torch.randn(batch_size, num_heads, seq_len, head_dim)

# Compute attention output
output = attend(q, k, v)
print(f"Attention output shape: {output.shape}")  # torch.Size([2, 8, 137, 64])
```

```python
# Causal attention for temporal processing (used in iTransformer2D)
causal_attend = Attend(
    dropout=0.1,
    flash=True,
    causal=True            # Enable causal masking
)

# Time-based attention with causal mask
output_causal = causal_attend(q, k, v)
print(f"Causal attention output: {output_causal.shape}")  # torch.Size([2, 8, 137, 64])
```

--------------------------------

### Initialize and Run iTransformerFFT

Source: https://github.com/lucidrains/itransformer/blob/main/README.md

Configures the iTransformerFFT model with specific dimensions and normalization settings, then executes a forward pass on random time series data.

```python
model = iTransformerFFT(
    num_variates = 137,
    lookback_len = 96,                  # or the lookback length in the paper
    dim = 256,                          # model dimensions
    depth = 6,                          # depth
    heads = 8,                          # attention heads
    dim_head = 64,                      # head dimension
    pred_length = (12, 24, 36, 48),     # can be one prediction, or many
    num_tokens_per_variate = 1,         # experimental setting that projects each variate to more than one token. the idea is that the network can learn to divide up into time tokens for more granular attention across time. thanks to flash attention, you should be able to accommodate long sequence lengths just fine
    use_reversible_instance_norm = True # use reversible instance normalization, proposed here https://openreview.net/forum?id=cGDAkQo1C0p . may be redundant given the layernorms within iTransformer (and whatever else attention learns emergently on the first layer, prior to the first layernorm). if i come across some time, i'll gather up all the statistics across variates, project them, and condition the transformer a bit further. that makes more sense
)

time_series = torch.randn(2, 96, 137)  # (batch, lookback len, variates)

preds = model(time_series)
```

--------------------------------

### Initialize and Use iTransformer Model

Source: https://github.com/lucidrains/itransformer/blob/main/README.md

Demonstrates how to initialize and use the standard iTransformer model for time series forecasting. The model can predict multiple future lengths simultaneously. Ensure input data is formatted as (batch, lookback_len, variates).

```python
import torch
from iTransformer import iTransformer

# using solar energy settings

model = iTransformer(
    num_variates = 137,
    lookback_len = 96,                  # or the lookback length in the paper
    dim = 256,                          # model dimensions
    depth = 6,                          # depth
    heads = 8,                          # attention heads
    dim_head = 64,                      # head dimension
    pred_length = (12, 24, 36, 48),     # can be one prediction, or many
    num_tokens_per_variate = 1,         # experimental setting that projects each variate to more than one token. the idea is that the network can learn to divide up into time tokens for more granular attention across time. thanks to flash attention, you should be able to accommodate long sequence lengths just fine
    use_reversible_instance_norm = True # use reversible instance normalization, proposed here https://openreview.net/forum?id=cGDAkQo1C0p . may be redundant given the layernorms within iTransformer (and whatever else attention learns emergently on the first layer, prior to the first layernorm). if i come across some time, i'll gather up all the statistics across variates, project them, and condition the transformer a bit further. that makes more sense
)

time_series = torch.randn(2, 96, 137)  # (batch, lookback len, variates)

preds = model(time_series)

# preds -> Dict[int, Tensor[batch, pred_length, variate]]
#       -> (12: (2, 12, 137), 24: (2, 24, 137), 36: (2, 36, 137), 48: (2, 48, 137))
```

--------------------------------

### Initialize and train standard iTransformer

Source: https://context7.com/lucidrains/itransformer/llms.txt

Demonstrates model instantiation with specific hyperparameters for multivariate time series forecasting and the usage of multi-horizon prediction and training.

```python
import torch
from iTransformer import iTransformer

# Initialize model for solar energy forecasting (137 variates)
model = iTransformer(
    num_variates=137,                    # Number of input variables/features
    lookback_len=96,                     # Historical window size (input sequence length)
    dim=256,                             # Model embedding dimension
    depth=6,                             # Number of transformer layers
    heads=8,                             # Number of attention heads
    dim_head=64,                         # Dimension per attention head
    pred_length=(12, 24, 36, 48),        # Multiple prediction horizons
    num_tokens_per_variate=1,            # Tokens per variate (increase for finer granularity)
    num_mem_tokens=4,                    # Memory tokens for additional capacity
    num_residual_streams=4,              # Hyper-connection residual streams
    use_reversible_instance_norm=True,   # Handle distribution shift
    reversible_instance_norm_affine=False,
    attn_dropout=0.1,                    # Attention dropout rate
    ff_dropout=0.1,                      # Feedforward dropout rate
    ff_mult=4,                           # Feedforward hidden dim multiplier
    flash_attn=True                      # Use Flash Attention for efficiency
)

# Input: (batch_size, lookback_len, num_variates)
time_series = torch.randn(2, 96, 137)

# Inference: returns dict with predictions for each horizon
preds = model(time_series)
# Output: {12: (2, 12, 137), 24: (2, 24, 137), 36: (2, 36, 137), 48: (2, 48, 137)}

print(f"12-step prediction shape: {preds[12].shape}")  # torch.Size([2, 12, 137])
print(f"48-step prediction shape: {preds[48].shape}")  # torch.Size([2, 48, 137])

# Training with targets: returns MSE loss directly
model.train()
targets = tuple(torch.randn(2, length, 137) for length in (12, 24, 36, 48))
loss = model(time_series, targets=targets)
print(f"Training loss: {loss.item():.4f}")
loss.backward()
```

--------------------------------

### Import iTransformer2D

Source: https://context7.com/lucidrains/itransformer/llms.txt

Basic import statement for the iTransformer2D variant.

```python
import torch
from iTransformer import iTransformer2D
```

--------------------------------

### Initialize and Use iTransformer2D Model

Source: https://github.com/lucidrains/itransformer/blob/main/README.md

Initializes and uses the iTransformer2D model, which supports granular attention across time tokens. Set `num_time_tokens` to enable this feature. Input data should be (batch, lookback_len, variates).

```python
import torch
from iTransformer import iTransformer2D

# using solar energy settings

model = iTransformer2D(
    num_variates = 137,
    num_time_tokens = 16,               # number of time tokens (patch size will be (look back length // num_time_tokens))
    lookback_len = 96,                  # the lookback length in the paper
    dim = 256,                          # model dimensions
    depth = 6,                          # depth
    heads = 8,                          # attention heads
    dim_head = 64,                      # head dimension
    pred_length = (12, 24, 36, 48),     # can be one prediction, or many
    use_reversible_instance_norm = True # use reversible instance normalization
)

time_series = torch.randn(2, 96, 137)  # (batch, lookback len, variates)

preds = model(time_series)

# preds -> Dict[int, Tensor[batch, pred_length, variate]]
#       -> (12: (2, 12, 137), 24: (2, 24, 137), 36: (2, 36, 137), 48: (2, 48, 137))
```

--------------------------------

### Initialize and Use iTransformerFFT Model

Source: https://context7.com/lucidrains/itransformer/llms.txt

Initialize the iTransformerFFT variant which incorporates Fourier domain information. The model processes both time-domain and frequency-domain representations. Input time series shape is (batch_size, lookback_len, num_variates). Inference returns predictions, and training requires targets.

```python
import torch
from iTransformer import iTransformerFFT

# Initialize FFT-enhanced variant
model = iTransformerFFT(
    num_variates=137,                    # Number of input variables
    lookback_len=96,                     # Input sequence length
    dim=256,                             # Model embedding dimension
    depth=6,                             # Number of transformer layers
    heads=8,                             # Number of attention heads
    dim_head=64,                         # Dimension per attention head
    pred_length=(12, 24, 36, 48),        # Multiple prediction horizons
    num_tokens_per_variate=1,            # Tokens per variate
    num_mem_tokens=4,                    # Memory tokens
    use_reversible_instance_norm=True,   # Handle distribution shift
    reversible_instance_norm_affine=False,
    attn_dropout=0.1,
    ff_dropout=0.1,
    ff_mult=4,
    flash_attn=True
)

# Input time series with potential periodic patterns
time_series = torch.randn(2, 96, 137)

# Model automatically computes FFT and processes both domains
preds = model(time_series)
# Output: {12: (2, 12, 137), 24: (2, 24, 137), 36: (2, 36, 137), 48: (2, 48, 137)}

print(f"FFT model 36-step prediction: {preds[36].shape}")  # torch.Size([2, 36, 137])

# Training with targets
model.train()
targets = tuple(torch.randn(2, l, 137) for l in (12, 24, 36, 48))
loss = model(time_series, targets=targets)
print(f"FFT model loss: {loss.item():.4f}")
```

--------------------------------

### Initialize and Use RevIN for Normalization

Source: https://context7.com/lucidrains/itransformer/llms.txt

Initialize RevIN for normalizing time series data, handling distribution shifts. The forward pass returns normalized data and a reverse function. The reverse function is used to denormalize the model's output back to the original scale.

```python
import torch
from iTransformer.revin import RevIN

# Initialize RevIN for 512 variates
revin = RevIN(
    num_variates=512,      # Number of variables to normalize
    affine=True,           # Learn scale (gamma) and shift (beta) parameters
    eps=1e-5               # Epsilon for numerical stability
)

# Input: (batch, variates, time)
x = torch.randn(2, 512, 1024)

# Forward pass returns normalized tensor and reverse function
normalized, reverse_fn = revin(x)
print(f"Normalized mean: {normalized.mean():.4f}")  # Close to 0
print(f"Normalized std: {normalized.std():.4f}")    # Close to 1

# After model processing, reverse the normalization
model_output = normalized * 0.5 + 0.1  # Simulated model processing
denormalized = reverse_fn(model_output)
print(f"Denormalized shape: {denormalized.shape}")  # torch.Size([2, 512, 1024])
```

--------------------------------

### Initialize and Use iTransformer2D Model

Source: https://context7.com/lucidrains/itransformer/llms.txt

Initialize the 2D variant of iTransformer for multivariate time series forecasting. Input shape is (batch_size, lookback_len, num_variates). Inference returns a dictionary of predictions for specified horizons. Training mode requires targets for loss computation.

```python
import torch
from iTransformer import iTransformer2D

# Initialize 2D variant with time tokenization
model = iTransformer2D(
    num_variates=137,                    # Number of input variables
    lookback_len=96,                     # Must be divisible by num_time_tokens
    num_time_tokens=16,                  # Number of time tokens (patch_size = 96/16 = 6)
    dim=256,                             # Model embedding dimension
    depth=6,                             # Number of transformer layers
    heads=8,                             # Number of attention heads
    dim_head=64,                         # Dimension per attention head
    pred_length=(12, 24, 36, 48),        # Multiple prediction horizons
    num_mem_tokens=4,                    # Memory tokens
    use_reversible_instance_norm=True,   # Handle distribution shift
    reversible_instance_norm_affine=True,
    attn_dropout=0.1,
    ff_dropout=0.1,
    ff_mult=4,
    flash_attn=True
)

# Input: (batch_size, lookback_len, num_variates)
time_series = torch.randn(2, 96, 137)

# Inference returns dict with predictions
preds = model(time_series)
# Output: {12: (2, 12, 137), 24: (2, 24, 137), 36: (2, 36, 137), 48: (2, 48, 137)}

print(f"Predictions for 24-step horizon: {preds[24].shape}")  # torch.Size([2, 24, 137])

# Training mode with automatic loss computation
model.train()
targets = (
    torch.randn(2, 12, 137),
    torch.randn(2, 24, 137),
    torch.randn(2, 36, 137),
    torch.randn(2, 48, 137)
)
loss = model(time_series, targets=targets)
print(f"2D model training loss: {loss.item():.4f}")
```

--------------------------------

### PyTorch Training Loop for iTransformer

Source: https://context7.com/lucidrains/itransformer/llms.txt

Standard PyTorch training loop for the iTransformer model. Ensure the model, dataloader, optimizer, and scheduler are properly initialized and configured.

```python
model.train()
for epoch in range(config['epochs']):
    total_loss = 0
    for batch_x, *batch_targets in dataloader:
        batch_x = batch_x.to(device)
        batch_targets = tuple(t.to(device) for t in batch_targets)

        optimizer.zero_grad()
        loss = model(batch_x, targets=batch_targets)
        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()

        total_loss += loss.item()

    scheduler.step()
    avg_loss = total_loss / len(dataloader)
    if (epoch + 1) % 10 == 0:
        print(f"Epoch {epoch+1}/{config['epochs']}, Loss: {avg_loss:.4f}")
```

--------------------------------

### PyTorch Evaluation Loop for iTransformer

Source: https://context7.com/lucidrains/itransformer/llms.txt

PyTorch evaluation loop for the iTransformer model. This snippet demonstrates how to perform inference and print prediction shapes for different horizons.

```python
model.eval()
with torch.no_grad():
    test_input = torch.randn(1, 96, 137).to(device)
    predictions = model(test_input)
    for horizon, pred in predictions.items():
        print(f"Horizon {horizon}: {pred.shape}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.