### Install Prodigy via pip

Source: https://github.com/konstmish/prodigy/blob/main/README.md

Command to install the prodigyopt package.

```bash
pip install prodigyopt
```

--------------------------------

### Initialize Prodigy Optimizer

Source: https://github.com/konstmish/prodigy/blob/main/README.md

Basic setup for the Prodigy optimizer using a neural network's parameters.

```python
from prodigyopt import Prodigy
# choose weight decay value based on your problem, 0 by default
# set slice_p to 11 if you have limited memory, 1 by default
opt = Prodigy(net.parameters(), lr=1., weight_decay=weight_decay, slice_p=slice_p)
```

--------------------------------

### d_coef Tuning Guide Output

Source: https://context7.com/konstmish/prodigy/llms.txt

Prints a guide to help users understand the impact of different `d_coef` values on training behavior, ranging from very conservative to very aggressive.

```python
print("d_coef tuning guide:")
print("  0.1  - Very conservative, slow training")
print("  0.5  - Conservative, stable training")
print("  1.0  - Default, balanced")
print("  2.0  - Aggressive, faster training")
print("  10.0 - Very aggressive, may be unstable")
```

--------------------------------

### Prodigy Optimizer with Monitoring

Source: https://context7.com/konstmish/prodigy/llms.txt

Initializes the Prodigy optimizer with specified learning rate and weight decay. This setup is used for detailed monitoring of optimizer state during training.

```python
optimizer = Prodigy(net.parameters(), lr=1.0, weight_decay=0.01)
```

--------------------------------

### Initialize Prodigy Optimizer

Source: https://context7.com/konstmish/prodigy/llms.txt

Initialize the Prodigy optimizer with recommended lr=1.0. Options include weight decay, memory-efficient slicing, and scaling the learning rate estimate with d_coef.

```python
import torch
import torch.nn as nn
from prodigyopt import Prodigy

# Define a simple neural network
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        return self.fc3(x)

# Initialize network and optimizer
net = SimpleNet()

# Basic initialization - lr=1.0 is recommended
optimizer = Prodigy(net.parameters(), lr=1.0)

# With weight decay (AdamW style decoupled weight decay by default)
optimizer = Prodigy(
    net.parameters(),
    lr=1.0,
    weight_decay=0.01  # Common values: 0, 0.001, 0.01, 0.1
)

# Memory-efficient mode for limited GPU memory
optimizer = Prodigy(
    net.parameters(),
    lr=1.0,
    weight_decay=0.01,
    slice_p=11  # Reduces memory by approximating LR statistics
)

# Force larger/smaller learning rate estimates using d_coef
optimizer = Prodigy(
    net.parameters(),
    lr=1.0,
    d_coef=2.0  # Values > 1 for larger LR, < 1 for smaller LR
)
```

--------------------------------

### Training Loop with Growth Rate Monitoring

Source: https://context7.com/konstmish/prodigy/llms.txt

Demonstrates a training loop using the Prodigy optimizer with a limited growth rate. It monitors and prints the current `d` and `d_max` values at specified intervals.

```python
criterion = nn.CrossEntropyLoss()
for step in range(500):
    optimizer_limited.zero_grad()
    x = torch.randn(32, 784)
    y = torch.randint(0, 10, (32,))
    outputs = net(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer_limited.step()

    if step % 50 == 0:
        d = optimizer_limited.param_groups[0]['d']
        d_max = optimizer_limited.param_groups[0]['d_max']
        print(f"Step {step}: d={d:.6f}, d_max={d_max:.6f}")
```

--------------------------------

### Training Loop with Optimizer State Monitoring

Source: https://context7.com/konstmish/prodigy/llms.txt

Executes a training loop and, after each epoch, prints statistics from the optimizer's parameter groups. This includes current step size (`d`), maximum step size seen (`d_max`), current estimate (`d_hat`), step count (`k`), and effective learning rate.

```python
criterion = nn.CrossEntropyLoss()

# Training with detailed monitoring
for epoch in range(5):
    for step in range(100):
        optimizer.zero_grad()
        x = torch.randn(32, 784)
        y = torch.randint(0, 10, (32,))
        outputs = net(x)
        loss = criterion(outputs, y)
        loss.backward()
        optimizer.step()

    # Access optimizer state after training steps
    group = optimizer.param_groups[0]
    print(f"\nEpoch {epoch + 1} Statistics:")
    print(f"  Current d (step size):     {group['d']:.8f}")
    print(f"  Maximum d seen (d_max):    {group['d_max']:.8f}")
    print(f"  Current estimate (d_hat):  {group.get('d_hat', 'N/A')}")
    print(f"  Step count (k):            {group['k']}")
    print(f"  Effective LR:              {group['d'] * group['lr']:.8f}")
```

--------------------------------

### Basic Training Loop with Prodigy

Source: https://context7.com/konstmish/prodigy/llms.txt

Implement a standard PyTorch training loop using the Prodigy optimizer. Ensure zero_grad() is called before backpropagation and step() is called after.

```python
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from prodigyopt import Prodigy

# Setup model and data
net = SimpleNet()
criterion = nn.CrossEntropyLoss()
optimizer = Prodigy(net.parameters(), lr=1.0, weight_decay=0.01)

# Create dummy data
X = torch.randn(1000, 784)
y = torch.randint(0, 10, (1000,))
dataset = TensorDataset(X, y)
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

# Training loop
num_epochs = 10
for epoch in range(num_epochs):
    total_loss = 0.0
    for batch_x, batch_y in dataloader:
        optimizer.zero_grad()
        outputs = net(batch_x)
        loss = criterion(outputs, batch_y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()

    # Access the estimated learning rate (d parameter)
    current_d = optimizer.param_groups[0]['d']
    print(f"Epoch {epoch+1}, Loss: {total_loss/len(dataloader):.4f}, Estimated d: {current_d:.6f}")
```

--------------------------------

### Configure Prodigy with Warmup Safeguard

Source: https://context7.com/konstmish/prodigy/llms.txt

Enable safeguard_warmup to prevent overestimation of the learning rate during the initial training phase.

```python
optimizer_with_warmup = Prodigy(
    net.parameters(),
    lr=1.0,
    safeguard_warmup=True  # Prevents overestimation during warmup
)
```

--------------------------------

### Configure Cosine Annealing Scheduler

Source: https://github.com/konstmish/prodigy/blob/main/README.md

Recommended scheduler configuration for use with the Prodigy optimizer.

```python
# n_epoch is the total number of epochs to train the network
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=total_steps)
```

--------------------------------

### Configure Prodigy for Diffusion Models

Source: https://context7.com/konstmish/prodigy/llms.txt

Recommended settings for diffusion models include bias correction, specific betas, and weight decay. Increase d0 if initialization is unstable.

```python
import torch
import torch.nn as nn
from prodigyopt import Prodigy

# Example diffusion model (simplified UNet-like architecture)
class SimpleDiffusionModel(nn.Module):
    def __init__(self, in_channels=3, hidden_dim=256):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(in_channels, hidden_dim, 3, padding=1),
            nn.GroupNorm(8, hidden_dim),
            nn.SiLU(),
            nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),
        )
        self.decoder = nn.Sequential(
            nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),
            nn.GroupNorm(8, hidden_dim),
            nn.SiLU(),
            nn.Conv2d(hidden_dim, in_channels, 3, padding=1),
        )

    def forward(self, x, t):
        h = self.encoder(x)
        return self.decoder(h)

model = SimpleDiffusionModel()

# Recommended settings for diffusion models
optimizer = Prodigy(
    model.parameters(),
    lr=1.0,
    betas=(0.9, 0.99),          # Sometimes helpful for diffusion
    weight_decay=0.01,           # Recommended for diffusion
    use_bias_correction=True,    # Enable Adam's bias correction
    safeguard_warmup=True,       # Prevent warmup issues
)

# If model is not training well, try increasing d0
optimizer_with_higher_d0 = Prodigy(
    model.parameters(),
    lr=1.0,
    betas=(0.9, 0.99),
    weight_decay=0.01,
    use_bias_correction=True,
    safeguard_warmup=True,
    d0=1e-5,  # Increase from default 1e-6 to 1e-5 or even 1e-4
)

# Training loop
criterion = nn.MSELoss()
for step in range(1000):
    optimizer.zero_grad()
    # Dummy diffusion training step
    x = torch.randn(4, 3, 64, 64)  # Batch of images
    t = torch.randint(0, 1000, (4,))  # Timesteps
    noise = torch.randn_like(x)
    predicted_noise = model(x + noise, t)
    loss = criterion(predicted_noise, noise)
    loss.backward()
    optimizer.step()

    if step % 100 == 0:
        d_estimate = optimizer.param_groups[0]['d']
        print(f"Step {step}, Loss: {loss.item():.4f}, d: {d_estimate:.6f}")
```

--------------------------------

### Prodigy Optimizer with Limited Growth Rate

Source: https://context7.com/konstmish/prodigy/llms.txt

Initializes the Prodigy optimizer with a limited growth rate for stability during warmup. The `growth_rate` parameter controls the maximum percentage increase of `d` per step.

```python
optimizer_limited = Prodigy(
    net.parameters(),
    lr=1.0,
    growth_rate=1.02  # d can grow at most 2% per step
)
```

--------------------------------

### Prodigy Optimizer with Default d_coef

Source: https://context7.com/konstmish/prodigy/llms.txt

Initializes the Prodigy optimizer with the default `d_coef` value of 1.0. This is the standard configuration for balanced learning rate estimation.

```python
optimizer_default = Prodigy(net.parameters(), lr=1.0, d_coef=1.0)
```

--------------------------------

### Prodigy with Cosine Annealing Scheduler

Source: https://context7.com/konstmish/prodigy/llms.txt

Integrate Prodigy with PyTorch's CosineAnnealingLR scheduler. For warmup phases, set safeguard_warmup=True to prevent premature overestimation of the learning rate.

```python
import torch
import torch.nn as nn
from prodigyopt import Prodigy

net = SimpleNet()
num_epochs = 100
steps_per_epoch = 100
total_steps = num_epochs * steps_per_epoch

# Initialize optimizer
optimizer = Prodigy(net.parameters(), lr=1.0, weight_decay=0.01)

# Cosine annealing scheduler (recommended)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer,
    T_max=total_steps  # Set to total steps to avoid restarts
)

# Training loop with scheduler
criterion = nn.CrossEntropyLoss()
for epoch in range(num_epochs):
    for step in range(steps_per_epoch):
        optimizer.zero_grad()
        # Forward pass with dummy data
        x = torch.randn(32, 784)
        y = torch.randint(0, 10, (32,))
        outputs = net(x)
        loss = criterion(outputs, y)
        loss.backward()
        optimizer.step()
        scheduler.step()  # Step scheduler after optimizer

```

--------------------------------

### Configure Prodigy for FSDP Training

Source: https://context7.com/konstmish/prodigy/llms.txt

Manually set fsdp_in_use for custom FSDP implementations, though Prodigy typically auto-detects PyTorch's built-in FSDP.

```python
import torch
import torch.nn as nn
import torch.distributed as dist
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from prodigyopt import Prodigy

# Initialize distributed training
# dist.init_process_group(backend='nccl')

# Wrap model with FSDP
net = SimpleNet()
# fsdp_model = FSDP(net.cuda())

# Prodigy auto-detects FSDP, but can be set manually
optimizer = Prodigy(
    net.parameters(),  # Use fsdp_model.parameters() when using FSDP
    lr=1.0,
    weight_decay=0.01,
    fsdp_in_use=True  # Set True for custom FSDP implementations
)

# FSDP training loop (conceptual)
criterion = nn.CrossEntropyLoss()
for step in range(100):
    optimizer.zero_grad()
    x = torch.randn(32, 784)
    y = torch.randint(0, 10, (32,))
    outputs = net(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
```

--------------------------------

### Prodigy Optimizer with Higher d_coef

Source: https://context7.com/konstmish/prodigy/llms.txt

Initializes the Prodigy optimizer with a `d_coef` of 2.0, resulting in a larger learning rate estimate. Use this for potentially faster training, but with caution.

```python
optimizer_aggressive = Prodigy(
    net.parameters(),
    lr=1.0,
    d_coef=2.0  # 2x larger learning rate estimate
)
```

--------------------------------

### Prodigy Optimizer with Lower d_coef

Source: https://context7.com/konstmish/prodigy/llms.txt

Initializes the Prodigy optimizer with a `d_coef` of 0.5, leading to a smaller learning rate estimate. This promotes more stable and conservative training.

```python
optimizer_conservative = Prodigy(
    net.parameters(),
    lr=1.0,
    d_coef=0.5  # 0.5x smaller learning rate estimate
)
```

--------------------------------

### Prodigy Optimizer with Very High d_coef

Source: https://context7.com/konstmish/prodigy/llms.txt

Initializes the Prodigy optimizer with a very high `d_coef` of 10.0 for a significantly increased learning rate estimate. This setting is highly aggressive and should be used with extreme caution due to potential instability.

```python
optimizer_very_aggressive = Prodigy(
    net.parameters(),
    lr=1.0,
    d_coef=10.0  # 10x larger learning rate estimate
)
```

--------------------------------

### Toggle Weight Decay Strategies

Source: https://context7.com/konstmish/prodigy/llms.txt

Switch between AdamW-style decoupled weight decay and standard L2 regularization using the decouple parameter.

```python
import torch
import torch.nn as nn
from prodigyopt import Prodigy

net = SimpleNet()

# AdamW-style decoupled weight decay (default)
optimizer_adamw = Prodigy(
    net.parameters(),
    lr=1.0,
    weight_decay=0.01,
    decouple=True  # Default: AdamW style
)

# Standard L2 regularization (Adam style)
optimizer_adam = Prodigy(
    net.parameters(),
    lr=1.0,
    weight_decay=0.01,
    decouple=False  # Standard L2 penalty added to gradients
)

# Comparison of weight decay behavior
print("AdamW style (decouple=True): Weight decay applied directly to parameters")
print("Adam style (decouple=False): Weight decay added to gradients as L2 penalty")
```

--------------------------------

### Limit Learning Rate Growth

Source: https://context7.com/konstmish/prodigy/llms.txt

Use the growth_rate parameter to constrain the speed of learning rate increases, providing a stabilizing effect.

```python
import torch
import torch.nn as nn
from prodigyopt import Prodigy

net = SimpleNet()

# Unlimited growth rate (default)
optimizer_unlimited = Prodigy(
    net.parameters(),
    lr=1.0,
    growth_rate=float('inf')  # Default: no limit on growth
)
```

--------------------------------

### Cite Prodigy Paper

Source: https://github.com/konstmish/prodigy/blob/main/README.md

BibTeX citation for the Prodigy optimizer paper.

```bibtex
@inproceedings{mishchenko2024prodigy,
    title={Prodigy: An Expeditiously Adaptive Parameter-Free Learner},
    author={Mishchenko, Konstantin and Defazio, Aaron},
    booktitle={Forty-first International Conference on Machine Learning},
    year={2024},
    url={https://openreview.net/forum?id=JJpOssn0uP}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.