### Initial Approximation Setup

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 13/Семинар 13.ipynb

Sets up an initial random approximation for the optimization problem and plots it.

```python
x0 = np.random.random(n)
plot_x(x0)
```

--------------------------------

### Start Training with Configuration

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

Initiates the training process using a provided configuration object. It returns training and testing metrics.

```python
train_loss, train_acc, test_loss, test_acc = trainer(**config)
```

--------------------------------

### Instantiate LIBSVM Dataset

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

Example of how to instantiate the LIBSVM Dataset class to load the 'a9a' dataset.

```python
test = LIBSVM(root='.', dataset_name='a9a')
```

--------------------------------

### Training with DoG Optimizer

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

Example of using the DoG optimizer within a training loop. Ensure the `trainer` function and `UNetSpectrogramDenoiser` model are defined elsewhere.

```python
train_losses, test_losses, test_snr, test_si_sdr, model = trainer(
    num_epochs=1,
    batch_size=16,
    model_class=UNetSpectrogramDenoiser,
    criterion=torch.nn.MSELoss(),
    optimizer_class=DoG,
    optimizer_params={'weight_decay': 0},
    train_loader=train_loader,
    test_loader=test_loader,
    save_models=False,
    optimizer_name='DoG',
    test_on_one_batch=True
)
```

--------------------------------

### Define Weight Matrix and Number of Nodes

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 21/CvxpyScipy.ipynb

Initializes the weight matrix W for the graph and the number of nodes. This setup is required before solving the optimization problem.

```python
W = np.array(
    [
        [0, 4, 10, -3, 0, 0, 0],
        [4, 0, 0, 0, 6, 0, 0],
        [10, 0, 0, 7, 0, 0, 0],
        [-3, 0, 7, 0, 0, 0, 8],
        [0, 6, 0, 0, 0, 4, 0],
        [0, 0, 0, 0, 4, 0, 1],
        [0, 0, 0, 8, 0, 1, 0],
    ],
    dtype=np.float32,
)
num_of_nodes = 7
```

--------------------------------

### Training with AdamW Optimizer

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

This code demonstrates initiating a training process using the AdamW optimizer. It configures the model, loss, optimizer, and data loaders, similar to the Adam example but with AdamW.

```python
train_losses, test_losses, test_snr, test_si_sdr, model = trainer(
    num_epochs=1,
    batch_size=16,
    model_class=UNetSpectrogramDenoiser,
    criterion=torch.nn.MSELoss(),
    optimizer_class=AdamW,
    optimizer_params={'lr': 1e-3, 'weight_decay': 1e-2},
    train_loader=train_loader,
    test_loader=test_loader,
    save_models=False,
    optimizer_name='AdamW',
    test_on_one_batch=True
)
```

--------------------------------

### Training with COCOB Optimizer

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

Demonstrates training a model using the COCOB optimizer. This example assumes the `trainer` function and `UNetSpectrogramDenoiser` are available.

```python
train_losses, test_losses, test_snr, test_si_sdr, model = trainer(
    num_epochs=1,
    batch_size=16,
    model_class=UNetSpectrogramDenoiser,
    criterion=torch.nn.MSELoss(),
    optimizer_class=COCOB,
    optimizer_params={'weight_decay': 1e-2},
    train_loader=train_loader,
    test_loader=test_loader,
    save_models=False,
    optimizer_name='COCOB',
    test_on_one_batch=True
)
```

--------------------------------

### Use JAX JIT Decorator for Function Compilation

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

This example shows how to use the `@jax.jit` decorator to automatically compile a function upon definition. This approach is often cleaner than explicitly calling `jax.jit`. The function must be pure for `jit` to work effectively.

```python
@jax.jit
def f_2(x: np.ndarray) -> float:
    return jnp.sin(x[0] ** 2) * jnp.exp(-x.T @ x / 5)

print(f"jitted f_2 в точке (0, 0): {f_2(np.array([0., 0.]))}")
```

--------------------------------

### Aircraft Flight Optimization Problem Setup

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 21/CvxpyScipy.ipynb

Initializes arrays for distances, minimum speeds, maximum speeds, and minimum arrival times for aircraft flight segments. These arrays are used to define the constraints for an optimization problem.

```python
d = np.array([1.9501,1.2311,1.6068,1.486,1.8913,1.7621,1.4565,1.0185,1.8214,1.4447,1.6154,1.7919,1.9218,1.7382,1.1763,1.4057,1.9355,1.9169,1.4103,1.8936,1.0579,1.3529,1.8132,1.0099,1.1389,1.2028,1.1987,1.6038,1.2722,1.1988,1.0153,1.7468,1.4451,1.9318,1.466,1.4186,1.8462,1.5252,1.2026,1.6721,1.8381,1.0196,1.6813,1.3795,1.8318,1.5028,1.7095,1.4289,1.3046,1.1897,1.1934,1.6822,1.3028,1.5417,1.1509,1.6979,1.3784,1.86,1.8537,1.5936,1.4966,1.8998,1.8216,1.6449,1.818,1.6602,1.342,1.2897,1.3412,1.5341,1.7271,1.3093,1.8385,1.5681,1.3704,1.7027,1.5466,1.4449,1.6946,1.6213,1.7948,1.9568,1.5226,1.8801,1.173,1.9797,1.2714,1.2523,1.8757,1.7373,1.1365,1.0118,1.8939,1.1991,1.2987,1.6614,1.2844,1.4692,1.0648,1.9883])
smin = np.array([0.7828,0.6235,0.7155,0.534,0.6329,0.4259,0.7798,0.9604,0.7298,0.8405,0.4091,0.5798,0.9833,0.8808,0.6611,0.7678,0.9942,0.2592,0.8029,0.2503,0.6154,0.505,1.0744,0.215,0.968,1.1708,1.1901,0.9889,0.6387,0.6983,0.414,0.8435,0.52,1.1601,0.9266,0.612,0.9446,0.4679,0.6399,1.1334,0.8833,0.4126,1.0392,0.8288,0.3338,0.4071,0.8072,0.8299,0.5705,0.7751,0.6514,0.2439,0.2272,0.5127,0.2129,0.584,0.8831,0.2928,0.2353,0.8124,0.8085,0.2158,0.2164,0.3901,0.7869,0.2576,0.5676,0.8315,0.9176,0.8927,0.2841,0.6544,0.6418,0.5533,0.3536,0.8756,0.8992,0.9275,0.6784,0.7548,0.321,0.6508,0.9159,1.0928,0.4731,0.4548,1.0656,0.4324,1.0049,1.1084,0.4319,0.4393,0.2498,0.2784,0.8408,0.3909,1.0439,0.3739,0.3708,1.1943])
smax = np.array([1.9624,1.6036,1.6439,1.5641,1.7194,1.909,1.3193,1.3366,1.947,2.8803,2.5775,1.4087,1.6039,2.9266,1.4369,2.3595,3.228,1.889,2.8436,0.5701,1.1894,2.4425,2.2347,2.2957,2.7378,2.8455,2.1823,1.6209,1.2499,1.3805,1.5589,2.8554,1.8005,3.092,2.1482,1.8267,2.1459,1.5924,2.7431,1.4445,1.7781,0.8109,2.7256,2.429,2.5997,1.8125,1.9073,1.5275,2.1209,2.5419,1.7032,0.5636,1.3669,2.32,2.1006,2.7239,2.8726,1.3283,1.7769,2.575,1.4963,2.3254,1.6548,1.9537,1.5557,1.6551,2.7307,1.8018,2.5287,1.9765,1.8387,2.3525,1.7362,1.6805,1.964,2.8508,1.9424,2.078,2.1677,2.1863,2.0541,1.9734,2.7687,2.3715,1.1449,2.156,3.331,2.3456,2.712,2.3783,0.9611,2.069,1.2805,0.8585,2.2744,2.3369,2.6918,2.6728,2.5941,1.612])
tau_min = np.array([1.0809,2.7265,3.5118,5.3038,5.4516,7.1648,9.2674,12.1543,14.4058,16.6258,17.9214,19.8242,22.2333,22.4849,25.3213,28.0691,29.8751,30.6358,33.2561,34.7963,36.9943,38.261,41.1451,41.3613,43.0215,43.8974,46.4713,47.4786,49.5192,49.6795,50.7495,52.2444,53.5477,55.2351,57.085,57.425,60.1198,62.3834,64.7568,67.2016,69.2116,69.8143,70.6335,72.5122,74.1228,74.3013,74.5682,75.3821,76.6093,78.0315,80.7584,82.5472,83.534,84.9686,86.7601,87.2445,89.7329,92.6013,94.3879,94.4742,96.9105,98.7409,100.8453,101.1219,102.3966,103.5233,104.0218,106.5212,109.0372,110.392,113.2618,113.7033,116.3131,118.6214,119.9539,121.8157,124.6708,126.5908,127.3328,128.3909,128.9545,130.4264,131.6542,133.0448,134.8776,135.0912,136.034,137.8591,138.3842,140.2473,140.9852,142.7472,144.2654,145.6597,147.284,150.111,151.1363,152.3417,153.2647,154.4994])
```

--------------------------------

### Execute Gradient Descent and Plot Convergence

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 5/МатричноВекторноеДифференцирование.ipynb

This section involves running the `matrix_gd` function with generated starting matrices and selecting a learning rate (gamma) from provided options. It also includes plotting the convergence criterion against the iteration number to visualize the optimization process.

```python
# Ваше решение (Code)
```

```python
# Ваше решение (Code)
```

--------------------------------

### Iterate DataLoader and Get Batch Shape

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

Use `iter()` and `next()` to get a batch from the DataLoader. The output shape is (Batch Size, Channels, Height, Width) for data and (Batch Size,) for labels.

```python
dataloader_iterator = iter(dataloader)

data_batch, label_batch = next(dataloader_iterator)
print(data_batch.shape, label_batch.shape)
```

--------------------------------

### Initialize Model, Optimizer, and Scheduler

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

Sets up the model, optimizer, and optionally a learning rate scheduler. Ensure `model_class`, `optimizer_class`, `scheduler_class`, `optimizer_params`, and `scheduler_params` are defined before use.

```python
model = model_class.to(device)
optimizer = optimizer_class(model.parameters(), **(optimizer_params or {}))

# Инициализируем планировщик
scheduler = None
if scheduler_class is not None:
    scheduler = scheduler_class(optimizer, **(scheduler_params or {}))
```

--------------------------------

### Initialize Dataset Paths and Objects

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

Sets up the file paths for the downloaded audio data and initializes the VoiceBankDataset for both training and testing. It also defines ratios for splitting the data.

```python
base = "./data"
train_noisy = os.path.join(base, "noisy_trainset_28spk_wav")
train_clean = os.path.join(base, "clean_trainset_28spk_wav")
test_noisy  = os.path.join(base, "noisy_testset_wav")
test_clean  = os.path.join(base, "clean_testset_wav")

full_train_ds = VoiceBankDataset(train_noisy, train_clean)
full_test_ds = VoiceBankDataset(test_noisy, test_clean)

train_ratio = 0.1
test_ratio = 0.1
```

--------------------------------

### Instantiate XOR Dataset

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

This code snippet demonstrates how to create an instance of the `XORDataset` class, initializing it with 400 data points for training.

```python
# Создаем обучающий набор данных XOR размером 400 точек
train_dataset = XORDataset(size=400)
```

--------------------------------

### CGAN Training with KFACOptimizer

Source: https://context7.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/llms.txt

Example of using the KFACOptimizer for both the generator and discriminator in a CGAN trained on MNIST. This demonstrates practical application with specific optimizer parameters.

```python
# Использование для обучения CGAN на MNIST
generator, discriminator, losses = trainer(
    num_epochs=3,
    batch_size=128,
    latent_dim=100,
    generator_class=Generator,
    discriminator_class=Discriminator,
    generator_optimizer_class=KFACOptimizer,
    discriminator_optimizer_class=KFACOptimizer,
    generator_optimizer_params={
        'lr': 3e-4, 'momentum': 0.9, 'damping': 1e-4, 'factor_decay': 0.95
    },
    discriminator_optimizer_params={
        'lr': 3e-4, 'momentum': 0.9, 'damping': 1e-4, 'factor_decay': 0.95
    },
    dataset=mnist_dataset,
    device='cuda' if torch.cuda.is_available() else 'cpu',
)
print(f"Generator loss: {losses['generator'][-1]:.4f}")
print(f"Discriminator loss: {losses['discriminator'][-1]:.4f}")
```

--------------------------------

### Train Model with Shampoo Optimizer

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 10/МетодНьютона.ipynb

Initiates model training using ShampooOptimizer for both generator and discriminator. Recommended parameters for Shampoo are provided.

```python
generator, discriminator, losses = trainer(
    num_epochs=3,
    batch_size=64,
    latent_dim=100,
    generator_class=Generator,
    discriminator_class=Discriminator,
    generator_optimizer_class=ShampooOptimizer,
    discriminator_optimizer_class=ShampooOptimizer,
    generator_optimizer_params={
        'lr': 3e-4, 'momentum': 0.9, 'damping': 1e-4, 'factor_decay': 0.9, 'weight_decay': 3e-4
        },
    discriminator_optimizer_params={
        'lr': 3e-4, 'momentum': 0.9, 'damping': 1e-4, 'factor_decay': 0.9, 'weight_decay': 3e-4
        },
    dataset=dataset,
    visualize=True,
    n_disc_steps=1,
    n_gen_steps=1
)
```

--------------------------------

### Train ResNet18 with CosineAnnealingLR

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

Configures and trains a ResNet18 model on CIFAR10 using SGD optimizer and CosineAnnealingLR scheduler. This serves as an example for using the trainer function.

```python
resnet = resnet18(weights=None, num_classes=10)

config = {
    'num_epochs': 10,
    'batch_size': 128,
    'model_class': resnet,
    'criterion': nn.CrossEntropyLoss(reduction='mean'),
    'optimizer_class': SGD,
    'optimizer_params': {'lr': 1e-2, 'momentum': 0.8, 'weight_decay': 1e-4},
    'scheduler_class': optim.lr_scheduler.CosineAnnealingLR,
    'scheduler_params': {'T_max': 10, 'eta_min': 0.001},
    'dataset': CIFAR10_dataset,
    'device' : 'cuda',
}

# Запуск обучения
trained_resnet, metrics = trainer(**config)
```

--------------------------------

### Create a Tensor with a Range of Values

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

Generates a tensor containing a sequence of numbers from a specified start to end. Useful for creating index tensors or sequential data.

```python
tensor_arange = torch.arange(0, 10)
print(f'Tensor диапазона значений от 0 до 10: 
{tensor_arange}')
```

--------------------------------

### Muon Optimizer Initialization and Step

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

This snippet shows the `__init__` and `step` methods of the Muon optimizer. It outlines how parameters are set and how the optimizer updates model weights during training. The `step` method iterates through parameter groups and applies updates based on gradients and optimizer states.

```python
def __init__(self, params, lr=0.001, betas=(0.9, 0.999), weight_decay=0, ns_steps=5):
        defaults = dict(lr=lr, betas=betas, weight_decay=weight_decay, ns_steps=ns_steps)
        super().__init__(params, defaults)

    def step(self, closure=None):
        """
        Выполняет один шаг оптимизатора.

        Параметры:
            closure (Сallable): Замыкание, которое пересчитывает модель и возвращает loss
        
        Возвращает:
            loss (float): Значение функции потерь
        """
        loss = closure() if closure is not None else None

        for group in self.param_groups:
            lr = group['lr']
            beta1, beta2 = group['betas']
            weight_decay = group['weight_decay']
            ns_steps = group['ns_steps']
            
            for p in group['params']:
                if p.grad is None:
                    continue

                # YOUR CODE HERE

        return loss
```

--------------------------------

### Download Model Components

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

Downloads the 'model.py' file containing the dataset class, U-Net architecture, metrics, and trainer. This script assumes the file is available at the specified GitHub URL.

```python
url = "https://raw.githubusercontent.com/BRAIn-Lab-teaching/OPTIMIZATION-METHODS-COURSE/%D0%9F%D0%9C%D0%98_%D0%BE%D1%81%D0%B5%D0%BD%D1%8C_2025/%D0%94%D0%BE%D0%BC%D0%B0%D1%88%D0%BD%D0%B8%D0%B5%20%D0%B7%D0%B0%D0%B4%D0%B0%D0%BD%D0%B8%D1%8F/%D0%94%D0%BE%D0%BC%D0%B0%D1%88%D0%BD%D0%B5%D0%B5%20%D0%B7%D0%B0%D0%B4%D0%B0%D0%BD%D0%B8%D0%B5%2012/model.py"
!wget -O model.py "$url"

from model import VoiceBankDataset, UNetSpectrogramDenoiser, pad_and_crop_collate, trainer, extract_phase_and_chunk, istft_from_mag_phase
```

--------------------------------

### Compare Matrix Multiplication Time on CPU vs GPU

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

Compares the execution time of a matrix multiplication operation on CPU and GPU. Requires PyTorch to be installed.

```python
import time

x = torch.randn(5000, 5000)
```

--------------------------------

### GDAgent Initialization (Python)

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 16/ЗеркальныйСпуск.ipynb

Initializes the GDAgent with a learning rate gamma and the number of actions. This agent uses a gradient descent approach.

```python
class GDAgent(AbstractAgent):
    def __init__(self, gamma=0.1, n_actions=5):
        self._gamma = gamma
        self._tetas = np.zeros(n_actions)
        
    def init_actions_agent(self, n_actions):
        self._tetas = np.zeros(n_actions)
    
    def get_action(self, action, reward):
        
        # YOUR CODE HERE
    
    @property
    def name(self):
        return self.__class__.__name__ + "(gamma={})".format(self._gamma)
```

--------------------------------

### Test ClassNLLCriterion Implementation

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 5/МатричноВекторноеДифференцирование.ipynb

Initializes tests for the custom ClassNLLCriterion, comparing it against PyTorch's NLLLoss. The test setup involves generating random inputs and target labels.

```python
    def test_ClassNLLCriterion(self):
        np.random.seed(42)
        torch.manual_seed(42)

        batch_size, n_in = 2, 4
        for _ in range(100):
            torch_layer = torch.nn.NLLLoss()
            custom_layer = ClassNLLCriterion()

            layer_input = np.random.uniform(0, 1, (batch_size, n_in)).astype(np.float32)
            layer_input /= layer_input.sum(axis=-1, keepdims=True)
            layer_input = layer_input.clip(custom_layer.EPS, 1. - custom_layer.EPS)
            target_labels = np.random.choice(n_in, batch_size)
```

--------------------------------

### UCBAgent Initialization (Python)

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 16/ЗеркальныйСпуск.ipynb

Initializes the UCBAgent with a confidence parameter C. This agent uses the Upper Confidence Bound strategy for action selection.

```python
class UCBAgent(AbstractAgent):
    def __init__(self, C=1):
        self._C = C
    
    def get_action(self, action=None, reward=None):
        
        # YOUR CODE HERE
    
    @property
    def name(self):
        return self.__class__.__name__ + "(C={})".format(self._C)
```

--------------------------------

### Set Initial Parameters for Heavy Ball Method

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 4/УскоренныеОптимальныеМетоды.ipynb

Define the initial starting point `x_0` for the heavy ball optimization method using a random number generator.

```python
x_0 = rng.random(A_train.shape[1])
```

--------------------------------

### CPU Matrix Multiplication Timing

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

Measures the time taken for a matrix multiplication operation on the CPU using PyTorch. Ensure PyTorch is installed and the input tensor 'x' is defined.

```python
start_time = time.time()
_ = torch.matmul(x, x)
print(f"CPU time: {(time.time() - start_time):6.5f}s")
```

--------------------------------

### MDAgent Initialization (Python)

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 16/ЗеркальныйСпуск.ipynb

Initializes the MDAgent with a learning rate gamma and the number of actions. This agent implements a multi-armed bandit strategy.

```python
class MDAgent(AbstractAgent):
    def __init__(self, gamma=0.1, n_actions=5):
        self._gamma = gamma
        self._tetas = np.zeros(n_actions)

    def init_actions_agent(self, n_actions):
        self._tetas = np.zeros(n_actions)
        
    def get_action(self, action, reward):
        
        # YOUR CODE HERE
    
    @property
    def name(self):
        return self.__class__.__name__ + "(gamma={})".format(self._gamma)
```

--------------------------------

### PyTorch Generator Model

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 10/МетодНьютона.ipynb

Defines the Generator network architecture for a GAN using PyTorch. It takes a latent vector and labels as input to generate images. Ensure PyTorch is installed.

```python
class Generator(nn.Module):
    def __init__(self, latent_dim=100, num_classes=10, img_size=28):
        super().__init__()
        self.img_size = img_size
        self.label_emb = nn.Embedding(num_classes, latent_dim)

        self.model = nn.Sequential(
            nn.Linear(latent_dim * 2, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, img_size * img_size),
            nn.Tanh()
        )

    def forward(self, z, labels):
        label_embedding = self.label_emb(labels)
        gen_input = torch.cat((z, label_embedding), dim=1)
        img = self.model(gen_input)
        return img.view(-1, self.img_size, self.img_size)
```

--------------------------------

### Shampoo Optimizer Implementation

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 10/МетодНьютона.ipynb

Provides a PyTorch implementation of the Shampoo optimizer, including methods for computing covariance matrices and matrix inversion for adaptive learning rates.

```python
class ShampooOptimizer(torch.optim.Optimizer):
    """
    Реализация оптимайзера Shampoo.

    Параметры:
        model (nn.Module): модель
        lr (float): Скорость обучения
        momentum (float): Коэффициент моментума
        damping (float): Коэффициент устойчивости
        weight_decay (float): Коэффициент L2-регуляризации
        factor_decay (float): Коэффициент сглаживания
    """
    def __init__(self, model, lr=1e-3, momentum=0.9, damping=1e-2, weight_decay=0, factor_decay=0.95):
        defaults = dict(lr=lr, momentum=momentum, damping=damping,
                        weight_decay=weight_decay, factor_decay=factor_decay)
        super().__init__(model.parameters(), defaults)

        self.model = model
        self.param_to_module = {}

        for module in model.modules():
            if isinstance(module, nn.Linear):
                self.param_to_module[module.weight] = module

    def _compute_covariance(self, grad):
        """
        Вычисляет grad @ grad^T и grad^T @ grad
        """
        d_out, d_in = grad.shape
        L = grad @ grad.t() / d_in
        R = grad.t() @ grad / d_out
        return L, R
    
    def matrix_inverse(self, matrix, power=-0.25, damping=1e-4):
        """
        Возводит матрицу в степень.
        """
        try:
            eigvals, eigvecs = torch.linalg.eigh(matrix)
            eigvals = torch.clamp(eigvals, min=0.0)
            powered = (eigvals + damping) ** power
            return eigvecs @ torch.diag(powered) @ eigvecs.t()
        except Exception:
            return torch.eye(matrix.size(0), device=matrix.device, dtype=matrix.dtype) * (damping ** power)

    def step(self, closure=None):
        """
        Выполняет один шаг оптимизатора.

        Параметры:
            closure (Сallable): Замыкание, которое пересчитывает модель и возвращает loss
        
        Возвращает:
            loss (float): Значение функции потерь
        """
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            lr = group['lr']
            momentum = group['momentum']
            damping = group['damping']
            weight_decay = group['weight_decay']
            decay = group['factor_decay']

            for p in group['params']:
                if p.grad is None:
                    continue
                
                # YOUR CODE HERE

        return loss
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

Imports essential Python libraries for data handling, numerical operations, deep learning, and audio processing. Includes PyTorch for model building and tqdm for progress bars.

```python
import os
from tqdm import tqdm
import numpy as np
import urllib.request
import zipfile
from IPython.display import Audio, display
import random
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore")

import torch
from torch.utils.data import DataLoader, random_split
from torch.optim.optimizer import Optimizer
```

--------------------------------

### PyTorch Discriminator Model

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 10/МетодНьютона.ipynb

Defines the Discriminator network architecture for a GAN using PyTorch. It takes images and labels as input and outputs a probability of the image being real. Ensure PyTorch is installed.

```python
class Discriminator(nn.Module):
    def __init__(self, num_classes=10, img_size=28):
        super().__init__()
        self.img_size = img_size
        self.label_emb = nn.Embedding(num_classes, img_size * img_size)

        self.model = nn.Sequential(
            nn.Linear(img_size * img_size * 2, 512),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )

    def forward(self, x, labels):
        x = x.view(-1, self.img_size * self.img_size)
        label_embedding = self.label_emb(labels)
        disc_input = torch.cat((x, label_embedding), dim=1)
        return self.model(disc_input)
```

--------------------------------

### Get Learning Rate Schedule

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

A utility function to simulate and return learning rates for various PyTorch learning rate schedulers over a specified number of epochs. It requires model, optimizer, and scheduler classes with their parameters.

```python
def get_lr_schedule(model_class, optimizer_class, optimizer_params,
                    scheduler_class, scheduler_params,
                    num_epochs=100, dummy_loss=None):
    """
    Возвращает список значений learning rate для заданного планировщика.

    Параметры:
        model_class (nn.Module): Модель
        optimizer_class (optim.Optimizer): Оптимизатор
        optimizer_params (dict): Параметры оптимизатора
        scheduler_class (optim.lr_scheduler): Планировщик скорости обучения
        scheduler_params (dict): Параметры планировщика
        num_epochs (int): Количество эпох для симуляции
        dummy_loss (list or None): Список значений потерь для ReduceLROnPlateau

    Возвращает:
        lrs (list): Список learning rate на каждой итерации
    """

    model = model_class()
    optimizer = optimizer_class(model.parameters(), **optimizer_params)
    scheduler = scheduler_class(optimizer, **scheduler_params)

    lrs = []

    for i in range(num_epochs):
        if isinstance(scheduler, optim.lr_scheduler.ReduceLROnPlateau):
            loss = dummy_loss[i] if dummy_loss is not None else 1.0
            scheduler.step(loss)
        else:
            scheduler.step()

        lrs.append(optimizer.param_groups[0]['lr'])

    return lrs
```

--------------------------------

### DoG Optimizer Implementation

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

Implements the DoG (Difference of Gaussians) optimizer. Use this when manual tuning of learning rates is complex or undesirable.

```python
class DoG(Optimizer):
    """
    Реализация оптимизатора DoG.

    Параметры:
        params (Iterable): Итерируемый объект параметров для оптимизации или словарь
        d_0 (float): Начальный шаг
        weight_decay (float): Коэффициент L2-регуляризации
    """

    def __init__(self, params, d_0=1e-5, weight_decay=0):
        defaults = dict(d_0=d_0, weight_decay=weight_decay)
        super(DoG, self).__init__(params, defaults)

    def step(self, closure=None):
        """
        Выполняет один шаг оптимизатора.

        Параметры:
            closure (Сallable): Замыкание, которое пересчитывает модель и возвращает loss
        
        Возвращает:
            loss (float): Значение функции потерь
        """
        loss = closure() if closure is not None else None

        for group in self.param_groups:
            d_0 = group['d_0']
            weight_decay = group['weight_decay']
            
            for p in group['params']:
                if p.grad is None:
                    continue

                # YOUR CODE HERE

        return loss
```

--------------------------------

### FashionMNIST Visualization Utilities

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

Provides utility functions for displaying FashionMNIST images and sample images from the dataset. `imshow_fashionmnist` displays a single image, while `show_fashionmnist_samples` visualizes a grid of random samples with their corresponding class labels.

```python
fashion_classes = (
    'T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
    'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot'
) 

def imshow_fashionmnist(img, ax=None, title=None):
    npimg = img.numpy().squeeze()  # (1, H, W) → (H, W)
    if ax is None:
        plt.imshow(npimg, cmap='gray')
        if title:
            plt.title(title)
        plt.axis('off')
    else:
        ax.imshow(npimg, cmap='gray')
        if title:
            ax.set_title(title)
        ax.axis('off')

def show_fashionmnist_samples(dataset, num_samples=20, rows=4, cols=5):
    fig, axes = plt.subplots(rows, cols, figsize=(cols * 2, rows * 2))
    axes = axes.flatten()

    indices = np.random.choice(len(dataset), size=num_samples, replace=False)

    for idx, ax in zip(indices, axes):
        image, label = dataset[idx]
        imshow_fashionmnist(image, ax=ax, title=fashion_classes[label])

    for j in range(num_samples, len(axes)):
        axes[j].axis('off')

    plt.tight_layout()
    plt.show()
```

--------------------------------

### SGD Optimizer Configuration for GAN Training

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 10/МетодНьютона.ipynb

Demonstrates how to configure the SGD optimizer with specific parameters for both the generator and discriminator in a Generative Adversarial Network (GAN) training setup. Recommended parameters include lr=1e-3, momentum=0.8, and weight_decay=1e-4.

```python
generator, discriminator, losses = trainer(
    num_epochs=50,
    batch_size=128,
    latent_dim=100,
    generator_class=Generator,
    discriminator_class=Discriminator,
    generator_optimizer_class=SGD,
    discriminator_optimizer_class=SGD,
    generator_optimizer_params={'lr': 1e-3, 'momentum': 0.8, 'weight_decay': 1e-4},
    discriminator_optimizer_params={'lr': 1e-3, 'momentum': 0.8, 'weight_decay': 1e-4},
    dataset=dataset,
    device='cuda' if torch.cuda.is_available() else 'cpu',
    visualize=True,
    n_disc_steps=1,
    n_gen_steps=1
)
```

--------------------------------

### Configure and Train ResNet Model

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

Sets up configuration parameters for training a ResNet model and then trains it using a trainer function. Ensure all necessary imports and dataset configurations are in place before execution.

```python
resnet = resnet18(weights=None, num_classes=10)

config = {
    'num_epochs': 10,
    'batch_size': 128,
    'model_class': resnet,
    'criterion': nn.CrossEntropyLoss(reduction='mean'),
    'optimizer_class': SGD,
    'optimizer_params': {'lr': 1e-2, 'momentum': 0.8, 'weight_decay': 1e-4},
    'scheduler_class': ...,
    'scheduler_params': ...,
    'dataset': CIFAR10_dataset,
    'device' : 'cuda',
}

trained_resnet, metrics = trainer(**config)
```

--------------------------------

### Universal PyTorch Trainer Function

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 6/СтохастическийГрадиентныйСпуск.ipynb

A versatile function for training PyTorch models. It handles DataLoader initialization, model and optimizer setup, and provides a framework for training and testing epochs. It returns training and testing losses and accuracies.

```python
def trainer(num_epochs, batch_size, model_class, criterion, optimizer_class,
            optimizer_params, dataset, device='cuda' if torch.cuda.is_available() else 'cpu'):

    """
    Универсальная функция для обучения моделей PyTorch.

    Параметры:
        num_epochs (int): Количество эпох обучения
        batch_size (int): Размер батча для DataLoader
        model_class (nn.Module): Класс модели
        criterion (nn.Module): Функция потерь
        optimizer_class (optim.Optimizer): Класс оптимизатора
        optimizer_params (dict): Параметры оптимизатора
        dataset (Dataset): Объект датасета
        device (str): Устройство для вычислений

    Возвращает:
        tuple: (train_losses, train_accuracies, test_losses, test_accuracies)
            train_losses: значения потерь на обучении
            train_accuracies: значения accuracy на обучении
            test_losses: значения потерь на тесте
            test_accuracies: значения accuracy на тесте
    """
    # Инициализация DataLoader
    train_loader = DataLoader(dataset.train_dataset, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(dataset.test_dataset, batch_size=batch_size, shuffle=False)

    # Инициализация модели
    model = model_class().to(device)
    optimizer = optimizer_class(model.parameters(), **optimizer_params)

    # Метрики
    train_losses = []
    train_accuracies = []
    test_losses = []
    test_accuracies = []

    # Функции для train/test одной эпохи
    def train_epoch(epoch):
        model.train()
        
        # YOUR CODE HERE
        
        return train_loss, train_acc 

    def test_epoch(epoch):
        model.eval()
        
```

--------------------------------

### Create XOR Dataset

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

Initializes an XOR dataset of a specified size. This is useful for creating synthetic data for binary classification tasks.

```python
test_dataset = XORDataset(size=80)
```

--------------------------------

### Computation Graph with Multiple Leaf Nodes

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

This example demonstrates building a computation graph with multiple leaf nodes (`x1`, `x2`, `x3`). The `.backward()` method computes gradients for all leaf nodes involved in the computation of `z`.

```python
x1 = torch.randn(1, 1, requires_grad=True)
x2 = torch.randn(1, 1, requires_grad=True)
x3 = torch.randn(1, 1, requires_grad=True)
print(f'Листья: {x1}, {x2}, {x3}')

y1 = x1 * x2
y2 = x1 + x3

z = (y1 - y2)**2
print(f'Корень: {z}')
```

--------------------------------

### Universal PyTorch Trainer Function

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 14/АлгоритмФранкВульфа.ipynb

A flexible function for training PyTorch models. It handles data loading, model setup, optimization with custom constraints, and metric tracking for both training and testing epochs. Ensure necessary PyTorch modules and time are imported.

```python
def trainer(num_epochs, batch_size, model_class, criterion, optimizer_class=SFW, optimizer_params=None,
            constraintList=None, dataset=None, device='cuda' if torch.cuda.is_available() else 'cpu'
           ):
    """
    Универсальная функция для обучения моделей PyTorch.

    Параметры:
        num_epochs (int): Количество эпох обучения
        batch_size (int): Размер батча для DataLoader
        model_class (nn.Module): Класс модели
        criterion (nn.Module): Функция потерь
        optimizer_class (optim.Optimizer): Класс оптимизатора
        optimizer_params (dict): Параметры оптимизатора
        constraintList (list): Список ограничений
        dataset (Dataset): Объект датасета
        device (str): Устройство для вычислений

    Возвращает:
        model (nn.Module): Обученная модель
        metrics (dict): Словарь с логами
    """
    
    # Создаем загрузчики данных
    train_loader = DataLoader(dataset.train_dataset, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(dataset.test_dataset, batch_size=batch_size, shuffle=False)
    
    model = model_class.to(device)

    make_feasible(constraintList)
    param_groups = []
    for constraint, param_list in constraintList:
        param_group = {'params': param_list, 'constraint': constraint}
        if optimizer_params:
             param_group.update(optimizer_params)
        param_groups.append(param_group)
    optimizer = optimizer_class(param_groups)

    metrics = {
        "train_loss": [],
        "train_acc": [],
        "test_loss": [],
        "test_acc": [],
    }
    
    def train_epoch():
        model.train()
        
        # YOUR CODE HERE

        return train_loss, train_acc
    
    def test_epoch():
        model.eval()
        
        # YOUR CODE HERE

        return test_loss, test_acc
    
    for epoch in range(num_epochs):
        start = time.time()
        
        train_loss, train_acc = train_epoch()
        test_loss, test_acc = test_epoch()
        
        # Сохраняем метрики
        metrics["train_loss"].append(train_loss)
        metrics["train_acc"].append(train_acc)
        metrics["test_loss"].append(test_loss)
        metrics["test_acc"].append(test_acc)

        elapsed = time.time() - start

        print(f"Epoch {epoch+1}/{num_epochs} | "
              f"Train Loss: {train_loss:.4f}, Acc: {train_acc:.2f}% | "
              f"Test Loss: {test_loss:.4f}, Acc: {test_acc:.2f}% | "
              f"Time: {elapsed:.2f}s")
    return model, metrics
```

--------------------------------

### JIT Compile and Call a Function in JAX

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

This snippet demonstrates how to apply JAX's `jit` transformation to a function and then call the compiled version. Ensure the function `f_2` and `jax.numpy` (`jnp`) are imported.

```python
f_2_jitted = jax.jit(f_2)
print(f"jitted f_2 в точке (0, 0): {f_2_jitted(np.array([0., 0.]))}")
```

--------------------------------

### Initialize Model and Loss Function

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Семинары/Семинар 4/Семинар 4.ipynb

Initializes a neural network model for a 2D to 1D XOR problem and sets up the Mean Squared Error (MSE) loss function for regression tasks.

```python
# Инициализация модели для XOR (2D -> 1D)
model = Net(in_dim=2, out_dim=1).to(device)
# MSE лосс для регрессии
criterion = nn.MSELoss(reduction='mean')
```

--------------------------------

### Comparing Optimizers with Adam

Source: https://github.com/open-education-club-by-yandex/optimization-methods-course-brain-lab-mipt.git/blob/yc-public/Домашние задания/Домашнее задание 12/АдаптивныеМетоды.ipynb

Sets up a training loop to compare adaptive optimizers with Adam. Ensure the number of epochs is set to 5 for a meaningful comparison.

```python

```