### Main Experiment Setup and Run

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/gmlp/experiment.html

Initializes and runs the gMLP experiment, setting up configurations, models, and starting the training process.

```python
def main():
    experiment.create(name="gMLP")

    conf = Configs()
    experiment.configs(conf, {
        'tokenizer': 'character',
        'prompt_separator': '',
        'prompt': 'It is ',
        'text': 'tiny_shakespeare',
        'seq_len': 256,
        'epochs': 128,
        'batch_size': 32,
        'inner_iterations': 10,
        'd_model': 512,
        'd_ffn': 2048,
        'optimizer.optimizer': 'Noam',
        'optimizer.learning_rate': 1.,
    })

    experiment.add_pytorch_models({'model': conf.model})

    with experiment.start():
        conf.run()
```

```python
if __name__ == '__main__':
    main()
```

--------------------------------

### Main Experiment Setup and Run

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/rl/dqn/experiment.html

Sets up and runs the DQN experiment. It defines configurations, initializes the trainer, starts the experiment, and then destroys the trainer.

```python
def main():
    experiment.create(name='dqn')

    configs = {
        'updates': 1_000_000,
        'epochs': 8,
        'n_workers': 8,
        'worker_steps': 4,
        'mini_batch_size': 32,
        'update_target_model': 250,
        'learning_rate': FloatDynamicHyperParam(1e-4, (0, 1e-3)),
    }

    experiment.configs(configs)

    m = Trainer(**configs)

    with experiment.start():
        m.run_training_loop()

    m.destroy()
```

--------------------------------

### Main Experiment Setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/vit/experiment.html

Sets up and runs the experiment. This includes creating the experiment, loading configurations, defining models, and starting the training loop.

```python
def main():
    experiment.create(name='ViT', comment='cifar10')

    conf = Configs()
    experiment.configs(conf, {
        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 2.5e-4,

        'transformer.d_model': 512,

        'epochs': 32,
        'train_batch_size': 64,

        'train_dataset': 'cifar10_train_augmented',

        'valid_dataset': 'cifar10_valid_no_augment',
    })

    experiment.add_pytorch_models({'model': conf.model})

    with experiment.start():
        conf.run()
```

--------------------------------

### Main Experiment Setup and Execution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/unet/experiment.html

Sets up and runs the U-Net experiment. This includes creating the experiment, configuring it, initializing models, and starting the training loop.

```python
def main():
    experiment.create(name='unet')
    configs = Configs()
    experiment.configs(configs, {})
    configs.init()
    experiment.add_pytorch_models({'model': configs.model})
    with experiment.start():
        configs.run()
```

--------------------------------

### LabML Experiment Setup and Run

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/resnet/experiment.html

Sets up a LabML experiment, loads configurations, adds the PyTorch model, and starts the experiment to run the training loop.

```python
def main():
    experiment.create(name='resnet', comment='cifar10')
    conf = Configs()
    experiment.configs(conf, {
        'bottlenecks': [8, 16, 16],
        'n_blocks': [6, 6, 6],

        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 2.5e-4,

        'epochs': 500,
        'train_batch_size': 256,

        'train_dataset': 'cifar10_train_augmented',
        'valid_dataset': 'cifar10_valid_no_augment',
    })
    experiment.add_pytorch_models({'model': conf.model})
    with experiment.start():
        conf.run()
```

--------------------------------

### GATv2 Experiment Setup and Run

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/graphs/gatv2/experiment.html

Sets up and runs a GATv2 experiment. This includes creating the experiment, configuring hyperparameters like optimizer, learning rate, and dropout, and then starting the training run.

```python
def main():
    # Create configurations
    conf = Configs()
    # Create an experiment
    experiment.create(name='gatv2')
    # Calculate configurations.
    experiment.configs(conf, {
        # Adam optimizer
        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 5e-3,
        'optimizer.weight_decay': 5e-4,

        'dropout': 0.7,
    })

    # Start and watch the experiment
    with experiment.start():
        # Run the training
        conf.run()

if __name__ == '__main__':
    main()
```

--------------------------------

### Main Experiment Setup and Execution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/rope/experiment.html

Sets up and runs the RoPE transformer experiment. This includes creating the experiment, defining configurations, overriding defaults, and starting the training run.

```python
42def main():

    # Create experiment
    experiment.create(name="rotary_pe_transformer", writers={'screen'})

    # Create configs
    conf = Configs()

    # Override configurations
    experiment.configs(conf, {

        # No fixed positional embeddings
        'transformer.src_embed': 'no_pos',
        'transformer.tgt_embed': 'no_pos',

        # Encoder with RoPE
        'transformer.encoder_attn': 'rotary',

        'model': 'rotary_pe_transformer',

        # Use character level tokenizer
        'tokenizer': 'character',

        # Prompt separator is blank
        'prompt_separator': '',

        # Starting prompt for sampling
        'prompt': 'It is ',

        # Use Tiny Shakespeare dataset
        'text': 'tiny_shakespeare',

        # Use a context size of 256
        'seq_len': 512,

        # Train for 32 epochs
        'epochs': 32,

        # Batch size 4
        'batch_size': 4,

        # Switch between training and validation for 10 times per epoch
        'inner_iterations': 10,

        # Model size
        'd_model': 128,
        'transformer.ffn.d_ff': 512,
        'transformer.n_heads': 16,
        'transformer.dropout': 0.0,

        # Use [Noam optimizer](../../optimizers/noam.html)
        'optimizer.optimizer': 'Noam',
        'optimizer.learning_rate': 1.,
        
        'dataloader_shuffle_with_replacement': True
    })

    # Set models for saving and loading
    experiment.add_pytorch_models({'model': conf.model})

    # Start the experiment
    with experiment.start():
        # Run training
        conf.run()

101if __name__ == '__main__':
    main()
```

--------------------------------

### Main Experiment Setup and Execution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/alibi/experiment.html

Sets up and runs a GPT ALiBi experiment. This includes creating the experiment, defining configurations, adding PyTorch models, and starting the training run. Use this as the entry point for launching the experiment.

```python
107def main():

    experiment.create(name="gpt_alibi")

    conf = Configs()

    experiment.configs(conf, {

        'tokenizer': 'character',

        'prompt_separator': '',

        'prompt': 'It is ',

        'text': 'tiny_shakespeare',

        'text': 'tiny_shakespeare_no_split',

        'seq_len': 64,

        'valid_seq_len': 80,

        'epochs': 128,

        'batch_size': 128,

        'inner_iterations': 10,

        'transformer.d_model': 128,
        'transformer.ffn.d_ff': 512,
        'transformer.n_heads': 8,
        'transformer.n_layers': 4,
        'transformer.dropout': 0.1,
    })

    experiment.add_pytorch_models({'model': conf.model})

    with experiment.start():
        conf.run()

if __name__ == '__main__':
    main()
```

--------------------------------

### Experiment Setup and Execution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/normalization/group_norm/experiment.html

Sets up a new experiment named 'cifar10' with a comment 'group norm'. It then loads configurations, including optimizer settings, and starts the experiment to run the training loop.

```python
def main():
    # Create experiment
    experiment.create(name='cifar10', comment='group norm')

    # Create configurations
    conf = Configs()

    # Load configurations
    experiment.configs(conf, {
        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 2.5e-4,
    })

    # Start the experiment and run the training loop
    with experiment.start():
        conf.run()
```

--------------------------------

### Experiment Setup and Execution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/fast_weights/experiment.html

Sets up and runs a machine learning experiment using the labml library. It defines configurations, registers models, and starts the training loop for the fast weights transformer.

```python
def main():
    experiment.create(name="fast_weights_transformer")

    conf = Configs()

    experiment.configs(conf,
                       {'tokenizer': 'character',
                        'text': 'tiny_shakespeare',
                        'optimizer.learning_rate': 1.0,
                        'optimizer.optimizer': 'Noam',
                        'prompt': 'It is',
                        'prompt_separator': '',

                        'train_loader': 'shuffled_train_loader',
                        'valid_loader': 'shuffled_valid_loader',

                        'seq_len': 128,
                        'epochs': 128,
                        'batch_size': 16,
                        'inner_iterations': 25})

    experiment.add_pytorch_models(get_modules(conf))

    with experiment.start():
        conf.run()


if __name__ == '__main__':
    main()
```

--------------------------------

### Start experiment and run training

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb

Starts the experiment and executes the training loop defined in `configs.run()`. This block should be used to begin the actual training process.

```python
# Start the experiment
with experiment.start():
    configs.run()
```

--------------------------------

### Train CycleGAN Experiment Setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/gan/cycle_gan/index.html

Sets up and runs a CycleGAN training experiment. Includes configuration loading, experiment creation, model registration, and starting the training run.

```python
def train():
    conf = Configs()

    experiment.create(name='cycle_gan')

    experiment.configs(conf, {'dataset_name': 'summer2winter_yosemite'})
    conf.initialize()

    experiment.add_pytorch_models(get_modules(conf))

    with experiment.start():
        conf.run()
```

--------------------------------

### Main Experiment Setup and Execution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/rope/value_pe/arithmetic_experiment.html

Sets up and runs the RoPER addition experiment. This includes creating the experiment, defining configurations, overriding parameters, and starting the training run.

```python
def main():
    experiment.create(name="roper_addition", comment="rotary value 7", writers={'screen', 'labml'})

    conf = Configs()
    experiment.configs(conf, {
        'max_digits': 7,
        'transformer.src_embed': 'no_pos',
        'transformer.tgt_embed': 'no_pos',
        'transformer.encoder_attn': 'rotary_value',
        'model': 'rotary_pe_transformer',
        'seq_len': 512,
        'epochs': 20,
        'batch_size': 16,
        'd_model': 128,
        'transformer.ffn.d_ff': 512,
        'transformer.n_heads': 4,
        'transformer.dropout': 0.0,
        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 2.5e-4,
    })

    experiment.add_pytorch_models({'model': conf.model})

    with experiment.start():
        conf.run()
```

```python
if __name__ == '__main__':
    main()
```

--------------------------------

### WGAN-GP Experiment Setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/gan/wasserstein/gradient_penalty/experiment.html

Sets up and runs a WGAN-GP experiment with MNIST. This includes creating configurations, defining experiment parameters, and starting the training loop.

```python
def main():
    # Create configs object
    conf = Configs()

    # Create experiment
    experiment.create(name='mnist_wassertein_gp_dcgan')

    # Override configurations
    experiment.configs(conf,
                       {
                           'discriminator': 'cnn',
                           'generator': 'cnn',
                           'label_smoothing': 0.01,
                           'generator_loss': 'wasserstein',
                           'discriminator_loss': 'wasserstein',
                           'discriminator_k': 5,
                       })

    # Start the experiment and run training loop
    with experiment.start():
        conf.run()


if __name__ == '__main__':
    main()
```

--------------------------------

### Initialization and Tracking Setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/gan/original/experiment.html

Initializes the state modules and sets up scalar and image tracking for generator and discriminator losses, as well as generated images. This is typically called once during setup.

```python
def init(self):
    self.state_modules = []

    tracker.set_scalar("loss.generator.*", True)
    tracker.set_scalar("loss.discriminator.*", True)
    tracker.set_image("generated", True, 1 / 100)
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/glu_variants/experiment.html

Sets up a new experiment, defines configurations, and overrides default parameters for training.

```python
def main():
    experiment.create(name="glu_variants")

    conf = Configs()
    experiment.configs(conf,
                       {'tokenizer': 'character',
                        'prompt_separator': '',
                        'prompt': 'It is ',
                        'text': 'tiny_shakespeare',

                        'optimizer.optimizer': 'Noam',
                        'optimizer.learning_rate': 1.,
                        'optimizer.d_model': 256,

                        'seq_len': 1024,
                        'epochs': 128,
                        'batch_size': 6,
                        'inner_iterations': 10,

                        'transformer.ffn.glu_variant': 'Bilinear',

                        'transformer.d_model': 256,
                        'transformer.ffn.d_ff': 1024,
                        'transformer.n_heads': 8,
                        'transformer.n_layers': 6})

    conf.n_tokens = conf.text.n_tokens
```

--------------------------------

### Start and Run Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/activations/fta/experiment.html

Initiates the experiment context and executes the training process defined in the configurations.

```python
with experiment.start():

        conf.run()
```

--------------------------------

### Main Experiment Setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/mlm/experiment.html

Sets up and runs the MLM experiment, including configuration overrides and model saving.

```python
experiment.create(name="mlm")

conf = Configs()

experiment.configs(conf, {
    'batch_size': 64,
    'seq_len': 32,
    'epochs': 1024,
    'inner_iterations': 1,
    'd_model': 128,
    'transformer.ffn.d_ff': 256,
    'transformer.n_heads': 8,
    'transformer.n_layers': 6,
    'optimizer.optimizer': 'Noam',
    'optimizer.learning_rate': 1.,
})

experiment.add_pytorch_models({'model': conf.model})

with experiment.start():
    conf.run()
```

--------------------------------

### Main execution function

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/glu_variants/simple.html

This function sets up and starts the experiment. It creates an experiment, loads configurations, initializes the trainer, adds PyTorch models to the experiment, and then starts the training process.

```python
def main():
    experiment.create(name="glu_variants")
    configs = Configs()
    experiment.configs(dataclasses.asdict(configs))
    trainer = Trainer(configs)
    experiment.add_pytorch_models({'model': trainer.model})
    with experiment.start():
        trainer.train()
```

--------------------------------

### Start Experiment and Run Training

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb

Starts the experiment context and executes the training loop defined in the configuration's 'run' method. This initiates the model training process.

```python
# Start the experiment
with experiment.start():
    conf.run()
```

--------------------------------

### Start experiment and train

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/glu_variants/simple.ipynb

Begin the experiment and execute the training loop. This starts the model training process.

```python
with experiment.start():
    trainer.train()
```

--------------------------------

### Install labml-nn

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/cfr/kuhn/experiment.ipynb

Installs the labml-nn library. This is a prerequisite for running the experiment.

```python
%%capture
!pip install labml-nn
```

--------------------------------

### Start and Run Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/hypernetworks/experiment.html

Starts the experiment and executes the training and validation run using the configured parameters. The `conf.run()` method is called within the experiment context.

```python
    with experiment.start():
        conf.run()
```

--------------------------------

### Synthetic experiment setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/optimizers/amsgrad.html

Sets up the synthetic experiment by defining the parameter x and the optimal x_star.

```python
def _synthetic_experiment(is_adam: bool):
    x = nn.Parameter(torch.tensor([.0]))
    x_star = nn.Parameter(torch.tensor([-1]), requires_grad=False)
```

--------------------------------

### Install labml-nn

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/capsule_networks/mnist.ipynb

Install the labml-nn library using pip. This is a prerequisite for running the experiment.

```python
!pip install labml-nn
```

--------------------------------

### Experiment Recording Setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/optimizers/amsgrad.html

Sets up an experiment recording context using `labml.experiment.record`. This allows tracking metrics and parameters during the experiment.

```python
with experiment.record(name='synthetic', comment='Adam' if is_adam else 'AMSGrad'):
    # Training loop content
```

--------------------------------

### Run the experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/hypernetworks/experiment.ipynb

Start the labml experiment and execute the training process defined in the configurations.

```python
# Start the experiment
with experiment.start():
    # `TrainValidConfigs.run`
    conf.run()
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/zh/transformers/basic/with_sophia.html

This section demonstrates how to set up and configure an experiment, including defining model parameters, optimizer settings, and dataset choices.

```python
def main():
    experiment.create(name="transformer")
    conf = Configs()
    experiment.configs(conf, {
        'tokenizer': 'character',
        'prompt_separator': '',
        'prompt': 'It is ',
        'text': 'tiny_shakespeare',
        'seq_len': 512,
        'epochs': 32,
        'batch_size': 16,
        'inner_iterations': 10,
        'd_model': 256,
        'transformer.n_heads': 16,
        'transformer.ffn.d_ff': 1024,
        'optimizer.optimizer': 'Sophia',
        'optimizer.learning_rate': 3e-4,
        'optimizer.rho': 0.03,
    })

    experiment.add_pytorch_models({'model': conf.model})

    with experiment.start():
        conf.run()
```

--------------------------------

### Run the Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/rope/value_pe/experiment.html

Starts the experiment and runs the training process defined in the configurations. This block executes the main training loop.

```python
89    with experiment.start():

91        conf.run()
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/hour_glass/experiment.html

Main function to set up and run the experiment. It creates the experiment, initializes configurations, and overrides specific settings like the tokenizer and prompt separator.

```python
def main():
    experiment.create(name="hour_glass")
    conf = Configs()
    experiment.configs(conf, {
        'tokenizer': 'character',
        'prompt_separator': '',
    })
```

--------------------------------

### Configure and Run Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/adaptive_computation/ponder_net/experiment.html

Sets up and starts a PonderNet experiment using labml.ai. It configures the optimizer and learning rate before running the experiment.

```python
experiment.create(name='ponder_net')

conf = Configs()
experiment.configs(conf, {
    'optimizer.optimizer': 'Adam',
    'optimizer.learning_rate': 0.0003,
})

with experiment.start():
    conf.run()
```

--------------------------------

### Start and Run Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/RWKV/experiment.html

Initiates the experiment and executes the training process. This block should be used within the main execution flow of your script.

```python
with experiment.start():
    conf.run()
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/knn/train_model.html

Sets up a new experiment named 'knn_lm' and initializes the experiment configurations. This includes defining various parameters like tokenizer, prompt, text dataset, optimizer settings, sequence length, epochs, batch size, and inner iterations.

```python
def main():
    # Create experiment
    experiment.create(name="knn_lm")

    # Create configs
    conf = Configs()

    # Load configurations
    experiment.configs(conf,
                       # A dictionary of configurations to override
                       {'tokenizer': 'character',
                        'prompt_separator': '',
                        'prompt': 'It is ',
                        'text': 'tiny_shakespeare',

                        'optimizer.optimizer': 'Noam',
                        'optimizer.learning_rate': 1.,
                        'optimizer.d_model': 256,

                        'seq_len': 1024,
                        'epochs': 128,
                        'batch_size': 6,
                        'inner_iterations': 10,

                        # Transformer configurations
                        })
```

--------------------------------

### Sampling Function for Text Generation

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/experiments/nlp_autoregression.html

Generates text samples periodically during training. It takes a starting prompt, tokenizes it, gets model predictions, and appends generated tokens to the prompt.

```python
def sample(self):
        prompt = self.prompt
        log = [(prompt, Text.subtle)]
        for i in monit.iterate('Sample', 25):
            data = self.text.text_to_i(prompt).unsqueeze(-1)
            data = data.to(self.device)
            output, *_ = self.model(data)
            output = output.argmax(dim=-1).squeeze()
            prompt += self.prompt_separator + self.text.itos[output[-1]]
            log += [(self.prompt_separator + self.text.itos[output[-1]], Text.value)]
        tracker.add({'sampled': prompt})
        logger.log(log)
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/normalization/weight_standardization/experiment.html

Sets up a new experiment named 'cifar10' with a comment, loads configurations, and specifies optimizer details and training batch size.

```python
def main():
    experiment.create(name='cifar10', comment='weight standardization')
    conf = CIFAR10Configs()
    experiment.configs(conf, {
        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 2.5e-4,
        'train_batch_size': 64,
    })
    with experiment.start():
        conf.run()
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/feedback/experiment.html

Sets up a new experiment named 'feedback_transformer' and loads configurations, overriding default parameters for tokenizer, text dataset, optimizer, and prompt.

```python
experiment.create(name="feedback_transformer")
conf = Configs()
experiment.configs(conf,
                   {'tokenizer': 'character',
                    'text': 'tiny_shakespeare',
                    'optimizer.learning_rate': 1.0,
                    'optimizer.optimizer': 'Noam',
                    'prompt': 'It is',
                    'prompt_separator': ''})
```

--------------------------------

### Run MNIST Experiment Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/si/optimizers/mnist_experiment.html

This Python snippet configures and runs an MNIST experiment. It sets the learning rate, initializes the experiment with a model, and starts the run. Ensure labml is installed and configured.

```python
conf.optimizer.learning_rate = 1.5e-4
conf.set_seed.set()
experiment.add_pytorch_models(dict(model=conf.model))
with experiment.start():
    conf.run()
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/aft/experiment.html

Initializes an experiment, creates configurations, and overrides default parameters for tokenizer, prompt, dataset, sequence length, epochs, batch size, and model dimensions.

```python
experiment.create(name="aft")
conf = Configs()
experiment.configs(conf, {
    'tokenizer': 'character',
    'prompt_separator': '',
    'prompt': 'It is ',
    'text': 'tiny_shakespeare',
    'seq_len': 256,
    'epochs': 128,
    'batch_size': 32,
    'inner_iterations': 10,
    'd_model': 128,
    'transformer.ffn.d_ff': 256,
    'optimizer.optimizer': 'Noam',
    'optimizer.learning_rate': 1.,
})
```

--------------------------------

### Experiment Configuration and Run

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/sketch_rnn/index.html

Sets up the experiment configuration, including optimizer, learning rate, dataset name, and inner iterations, then runs the experiment.

```python
configs = Configs()
experiment.create(name="sketch_rnn")
experiment.configs(configs, {
    'optimizer.optimizer': 'Adam',
    'optimizer.learning_rate': 1e-3,
    'dataset_name': 'bicycle',
    'inner_iterations': 10
})
with experiment.start():
    configs.run()
```

--------------------------------

### Install labml-nn Package

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/readme.md

Install the labml-nn package using pip. This command installs the library and its dependencies.

```bash
pip install labml-nn
```

--------------------------------

### Initialize configurations

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/gan/cycle_gan/experiment.ipynb

Initialize the configurations for the experiment. This step prepares the configuration object for use.

```python
conf.initialize()
```

--------------------------------

### Create and Configure Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/activations/fta/experiment.html

Sets up a new experiment with a specified name and writers, then loads and overrides default configurations for the experiment.

```python
experiment.create(name="fta", writers={'screen', 'labml'})
```

```python
conf = Configs()
```

```python
experiment.configs(conf, {

        'tokenizer': 'character',

        'prompt_separator': '',

        'prompt': 'It is ',

        'text': 'tiny_shakespeare',

        'seq_len': 256,

        'epochs': 32,

        'batch_size': 16,

        'inner_iterations': 10,

        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 3e-4,
    })
```

--------------------------------

### Install labml-nn Package

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb

Installs the labml-nn library quietly. This is a prerequisite for running the experiment.

```python
!pip install labml-nn --quiet
```

--------------------------------

### Main Finetuning Script

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/neox/samples/finetune.html

Sets up and runs a finetuning experiment for a Neox model. It includes creating an experiment, initializing configurations, starting the experiment, initializing the model, and training for a specified number of epochs.

```python
99def main():

101    experiment.create(name='pipe_neox_biases',
102                      writers={'screen', 'web_api'})

105    conf = PipelineParallelTrainerConf()
106    experiment.configs(conf, {
107        'learning_rate': 3e-4,
108        'is_checkpointing': False,
109        'max_seq_len': 128,
110        'batch_size': 64,
111        'chunks': 8,
112    })

115    with experiment.start():

117        _ = conf.model

120        for epoch in monit.loop(conf.epochs):
121            conf.train_epoch()
122            tracker.new_line()
123            torch.save(conf.fine_tuner.state_dict(), str(lab.get_data_path() / 'fine_tune.pt'))

127if __name__ == '__main__':
128    main()
```

--------------------------------

### Start experiment and run training

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/cfr/kuhn/experiment.ipynb

Starts the experiment context and initiates the CFR training loop by calling the iterate method on the configuration object.

```python
# Start the experiment
with experiment.start():
    conf.cfr.iterate()
```

--------------------------------

### Initialize and Run PPO Trainer

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/rl/ppo/experiment.html

Initializes the PPO trainer with the specified configurations and starts the experiment's training loop within a managed context.

```python
m = Trainer(**configs)
with experiment.start():
    m.run_training_loop()
```

--------------------------------

### Initialize configurations

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/capsule_networks/mnist.ipynb

Create an instance of the experiment's configurations. This object holds all parameters for the training run.

```python
conf = Configs()
```

--------------------------------

### Start New Line for Tracker

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/gan/cycle_gan/index.html

Instructs the tracker to start a new line for logging or statistics. This is often used to separate output from different epochs or training phases.

```python
tracker.new_line()
```

--------------------------------

### Create PPO Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb

Initializes a new experiment named 'ppo'. This sets up the logging and tracking for the training run.

```python
experiment.create(name="ppo")
```

--------------------------------

### Get module from ModuleList

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/jax_transformer/index.html

Retrieves the module at the specified index from the ModuleList.

```python
def __getitem__(self, idx: int) -> M:
    return self._submodules[idx]
```

--------------------------------

### Trainer Configurations

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/lora/experiment.html

Defines the base configurations for the trainer, which can be overridden when an experiment starts.

```python
class Trainer(BaseConfigs):
```

--------------------------------

### ModuleList: Get the idx-th module

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/jax_transformer/index.html

Retrieves the module at the specified index from the ModuleList.

```APIDOC
## __getitem__

### Description
Get the `idx`-th module from the list.

### Signature
`def __getitem__(self, idx: int) -> M:`

### Parameters
- **idx** (int) - The index of the module to retrieve.

### Returns
The module at the specified index.
```

--------------------------------

### Experiment Setup and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/normalization/batch_norm/cifar10.html

Sets up and configures a new experiment named 'cifar10' with a comment 'batch norm'. It then loads configurations for the optimizer, learning rate, and training batch size.

```python
experiment.create(name='cifar10', comment='batch norm')

conf = CIFAR10Configs()

experiment.configs(conf, {
    'optimizer.optimizer': 'Adam',
    'optimizer.learning_rate': 2.5e-4,
    'train_batch_size': 64,
})
```

--------------------------------

### StrokesDataset Get Item

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/sketch_rnn/index.html

Retrieves a data sample and its corresponding mask by index.

```python
def __getitem__(self, idx: int):
    return self.data[idx], self.mask[idx]
```

--------------------------------

### ModuleList: Get all parameters

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/jax_transformer/index.html

Collects all parameters from the ModuleList and its submodules, formatting keys as `index/param_name`.

```APIDOC
## get_params

### Description
Collects all parameters from the ModuleList and its submodules.

### Signature
`def get_params(self):`

### Returns
A dictionary containing all parameters with keys formatted as `index/param_name`.
```

--------------------------------

### Main Experiment Execution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/uncertainty/evidence/experiment.html

Sets up and runs the experiment. This includes creating the experiment, defining configurations for the model, optimizer, and loss functions, and starting the training loop.

```python
def main():
    experiment.create(name='evidence_mnist')
    conf = Configs()
    experiment.configs(conf, {
        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 0.001,
        'optimizer.weight_decay': 0.005,
        'loss_func': 'max_likelihood_loss',
        'loss_func': 'cross_entropy_bayes_risk',
        'loss_func': 'squared_error_bayes_risk',
        'outputs_to_evidence': 'softplus',
        'dropout': 0.5,
    })
    with experiment.start():
        conf.run()
```

```python
if __name__ == '__main__':
    main()
```

--------------------------------

### Main Experiment Setup

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/fnet/experiment.html

Sets up and runs a transformer experiment named 'fnet'. It configures parameters like tokenizer, epochs, inner iterations, transformer dimensions, attention mechanism, and optimizer.

```python
112def main():

    experiment.create(name="fnet")

    conf = Configs()

    experiment.configs(conf, {

        'tokenizer': 'basic_english',

        'epochs': 32,

        'inner_iterations': 10,

        'transformer.d_model': 512,
        'transformer.ffn.d_ff': 2048,
        'transformer.n_heads': 8,
        'transformer.n_layers': 6,

        'transformer.encoder_attn': 'fnet_mix',

        'optimizer.optimizer': 'Noam',
        'optimizer.learning_rate': 1.,
    })

    experiment.add_pytorch_models({'model': conf.model})

    with experiment.start():
        conf.run()
```

```python
153if __name__ == '__main__':
    main()
```

--------------------------------

### Get number of modules in ModuleList

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/jax_transformer/index.html

Returns the total number of modules stored in the ModuleList.

```python
def __len__(self):
    return len(self._submodules)
```

--------------------------------

### Get Trainable Parameters

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/neox/utils/finetune.html

Retrieves all trainable parameters from the layers, prefixed by their layer index.

```python
def get_trainable_params(self) -> Dict[str, nn.Parameter]:
    params = {}
    for i, layer in enumerate(self.layers):
        params.update(self.get_layer_trainable_params(layer, prefix=f'layer_{i:02d}'))

    return params
```

--------------------------------

### Initialize Tracker Configurations

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/experiments/mnist.html

Sets up scalar tracking for loss and accuracy metrics within the experiment.

```python
tracker.set_scalar("loss.*", True)
tracker.set_scalar("accuracy.*", True)
```

--------------------------------

### Initialize configurations

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb

Initializes the configurations after they have been set.

```python
configs.init()
```

--------------------------------

### Get Device from DeviceInfo

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/helpers/device.html

An option function that returns the torch.device object from the DeviceInfo instance.

```python
@option(DeviceConfigs.device)
def _device(c: DeviceConfigs):
    return c.device_info.device
```

--------------------------------

### Import bitsandbytes

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/neox/utils/llm_int8.html

Imports necessary components from the bitsandbytes library. Raises an ImportError if the package is not installed.

```python
try:
    from bitsandbytes.nn import Linear8bitLt, Int8Params
except ImportError:
    raise ImportError('''Please install `bitsandbytes` with `pip install bitsandbytes -U`''')

import torch
from torch import nn
```

--------------------------------

### Experiment Initialization and Configuration

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/diffusion/ddpm/experiment.html

Sets up a new experiment, defines configurations, and overrides default values. This includes specifying the dataset, image channels, and training epochs.

```python
def main():
    experiment.create(name='diffuse', writers={'screen', 'labml'})

    configs = Configs()
    experiment.configs(configs, {
        'dataset': 'CelebA',  # 'MNIST'
        'image_channels': 3,  # 1,
        'epochs': 100,  # 5,
    })

    configs.init()

    experiment.add_pytorch_models({'eps_model': configs.eps_model})

    with experiment.start():
        configs.run()
```

--------------------------------

### Import necessary modules

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/weight_standardization/experiment.ipynb

Import the experiment module from labml and the CIFAR10Configs class for experiment setup.

```python
from labml import experiment
from labml_nn.normalization.weight_standardization.experiment import CIFAR10Configs as Configs
```

--------------------------------

### Create and Configure Primer EZ Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/primer_ez/experiment.html

Sets up a new experiment named 'primer_ez' and configures it with specific parameters including tokenizer, dataset, sequence length, batch size, and optimizer settings.

```python
70def main():

72    experiment.create(name="primer_ez")

74    conf = Configs()

76    experiment.configs(conf, {

78        'tokenizer': 'character',

80        'prompt_separator': '',

82        'prompt': 'It is ',

84        'text': 'tiny_shakespeare',

87        'seq_len': 256,

89        'epochs': 128,

91        'batch_size': 32,

94        'inner_iterations': 10,

97        'd_model': 512,
        'transformer.ffn.d_ff': 2048,

101        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 2.5e-4,

107        'transformer.ffn.activation': 'SquaredReLU',

112        'transformer.encoder_attn': 'MultiDConvHeadAttention',
        7    })
```

--------------------------------

### Get Latent Vector z

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/sketch_rnn/index.html

Encodes the input data to obtain the latent vector z.

```python
z, _, _ = self.encoder(data)
```

--------------------------------

### Initialize configurations

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/diffusion/ddpm/experiment.ipynb

Creates an instance of the `Configs` class to manage experiment settings.

```python
configs = Configs()
```

--------------------------------

### Configure and create a small model for CIFAR-10

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/distillation/small.html

Sets up the experiment configuration, including defining the model to be used. It specifies the optimizer and learning rate, then initializes and starts the experiment.

```python
@option(Configs.model)
def _small_model(c: Configs):
    return SmallModel().to(c.device)

def main():
    experiment.create(name='cifar10', comment='small model')

    conf = Configs()
    experiment.configs(conf, {
        'optimizer.optimizer': 'Adam',
        'optimizer.learning_rate': 2.5e-4,
    })

    experiment.add_pytorch_models({'model': conf.model})

    logger.inspect(params=(sum(p.numel() for p in conf.model.parameters() if p.requires_grad)))

    with experiment.start():
        conf.run()

if __name__ == '__main__':
    main()
```

--------------------------------

### Get Means for Distribution

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/sketch_rnn/index.html

Stacks the mu_x and mu_y values to form the mean vector for the distribution.

```python
mean = torch.stack([self.mu_x, self.mu_y], -1)
```

--------------------------------

### Get Data Iterators - Python

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/neox/utils/trainer.html

Configures and returns iterators for training, validation, sampling, and checkpointing.

```python
def get_iterators(self):
    iterators = [('train', self.train_loader)]
    if self.valid_loader is not None:
        iterators.append(('valid', self.valid_loader))

    if self.samples_per_epoch > 0:
        iterators.append((self.sample, [i for i in range(self.samples_per_epoch)]))

    if self.checkpoints_per_epoch > 0:
        iterators.append((self.save_checkpoint, [i for i in range(self.checkpoints_per_epoch)]))

    return iterators
```

--------------------------------

### Run Kuhn Poker Experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/cfr/kuhn/index.html

Sets up and runs the Kuhn Poker experiment. It creates an experiment, configures it, and starts the iteration process. Uses SQLite for tracking to optimize performance.

```python
def main():
    experiment.create(name='kuhn_poker', writers={'sqlite'})
    conf = Configs()
    experiment.configs(conf)
    with experiment.start():
        conf.cfr.iterate()

if __name__ == '__main__':
    main()
```

--------------------------------

### Get Cache Instance

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/neox/utils/cache.html

Provides a singleton instance of the Cache class. Initializes it if it doesn't exist.

```python
def get_cache() -> Cache:
    global _INSTANCE
    if _INSTANCE is None:
        _INSTANCE = Cache()
    return _INSTANCE
```

--------------------------------

### Main Evaluation Script

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/knn/eval_knn.html

Sets up the model for evaluation, loads the experiment configuration and index, defines k-NN weights, and computes validation loss. The results are then inspected and printed.

```python
def main():
    from labml_nn.transformers.knn.build_index import load_experiment
    conf = load_experiment('4984b85c20bf11eb877a69c1a03717cd')
    conf.model.eval()
    index, keys_store, vals_store = load_index(conf)
    knn_weights = [i / 20 for i in range(10)]
    losses, n_samples = validation_loss(knn_weights, None, conf, index, keys_store, vals_store)
    inspect({c: np.sum(losses[i]) / np.sum(n_samples) for i, c in enumerate(knn_weights)})
```

```python
if __name__ == '__main__':
    main()
```

--------------------------------

### Import necessary libraries for RoPE

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/transformers/rope/index.html

Imports PyTorch, logger, and the MultiHeadAttention module. Ensure these are installed and available.

```python
import torch
from torch import nn

from labml.logger import inspect
from labml_nn.transformers.mha import MultiHeadAttention
```

--------------------------------

### Initialize configurations

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb

Create an instance of MNISTConfigs to manage experiment settings.

```python
conf = MNISTConfigs()
```

--------------------------------

### Run the experiment

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/lora/experiment.ipynb

Start and run the experiment within the labml experiment context. This executes the training loop.

```python
with experiment.start():
    trainer.run()
```

--------------------------------

### Initialize PPO Model

Source: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/docs/rl/ppo/experiment.html

Initializes the PPO model and moves it to the specified device (e.g., GPU).

```python
self.model = Model().to(device)
```