《动手学深度学习》

https://zh.d2l.ai/
Admin
An interactive deep learning textbook for Chinese readers, featuring executable Jupyter notebooks...
Tokens:53,303
Snippets:246
Trust Score:9.6
Update:1 year ago
Show doc for...
Tokens:
Raw
### PaddlePaddle Environment Setup

Source: https://zh.d2l.ai/_sources/chapter_optimization/lr-scheduler.rst

This snippet shows the initial imports required for setting up a deep learning environment using PaddlePaddle, including the d2l library and standard math utilities. It also includes a warning filter to ignore warnings.

```python
%matplotlib inline
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import math
import paddle
```

--------------------------------

### Import Libraries and Setup Environment for FCN Implementation

Source: https://zh.d2l.ai/_sources/chapter_computer-vision/fcn.rst

This snippet imports essential deep learning libraries for MXNet, PyTorch, and PaddlePaddle, along with the d2l utility library. It includes framework-specific setup like npx.set_np() for MXNet to enable NumPy-like operations and warnings.filterwarnings("ignore") for PaddlePaddle to suppress non-critical warnings, preparing the environment for FCN model development.

```mxnet
%matplotlib inline
from mxnet import gluon, image, init, np, npx
from mxnet.gluon import nn
from d2l import mxnet as d2l

npx.set_np()
```

```pytorch
%matplotlib inline
import torch
import torchvision
from torch import nn
from torch.nn import functional as F
from d2l import torch as d2l
```

```paddle
%matplotlib inline
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import paddle
import paddle.vision as paddlevision
from paddle import nn
from paddle.nn import functional as F
```

--------------------------------

### Test Multi-Head Attention Across Deep Learning Frameworks

Source: https://zh.d2l.ai/chapter_attention-mechanisms/multihead-attention

Demonstrates how to initialize and test the `MultiHeadAttention` class with sample input tensors across different deep learning frameworks. It shows the setup of `num_hiddens`, `num_heads`, and input tensors `X`, `Y` with `valid_lens`, then calls the attention mechanism and prints the output shape. This snippet groups examples for NumPy/MXNet, PyTorch, TensorFlow, and PaddlePaddle.

```NumPy/MXNet
num_hiddens, num_heads = 100, 5
attention = MultiHeadAttention(num_hiddens, num_heads, 0.5)
attention.initialize()

batch_size, num_queries = 2, 4
num_kvpairs, valid_lens =  6, np.array([3, 2])
X = np.ones((batch_size, num_queries, num_hiddens))
Y = np.ones((batch_size, num_kvpairs, num_hiddens))
attention(X, Y, Y, valid_lens).shape
```

```PyTorch
num_hiddens, num_heads = 100, 5
attention = MultiHeadAttention(num_hiddens, num_hiddens, num_hiddens,
                               num_hiddens, num_heads, 0.5)
attention.eval()

batch_size, num_queries = 2, 4
num_kvpairs, valid_lens =  6, torch.tensor([3, 2])
X = torch.ones((batch_size, num_queries, num_hiddens))
Y = torch.ones((batch_size, num_kvpairs, num_hiddens))
attention(X, Y, Y, valid_lens).shape
```

```TensorFlow
num_hiddens, num_heads = 100, 5
attention = MultiHeadAttention(num_hiddens, num_hiddens, num_hiddens,
                               num_hiddens, num_heads, 0.5)

batch_size, num_queries = 2, 4
num_kvpairs, valid_lens = 6, tf.constant([3, 2])
X = tf.ones((batch_size, num_queries, num_hiddens))
Y = tf.ones((batch_size, num_kvpairs, num_hiddens))
attention(X, Y, Y, valid_lens, training=False).shape
```

```PaddlePaddle
num_hiddens, num_heads = 100, 5
attention = MultiHeadAttention(num_hiddens, num_hiddens, num_hiddens,
                               num_hiddens, num_heads, 0.5)
attention.eval()

batch_size, num_queries = 2, 4
num_kvpairs, valid_lens =  6, paddle.to_tensor([3, 2])
X = paddle.ones((batch_size, num_queries, num_hiddens))
Y = paddle.ones((batch_size, num_kvpairs, num_hiddens))
attention(X, Y, Y, valid_lens).shape
```

--------------------------------

### Console Output for MXNet SNLI Data Loading

Source: https://zh.d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-bert

This console output shows the progress and results of downloading and reading the SNLI dataset using the MXNet setup. It indicates the download of the SNLI zip file and the number of examples successfully read for both training and testing sets.

```Console
Downloading ../data/snli_1.0.zip from https://nlp.stanford.edu/projects/snli/snli_1.0.zip...
read 549367 examples
read 9824 examples
```

--------------------------------

### Define Data Hub and URL for Dataset Downloads

Source: https://zh.d2l.ai/_sources/chapter_multilayer-perceptrons/kaggle-house-price.rst

Initializes a dictionary `DATA_HUB` to map dataset names to their URLs and SHA-1 hashes, and defines `DATA_URL` as the base URL for all hosted datasets. This setup facilitates centralized management and access to various datasets used in the book.

```python
import hashlib
import os
import tarfile
import zipfile
import requests

#@save
DATA_HUB = dict()
DATA_URL = 'http://d2l-data.s3-accelerate.amazonaws.com/'
```

--------------------------------

### Import Core Libraries for Machine Translation Setup

Source: https://zh.d2l.ai/chapter_recurrent-modern/machine-translation-and-dataset

These code snippets demonstrate the initial imports required for setting up a machine translation project across different deep learning frameworks. They import common libraries like 'os' and framework-specific modules from 'd2l' for data handling and model building.

```MXNet
import os
from mxnet import np, npx
from d2l import mxnet as d2l

npx.set_np()
```

```PyTorch
import os
import torch
from d2l import torch as d2l
```

```TensorFlow
import os
import tensorflow as tf
from d2l import tensorflow as d2l
```

```PaddlePaddle
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import os
import paddle
```

--------------------------------

### PyTorch `torch.ones` Usage Examples

Source: https://zh.d2l.ai/_sources/chapter_preliminaries/lookup-api.rst

Examples demonstrating how to create tensors filled with ones using `torch.ones` in PyTorch, showing creation of 2D and 1D tensors with their respective outputs.

```python
torch.ones(2, 3)
# tensor([[ 1.,  1.,  1.],
#         [ 1.,  1.,  1.]])

torch.ones(5)
# tensor([ 1.,  1.,  1.,  1.,  1.])
```

--------------------------------

### Model Serialization Across Frameworks

Source: https://zh.d2l.ai/chapter_computational-performance/hybridize

Demonstrates how to serialize (save) trained deep learning models to disk using different frameworks: MXNet, TensorFlow, and PaddlePaddle. This allows for deployment and use in other programming languages. Each example shows the specific API call and the resulting file structure.

```Python (MXNet)
net.export('my_mlp')
!ls -lh my_mlp*

-rw-r--r-- 1 ci ci 643K Aug 18 07:00 my_mlp-0000.params
-rw-r--r-- 1 ci ci 3.2K Aug 18 07:00 my_mlp-symbol.json
```

```Python (MXNet)
net.save('my_mlp')
!ls -lh my_mlp*

-rw-r--r-- 1 ci ci 651K Aug 18 06:58 my_mlp
```

```Python (TensorFlow)
net = get_net()
tf.saved_model.save(net, 'my_mlp')
!ls -lh my_mlp*

INFO:tensorflow:Assets written to: my_mlp/assets
total 64K
drwxr-xr-x 2 ci ci   6 Aug 18 07:38 assets
-rw-r--r-- 1 ci ci 64K Aug 18 07:38 saved_model.pb
drwxr-xr-x 2 ci ci  66 Aug 18 07:38 variables
```

```Python (PaddlePaddle)
paddle.jit.save(net, './my_mlp')
!ls -lh my_mlp*

-rw-r--r-- 1 ci ci 643K Aug 18 09:06 my_mlp.pdiparams
-rw-r--r-- 1 ci ci  340 Aug 18 09:06 my_mlp.pdiparams.info
-rw-r--r-- 1 ci ci  60K Aug 18 09:06 my_mlp.pdmodel
```

--------------------------------

### Console Output for PyTorch SNLI Data Loading

Source: https://zh.d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-bert

This console output shows the number of examples successfully read for both training and testing sets after processing the SNLI dataset with the PyTorch setup.

```Console
read 549367 examples
read 9824 examples
```

--------------------------------

### Console Output for PaddlePaddle SNLI Data Loading

Source: https://zh.d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-bert

This console output shows the progress and results of downloading and reading the SNLI dataset using the PaddlePaddle setup. It indicates the download of the SNLI zip file (with Chinese message) and the number of examples successfully read for both training and testing sets.

```Console
正在从https://nlp.stanford.edu/projects/snli/snli_1.0.zip下载../data/snli_1.0.zip...
read 549367 examples
read 9824 examples
```

--------------------------------

### PaddlePaddle `paddle.ones` API Documentation

Source: https://zh.d2l.ai/_sources/chapter_preliminaries/lookup-api.rst

Detailed API documentation for the `paddle.ones` function in PaddlePaddle, covering its purpose, parameters (shape, dtype, name), return value, and multiple Python examples demonstrating its usage with various shapes and data types.

```APIDOC
ones(shape, dtype=None, name=None)
  The OP creates a tensor of specified :attr:`shape` and :attr:`dtype`, and fills it with 1.

  Args:
    shape(tuple|list|Tensor): Shape of the Tensor to be created, the data type of shape is int32 or int64.
    dtype(np.dtype|str, optional): Data type of output Tensor, it supports
      bool, float16, float32, float64, int32 and int64. Default: if None, the data type is 'float32'.
    name(str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`

  Returns:
    Tensor: A tensor of data type :attr:`dtype` with shape :attr:`shape` and all elements set to 1.

  Examples:
    import paddle

    # default dtype for ones OP
    data1 = paddle.ones(shape=[3, 2])
    # [[1. 1.]
    #  [1. 1.]
    #  [1. 1.]]

    data2 = paddle.ones(shape=[2, 2], dtype='int32')
    # [[1 1]
    #  [1 1]]

    # shape is a Tensor
    shape = paddle.full(shape=[2], dtype='int32', fill_value=2)
    data3 = paddle.ones(shape=shape, dtype='int32')
    # [[1 1]
    #  [1 1]]
```

--------------------------------

### Initialize Example Input Tensor X for Cross-Correlation

Source: https://zh.d2l.ai/_sources/chapter_convolutional-neural-networks/conv-layer.rst

This snippet initializes a 3x3 2D array `X` with float values. This tensor serves as an example input for demonstrating the 2D cross-correlation operation, as described in the accompanying text and figures.

```mxnet
X = np.array([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])
```

--------------------------------

### Execute MXNet-like Skip-Gram Model Training

Source: https://zh.d2l.ai/_sources/chapter_natural-language-processing-pretraining/word2vec-pretraining.rst

This snippet shows how to initiate the training of the Skip-Gram model using the previously defined `train` function (presumably the MXNet-like version). It sets the learning rate and the number of training epochs, then calls the `train` function with the network, data iterator, and specified hyperparameters to start the training process.

```Python
    lr, num_epochs = 0.002, 5
    train(net, data_iter, lr, num_epochs)
```

--------------------------------

### Define and Train LeNet-like Model (TensorFlow)

Source: https://zh.d2l.ai/_sources/chapter_optimization/lr-scheduler.rst

This snippet defines a convolutional neural network architecture similar to LeNet using TensorFlow/Keras. It also provides a training function that compiles and fits the model using a specified optimizer and loss function, integrating with d2l utilities for data loading and callbacks.

```python
%matplotlib inline
import math
import tensorflow as tf
from tensorflow.keras.callbacks import LearningRateScheduler
from d2l import tensorflow as d2l

def net():
    return tf.keras.models.Sequential([
        tf.keras.layers.Conv2D(filters=6, kernel_size=5, activation='relu',
                               padding='same'),
        tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
        tf.keras.layers.Conv2D(filters=16, kernel_size=5,
                               activation='relu'),
        tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(120, activation='relu'),
        tf.keras.layers.Dense(84, activation='sigmoid'),
        tf.keras.layers.Dense(10)])

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)

# 代码几乎与d2l.train_ch6定义在卷积神经网络一章LeNet一节中的相同
def train(net_fn, train_iter, test_iter, num_epochs, lr,
              device=d2l.try_gpu(), custom_callback = False):
    device_name = device._device_name
    strategy = tf.distribute.OneDeviceStrategy(device_name)
    with strategy.scope():
        optimizer = tf.keras.optimizers.SGD(learning_rate=lr)
        loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
        net = net_fn()
        net.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])
    callback = d2l.TrainCallback(net, train_iter, test_iter, num_epochs,
                             device_name)
    if custom_callback is False:
        net.fit(train_iter, epochs=num_epochs, verbose=0,
                callbacks=[callback])
    else:
         net.fit(train_iter, epochs=num_epochs, verbose=0,
                 callbacks=[callback, custom_callback])
    return net
```

--------------------------------

### TensorFlow `tf.ones` API Documentation

Source: https://zh.d2l.ai/_sources/chapter_preliminaries/lookup-api.rst

Detailed API documentation for the `tf.ones` function in TensorFlow, including its purpose, parameters (shape, dtype, name), return value, and an example of creating a 3x4 integer tensor filled with ones.

```APIDOC
ones(shape, dtype=tf.float32, name=None)
  Creates a tensor with all elements set to one (1).

  See also `tf.ones_like`, `tf.zeros`, `tf.fill`, `tf.eye`.

  This operation returns a tensor of type `dtype` with shape `shape` and
  all elements set to one.

  Example:
    >>> tf.ones([3, 4], tf.int32)
    <tf.Tensor: shape=(3, 4), dtype=int32, numpy=
    array([[1, 1, 1, 1],
           [1, 1, 1, 1],
           [1, 1, 1, 1]], dtype=int32)>

  Args:
    shape: A `list` of integers, a `tuple` of integers, or
      a 1-D `Tensor` of type `int32`.
    dtype: Optional DType of an element in the resulting `Tensor`. Default is
      `tf.float32`.
    name: Optional string. A name for the operation.

  Returns:
    A `Tensor` with all elements set to one (1).
```

--------------------------------

### Initialize BERTClassifier Instance

Source: https://zh.d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-bert

This snippet demonstrates how to instantiate the `BERTClassifier` with a pre-trained BERT model. In the MXNet example, the output layer is explicitly initialized. For PyTorch and PaddlePaddle, the initialization is implicit or handled by the framework's default behavior.

```mxnet
net = BERTClassifier(bert)
net.output.initialize(ctx=devices)
```

```pytorch
net = BERTClassifier(bert)
```

```paddle
net = BERTClassifier(bert)
```

--------------------------------

### Predict Sequence-to-Sequence Translation and Calculate BLEU Score (PaddlePaddle)

Source: https://zh.d2l.ai/_sources/chapter_attention-mechanisms/bahdanau-attention.rst

This snippet demonstrates predicting translations and calculating BLEU scores using a sequence-to-sequence model within a PaddlePaddle context. It iterates through example sentences, performs prediction, and prints the translated output along with its BLEU score.

```Python
engs = ['go .', "i lost .", 'he\'s calm .', 'i\'m home .']
fras = ['va !', 'j\'ai perdu .', 'il est calme .', 'je suis chez moi .']
for eng, fra in zip(engs, fras):
    translation, dec_attention_weight_seq = d2l.predict_seq2seq(
        net, eng, src_vocab, tgt_vocab, num_steps, device, True)
    print(f'{eng} => {translation}, ',
          f'bleu {d2l.bleu(translation, fra, k=2):.3f}')
```

--------------------------------

### Train Deep Learning Model (PyTorch)

Source: https://zh.d2l.ai/_sources/chapter_optimization/lr-scheduler.rst

This function implements a general training loop for deep learning models using PyTorch. It handles epoch iteration, batch processing, loss calculation, backpropagation, optimizer steps, and evaluation on both training and test datasets. It also supports learning rate scheduling.

```python
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)

# 代码几乎与d2l.train_ch6定义在卷积神经网络一章LeNet一节中的相同
def train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
              scheduler=None):
    net.to(device)
    animator = d2l.Animator(xlabel='epoch', xlim=[0, num_epochs],
                            legend=['train loss', 'train acc', 'test acc'])

    for epoch in range(num_epochs):
        metric = d2l.Accumulator(3)  # train_loss,train_acc,num_examples
        for i, (X, y) in enumerate(train_iter):
            net.train()
            trainer.zero_grad()
            X, y = X.to(device), y.to(device)
            y_hat = net(X)
            l = loss(y_hat, y)
            l.backward()
            trainer.step()
            with torch.no_grad():
                metric.add(l * X.shape[0], d2l.accuracy(y_hat, y), X.shape[0])
            train_loss = metric[0] / metric[2]
            train_acc = metric[1] / metric[2]
            if (i + 1) % 50 == 0:
                animator.add(epoch + i / len(train_iter),
                             (train_loss, train_acc, None))

        test_acc = d2l.evaluate_accuracy_gpu(net, test_iter)
        animator.add(epoch+1, (None, None, test_acc))

        if scheduler:
            if scheduler.__module__ == lr_scheduler.__name__:
                # UsingPyTorchIn-Builtscheduler
                scheduler.step()
            else:
                # Usingcustomdefinedscheduler
                for param_group in trainer.param_groups:
                    param_group['lr'] = scheduler(epoch)

    print(f'train loss {train_loss:.3f}, train acc {train_acc:.3f}, '
          f'test acc {test_acc:.3f}')
```

--------------------------------

### Initialize Environment for Deep Learning Frameworks

Source: https://zh.d2l.ai/chapter_optimization/convexity

This code block sets up the necessary environment for running examples related to convexity. It imports plotting utilities (mpl_toolkits.mplot3d), numerical computation libraries (numpy or framework-specific equivalents), and the d2l library tailored for each deep learning framework (MXNet, PyTorch, TensorFlow, PaddlePaddle). The %matplotlib inline magic command is included for Jupyter notebook compatibility, and specific framework configurations like npx.set_np() for MXNet or warnings.filterwarnings("ignore") for PaddlePaddle are applied.

```Python
%matplotlib inline
from mpl_toolkits import mplot3d
from mxnet import np, npx
from d2l import mxnet as d2l

npx.set_np()
```

```Python
%matplotlib inline
import numpy as np
import torch
from mpl_toolkits import mplot3d
from d2l import torch as d2l
```

```Python
%matplotlib inline
import numpy as np
import tensorflow as tf
from mpl_toolkits import mplot3d
from d2l import tensorflow as d2l
```

```Python
%matplotlib inline
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import numpy as np
import paddle
from mpl_toolkits import mplot3d
```

--------------------------------

### Initialize and Train Seq2Seq Model Across Frameworks

Source: https://zh.d2l.ai/_sources/chapter_recurrent-modern/seq2seq.rst

This code demonstrates the setup of a sequence-to-sequence encoder-decoder model and initiates its training. It defines key hyperparameters such as embedding size, hidden units, layers, dropout, batch size, and learning rate. The snippet is provided for MXNet, PyTorch, and TensorFlow, showcasing how to prepare and train the model using a common `train_seq2seq` utility function across different deep learning frameworks.

```python
    embed_size, num_hiddens, num_layers, dropout = 32, 32, 2, 0.1
    batch_size, num_steps = 64, 10
    lr, num_epochs, device = 0.005, 300, d2l.try_gpu()

    train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)
    encoder = Seq2SeqEncoder(len(src_vocab), embed_size, num_hiddens, num_layers,
                            dropout)
    decoder = Seq2SeqDecoder(len(tgt_vocab), embed_size, num_hiddens, num_layers,
                            dropout)
    net = d2l.EncoderDecoder(encoder, decoder)
    train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)
```

```python
    embed_size, num_hiddens, num_layers, dropout = 32, 32, 2, 0.1
    batch_size, num_steps = 64, 10
    lr, num_epochs, device = 0.005, 300, d2l.try_gpu()

    train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)
    encoder = Seq2SeqEncoder(len(src_vocab), embed_size, num_hiddens, num_layers,
                            dropout)
    decoder = Seq2SeqDecoder(len(tgt_vocab), embed_size, num_hiddens, num_layers,
                            dropout)
    net = d2l.EncoderDecoder(encoder, decoder)
    train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)
```

```python
    embed_size, num_hiddens, num_layers, dropout = 32, 32, 2, 0.1
    batch_size, num_steps = 64, 10
    lr, num_epochs, device = 0.005, 300, d2l.try_gpu()

    train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)
    encoder = Seq2SeqEncoder(len(src_vocab), embed_size, num_hiddens, num_layers,
                            dropout)
    decoder = Seq2SeqDecoder(len(tgt_vocab), embed_size, num_hiddens, num_layers,
                            dropout)
    net = d2l.EncoderDecoder(encoder, decoder)
    train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)
```

--------------------------------

### Multi-Head Attention Model Structure (PyTorch Example)

Source: https://zh.d2l.ai/chapter_attention-mechanisms/multihead-attention

Displays the architectural structure of a `MultiHeadAttention` model, specifically showing its components like `DotProductAttention` with `Dropout` and various `Linear` layers for query, key, value, and output transformations. This is a typical representation of a PyTorch model's `__repr__` output.

```PyTorch
MultiHeadAttention(
  (attention): DotProductAttention(
    (dropout): Dropout(p=0.5, inplace=False)
  )
  (W_q): Linear(in_features=100, out_features=100, bias=False)
  (W_k): Linear(in_features=100, out_features=100, bias=False)
  (W_v): Linear(in_features=100, out_features=100, bias=False)
  (W_o): Linear(in_features=100, out_features=100, bias=False)
)
```

--------------------------------

### Predict Sequence-to-Sequence Translation and Calculate BLEU Score (PyTorch)

Source: https://zh.d2l.ai/_sources/chapter_attention-mechanisms/bahdanau-attention.rst

This snippet demonstrates predicting translations and calculating BLEU scores using a sequence-to-sequence model within a PyTorch context. It iterates through example sentences, performs prediction, and prints the translated output along with its BLEU score.

```Python
engs = ['go .', "i lost .", 'he\'s calm .', 'i\'m home .']
fras = ['va !', 'j\'ai perdu .', 'il est calme .', 'je suis chez moi .']
for eng, fra in zip(engs, fras):
    translation, dec_attention_weight_seq = d2l.predict_seq2seq(
        net, eng, src_vocab, tgt_vocab, num_steps, device, True)
    print(f'{eng} => {translation}, ',
          f'bleu {d2l.bleu(translation, fra, k=2):.3f}')
```

--------------------------------

### Train Model with Custom RMSProp Optimizer Across Frameworks

Source: https://zh.d2l.ai/_sources/chapter_optimization/rmsprop.rst

This snippet demonstrates how to train a model using the custom RMSProp implementation. It utilizes `d2l.get_data_ch11` to load data and `d2l.train_ch11` to run the training loop, passing the custom `rmsprop` function, initialized states, and hyperparameters (learning rate and gamma). This example is identical across MXNet, PyTorch, TensorFlow, and PaddlePaddle.

```Python
data_iter, feature_dim = d2l.get_data_ch11(batch_size=10)
d2l.train_ch11(rmsprop, init_rmsprop_states(feature_dim),
               {'lr': 0.01, 'gamma': 0.9}, data_iter, feature_dim);
```

```Python
data_iter, feature_dim = d2l.get_data_ch11(batch_size=10)
d2l.train_ch11(rmsprop, init_rmsprop_states(feature_dim),
               {'lr': 0.01, 'gamma': 0.9}, data_iter, feature_dim);
```

```Python
data_iter, feature_dim = d2l.get_data_ch11(batch_size=10)
d2l.train_ch11(rmsprop, init_rmsprop_states(feature_dim),
               {'lr': 0.01, 'gamma': 0.9}, data_iter, feature_dim);
```

```Python
data_iter, feature_dim = d2l.get_data_ch11(batch_size=10)
d2l.train_ch11(rmsprop, init_rmsprop_states(feature_dim),
               {'lr': 0.01, 'gamma': 0.9}, data_iter, feature_dim);
```

--------------------------------

### Predict Sequence-to-Sequence Translation and Calculate BLEU Score (NumPy/MXNet Context)

Source: https://zh.d2l.ai/_sources/chapter_attention-mechanisms/bahdanau-attention.rst

This snippet demonstrates predicting translations and calculating BLEU scores using a sequence-to-sequence model in a generic Python/NumPy context, likely associated with MXNet due to the subsequent attention weight calculation. It iterates through example sentences, performs prediction, and prints the translated output along with its BLEU score.

```Python
engs = ['go .', "i lost .", 'he\'s calm .', 'i\'m home .']
fras = ['va !', 'j\'ai perdu .', 'il est calme .', 'je suis chez moi .']
for eng, fra in zip(engs, fras):
    translation, dec_attention_weight_seq = d2l.predict_seq2seq(
        net, eng, src_vocab, tgt_vocab, num_steps, device, True)
    print(f'{eng} => {translation}, ',
          f'bleu {d2l.bleu(translation, fra, k=2):.3f}')
```

--------------------------------

### Initialize and Apply MultiHeadAttention Layer

Source: https://zh.d2l.ai/_sources/chapter_attention-mechanisms/self-attention-and-positional-encoding.rst

This snippet demonstrates the initialization and application of a MultiHeadAttention layer, a core component in transformer models. It showcases how to set up the attention mechanism with specified hidden dimensions and number of heads, and then apply it to input tensors, handling valid sequence lengths. The examples cover both TensorFlow and PaddlePaddle implementations.

```tensorflow
num_hiddens, num_heads = 100, 5
attention = d2l.MultiHeadAttention(num_hiddens, num_hiddens, num_hiddens,
                                   num_hiddens, num_heads, 0.5)

batch_size, num_queries, valid_lens = 2, 4, tf.constant([3, 2])
X = tf.ones((batch_size, num_queries, num_hiddens))
attention(X, X, X, valid_lens, training=False).shape
```

```paddle
num_hiddens, num_heads = 100, 5
attention = d2l.MultiHeadAttention(num_hiddens, num_hiddens, num_hiddens,
                                   num_hiddens, num_heads, 0.5)
attention.eval()

batch_size, num_queries, valid_lens = 2, 4, paddle.to_tensor([3, 2])
X = paddle.ones((batch_size, num_queries, num_hiddens))
attention(X, X, X, valid_lens).shape
```

--------------------------------

### PaddlePaddle `train_ch6` Function for Model Training

Source: https://zh.d2l.ai/_sources/chapter_convolutional-neural-networks/lenet.rst

Defines the `train_ch6` function tailored for PaddlePaddle, which handles the entire training loop for a deep learning model. It includes weight initialization, device placement, optimizer and loss setup, epoch iteration, gradient clearing, forward/backward passes, and metric accumulation, concluding with performance reporting.

```python
    #@save
    def train_ch6(net, train_iter, test_iter, num_epochs, lr, device):
        """用GPU训练模型(在第六章定义)"""
        def init_weights(m):
            if type(m) == nn.Linear or type(m) == nn.Conv2D:
                nn.initializer.XavierUniform(m.weight)
        net.apply(init_weights)
        print('training on', device)
        net.to(device)
        optimizer = paddle.optimizer.SGD(learning_rate=lr, parameters=net.parameters())
        loss = nn.CrossEntropyLoss()
        animator = d2l.Animator(xlabel='epoch', xlim=[1, num_epochs],
                                legend=['train loss', 'train acc', 'test acc'])
        timer, num_batches = d2l.Timer(), len(train_iter)
        for epoch in range(num_epochs):
            # 训练损失之和，训练准确率之和，样本数
            metric = d2l.Accumulator(3)
            net.train()
            for i, (X, y) in enumerate(train_iter):
                timer.start()
                optimizer.clear_grad()
                X, y = paddle.to_tensor(X, place=device), paddle.to_tensor(y, place=device)
                y_hat = net(X)
                l = loss(y_hat, y)
                l.backward()
                optimizer.step()
                with paddle.no_grad():
                    metric.add(l * X.shape[0], d2l.accuracy(y_hat, y), X.shape[0])
                timer.stop()
                train_l = metric[0] / metric[2]
                train_acc = metric[1] / metric[2]
                if (i + 1) % (num_batches // 5) == 0 or i == num_batches - 1:
                    animator.add(epoch + (i + 1) / num_batches,
                                 (train_l, train_acc, None))
            test_acc = evaluate_accuracy_gpu(net, test_iter)
            animator.add(epoch + 1, (None, None, test_acc))
        print(f'loss {train_l:.3f}, train acc {train_acc:.3f}, '
              f'test acc {test_acc:.3f}')
        print(f'{metric[2] * num_epochs / timer.sum():.1f} examples/sec '
              f'on {str(device)}')
```

--------------------------------

### Define Multi-Layer Perceptron (MLP) in Deep Learning Frameworks

Source: https://zh.d2l.ai/_sources/chapter_computational-performance/hybridize.rst

This snippet demonstrates how to define a simple multi-layer perceptron (MLP) across different deep learning frameworks (MXNet, PyTorch, TensorFlow). Each framework uses its respective API to construct a sequential network with dense layers and ReLU activations, preparing it for subsequent optimization. The code examples are presented in the order: MXNet, PyTorch, TensorFlow.

```python
from mxnet import np, npx
from mxnet.gluon import nn
from d2l import mxnet as d2l

npx.set_np()

# 生产网络的工厂模式
def get_net():
    net = nn.HybridSequential()
    net.add(nn.Dense(256, activation='relu'),
            nn.Dense(128, activation='relu'),
            nn.Dense(2))
    net.initialize()
    return net

x = np.random.normal(size=(1, 512))
net = get_net()
net(x)
```

```python
import torch
from torch import nn
from d2l import torch as d2l

# 生产网络的工厂模式
def get_net():
    net = nn.Sequential(nn.Linear(512, 256),
                nn.ReLU(),
                nn.Linear(256, 128),
                nn.ReLU(),
                nn.Linear(128, 2))
    return net

x = torch.randn(size=(1, 512))
net = get_net()
net(x)
```

```python
import tensorflow as tf
from tensorflow.keras.layers import Dense
from d2l import tensorflow as d2l

# 生产网络的工厂模式
def get_net():
    net = tf.keras.Sequential()
    net.add(Dense(256, input_shape = (512,), activation = "relu"))
    net.add(Dense(128, activation = "relu"))
    net.add(Dense(2, activation = "linear"))
    return net

x = tf.random.normal([1,512])
net = get_net()
net(x)
```

--------------------------------

### Import Core Libraries for Multi-GPU Training

Source: https://zh.d2l.ai/_sources/chapter_computational-performance/multiple-gpus-concise.rst

This snippet imports the necessary deep learning framework libraries (MXNet, PyTorch, PaddlePaddle) and the d2l utility library for setting up multi-GPU training environments. It includes specific initializations like setting numpy compatibility for MXNet and filtering warnings for PaddlePaddle.

```pytorch
import torch
from torch import nn
from d2l import torch as d2l
```

```paddle
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import paddle
from paddle import nn
```

--------------------------------

### Initialize Environment for Data Processing with Pandas and MXNet

Source: https://zh.d2l.ai/_sources/chapter_multilayer-perceptrons/kaggle-house-price.rst

Sets up the Python environment by importing necessary libraries for data manipulation and deep learning. It includes `pandas` for data handling, and `mxnet` for numerical operations and neural network components, along with `d2l` utilities. It also configures `mxnet` to use NumPy-compatible arrays.

```python
# 如果没有安装pandas，请取消下一行的注释
# !pip install pandas

%matplotlib inline
import pandas as pd
from mxnet import autograd, gluon, init, np, npx
from mxnet.gluon import nn
from d2l import mxnet as d2l

npx.set_np()
```

--------------------------------

### Load Fashion-MNIST Dataset and Set Batch Size

Source: https://zh.d2l.ai/_sources/chapter_multilayer-perceptrons/mlp-scratch.rst

This snippet demonstrates how to load the Fashion-MNIST image classification dataset and set the batch size for training and testing. It utilizes the `d2l` utility library specific to each deep learning framework to handle data loading.

```mxnet
from mxnet import gluon, np, npx
from d2l import mxnet as d2l

npx.set_np()

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)
```

```pytorch
import torch
from torch import nn
from d2l import torch as d2l

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)
```

```tensorflow
import tensorflow as tf
from d2l import tensorflow as d2l

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)
```

```paddle
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import paddle
from paddle import nn

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)
```

--------------------------------

### Initialize Environment and Set Print Options

Source: https://zh.d2l.ai/_sources/chapter_computer-vision/anchor.rst

These code snippets import necessary libraries for deep learning frameworks (MXNet, PyTorch, PaddlePaddle) and set the print options for numerical arrays to a precision of two decimal places, ensuring cleaner output for tensor operations.

```mxnet
%matplotlib inline
from mxnet import gluon, image, np, npx
from d2l import mxnet as d2l

np.set_printoptions(2)  # 精简输出精度
npx.set_np()
```

```pytorch
%matplotlib inline
import torch
from d2l import torch as d2l

torch.set_printoptions(2)  # 精简输出精度
```

```paddle
%matplotlib inline
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import numpy as np
import paddle

paddle.set_printoptions(2)  # 精简输出精度
```

--------------------------------

### Example Usage of Plot Function

Source: https://zh.d2l.ai/chapter_preliminaries/calculus

An example demonstrating how to use the `plot` function to visualize a function and its tangent line. It uses `numpy.arange` to generate x-values and calls `plot` with the function and tangent line data.

```Python
x = np.arange(0, 3, 0.1)
plot(x, [f(x), 2 * x - 3], 'x', 'f(x)', legend=['f(x)', 'Tangent line (x=1)'])
```

--------------------------------

### Utility Functions for Dataset Extraction and Bulk Download

Source: https://zh.d2l.ai/_sources/chapter_multilayer-perceptrons/kaggle-house-price.rst

Provides two utility functions: `download_extract` for downloading and decompressing zip/tar files, and `download_all` for iterating through `DATA_HUB` to download all registered datasets. These functions streamline dataset management for the entire project.

```python
def download_extract(name, folder=None):
    """下载并解压zip/tar文件"""
    fname = download(name)
    base_dir = os.path.dirname(fname)
    data_dir, ext = os.path.splitext(fname)
    if ext == '.zip':
        fp = zipfile.ZipFile(fname, 'r')
    elif ext in ('.tar', '.gz'):
        fp = tarfile.open(fname, 'r')
    else:
        assert False, '只有zip/tar文件可以被解压缩'
    fp.extractall(base_dir)
    return os.path.join(base_dir, folder) if folder else data_dir

def download_all():
    """下载DATA_HUB中的所有文件"""
    for name in DATA_HUB:
        download(name)
```

--------------------------------

### Multi-Framework Usage Example of Masked Softmax Cross-Entropy Loss

Source: https://zh.d2l.ai/_sources/chapter_recurrent-modern/seq2seq.rst

This snippet demonstrates how to instantiate and use the MaskedSoftmaxCELoss across different deep learning frameworks: MXNet, PyTorch, TensorFlow, and PaddlePaddle. It showcases a sanity check by applying the loss to example tensors with varying valid lengths, expecting specific loss outcomes.

```python
loss = MaskedSoftmaxCELoss()
loss(np.ones((3, 4, 10)), np.ones((3, 4)), np.array([4, 2, 0]))
```

```python
loss = MaskedSoftmaxCELoss()
loss(torch.ones(3, 4, 10), torch.ones((3, 4), dtype=torch.long),
         torch.tensor([4, 2, 0]))
```

```python
loss = MaskedSoftmaxCELoss(tf.constant([4, 2, 0]))
loss(tf.ones((3,4), dtype = tf.int32), tf.ones((3, 4, 10))).numpy()
```

```python
loss = MaskedSoftmaxCELoss()
loss(paddle.ones([3, 4, 10]), paddle.ones((3, 4), dtype=paddle.int64),
         paddle.to_tensor([4, 2, 0]))
```

--------------------------------

### Execute Model Training Across Deep Learning Frameworks

Source: https://zh.d2l.ai/_sources/chapter_convolutional-neural-networks/lenet.rst

This snippet shows the common pattern for initiating model training using the `train_ch6` function across different deep learning frameworks (MXNet, PyTorch, TensorFlow, PaddlePaddle). It sets learning rate and epochs, then calls `train_ch6` with the network, data iterators, and a device obtained via `d2l.try_gpu()`.

```python (MXNet)
lr, num_epochs = 0.9, 10
train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())
```

```python (PyTorch)
lr, num_epochs = 0.9, 10
train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())
```

```python (TensorFlow)
lr, num_epochs = 0.9, 10
train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())
```

```python (PaddlePaddle)
lr, num_epochs = 0.9, 10
train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())
```

--------------------------------

### Sample English-French Translation Pairs

Source: https://zh.d2l.ai/chapter_recurrent-modern/machine-translation-and-dataset

A small dataset containing simple English phrases and their corresponding French translations, suitable for machine translation or language learning examples. Each line represents an English phrase followed by its French equivalent.

```Text
go .        va !
hi .        salut !
run !       cours !
run !       courez !
who ?       qui ?
wow !       ça alors !
```

--------------------------------

### Train Model on GPU (PyTorch - Partial)

Source: https://zh.d2l.ai/chapter_convolutional-neural-networks/lenet

This partial code snippet shows the beginning of a training function for deep learning models using PyTorch, specifically including a helper function `init_weights` for parameter initialization.

```Python (PyTorch)
def train_ch6(net, train_iter, test_iter, num_epochs, lr, device):
    """用GPU训练模型(在第六章定义)"""
    def init_weights(m):
```

--------------------------------

### Python Function for Text Preprocessing

Source: https://zh.d2l.ai/chapter_recurrent-modern/machine-translation-and-dataset

This Python function normalizes text by replacing specific Unicode characters with spaces, converting text to lowercase, and inserting spaces between words and punctuation marks for natural language processing tasks. It also includes an example of how to use the function.

```Python
text = text.replace('\u202f', ' ').replace('\xa0', ' ').lower()
# 在单词和标点符号之间插入空格
out = [' ' + char if i > 0 and no_space(char, text[i - 1]) else char
           for i, char in enumerate(text)]
return ''.join(out)

text = preprocess_nmt(raw_text)
print(text[:80])
```

--------------------------------

### Demonstrate Masked Softmax with 2D Valid Lengths

Source: https://zh.d2l.ai/chapter_attention-mechanisms/attention-scoring-functions

This example illustrates the use of `masked_softmax` with a 2D `valid_lens` array. This allows specifying different valid lengths for each row within a sample, providing more granular control over masking in complex tensor structures.

```Python (MXNet)
masked_softmax(np.random.uniform(size=(2, 2, 4)),
               np.array([[1, 3], [2, 4]]))
```

--------------------------------

### Initialize MLP Parameters (Weights and Biases)

Source: https://zh.d2l.ai/_sources/chapter_multilayer-perceptrons/mlp-scratch.rst

This code initializes the weight matrices and bias vectors for a two-layer Multilayer Perceptron. It sets up the input, hidden, and output layer dimensions and uses random normal distribution for weights and zeros for biases. Parameters are prepared for gradient computation.

```mxnet
num_inputs, num_outputs, num_hiddens = 784, 10, 256

W1 = np.random.normal(scale=0.01, size=(num_inputs, num_hiddens))
b1 = np.zeros(num_hiddens)
W2 = np.random.normal(scale=0.01, size=(num_hiddens, num_outputs))
b2 = np.zeros(num_outputs)
params = [W1, b1, W2, b2]

for param in params:
    param.attach_grad()
```

```pytorch
num_inputs, num_outputs, num_hiddens = 784, 10, 256

W1 = nn.Parameter(torch.randn(
    num_inputs, num_hiddens, requires_grad=True) * 0.01)
b1 = nn.Parameter(torch.zeros(num_hiddens, requires_grad=True))
W2 = nn.Parameter(torch.randn(
    num_hiddens, num_outputs, requires_grad=True) * 0.01)
b2 = nn.Parameter(torch.zeros(num_outputs, requires_grad=True))

params = [W1, b1, W2, b2]
```

```tensorflow
num_inputs, num_outputs, num_hiddens = 784, 10, 256

W1 = tf.Variable(tf.random.normal(
    shape=(num_inputs, num_hiddens), mean=0, stddev=0.01))
b1 = tf.Variable(tf.zeros(num_hiddens))
W2 = tf.Variable(tf.random.normal(
    shape=(num_hiddens, num_outputs), mean=0, stddev=0.01))
b2 = tf.Variable(tf.zeros(num_outputs))

params = [W1, b1, W2, b2]
```

--------------------------------

### Implement Adam Optimization Algorithm from Scratch

Source: https://zh.d2l.ai/chapter_optimization/adam

This code snippet provides a custom implementation of the Adam optimization algorithm. It includes `init_adam_states` to initialize the first and second moment estimates (v and s) for weights and biases, and the `adam` function to update model parameters using these estimates, incorporating bias correction for the initial steps. The time step `t` is managed within the `hyperparams` dictionary.

```MXNet
%matplotlib inline
from mxnet import np, npx
from d2l import mxnet as d2l

npx.set_np()

def init_adam_states(feature_dim):
    v_w, v_b = np.zeros((feature_dim, 1)), np.zeros(1)
    s_w, s_b = np.zeros((feature_dim, 1)), np.zeros(1)
    return ((v_w, s_w), (v_b, s_b))

def adam(params, states, hyperparams):
    beta1, beta2, eps = 0.9, 0.999, 1e-6
    for p, (v, s) in zip(params, states):
        v[:] = beta1 * v + (1 - beta1) * p.grad
        s[:] = beta2 * s + (1 - beta2) * np.square(p.grad)
        v_bias_corr = v / (1 - beta1 ** hyperparams['t'])
        s_bias_corr = s / (1 - beta2 ** hyperparams['t'])
        p[:] -= hyperparams['lr'] * v_bias_corr / (np.sqrt(s_bias_corr) + eps)
    hyperparams['t'] += 1
```

```PyTorch
%matplotlib inline
import torch
from d2l import torch as d2l

def init_adam_states(feature_dim):
    v_w, v_b = torch.zeros((feature_dim, 1)), torch.zeros(1)
    s_w, s_b = torch.zeros((feature_dim, 1)), torch.zeros(1)
    return ((v_w, s_w), (v_b, s_b))

def adam(params, states, hyperparams):
    beta1, beta2, eps = 0.9, 0.999, 1e-6
    for p, (v, s) in zip(params, states):
        with torch.no_grad():
            v[:] = beta1 * v + (1 - beta1) * p.grad
            s[:] = beta2 * s + (1 - beta2) * torch.square(p.grad)
            v_bias_corr = v / (1 - beta1 ** hyperparams['t'])
            s_bias_corr = s / (1 - beta2 ** hyperparams['t'])
            p[:] -= hyperparams['lr'] * v_bias_corr / (torch.sqrt(s_bias_corr)
                                                       + eps)
        p.grad.data.zero_()
    hyperparams['t'] += 1
```

```TensorFlow
%matplotlib inline
import tensorflow as tf
from d2l import tensorflow as d2l

def init_adam_states(feature_dim):
    v_w = tf.Variable(tf.zeros((feature_dim, 1)))
    v_b = tf.Variable(tf.zeros(1))
    s_w = tf.Variable(tf.zeros((feature_dim, 1)))
    s_b = tf.Variable(tf.zeros(1))
    return ((v_w, s_w), (v_b, s_b))

def adam(params, grads, states, hyperparams):
    beta1, beta2, eps = 0.9, 0.999, 1e-6
    for p, (v, s), grad in zip(params, states, grads):
        v[:].assign(beta1 * v  + (1 - beta1) * grad)
        s[:].assign(beta2 * s + (1 - beta2) * tf.math.square(grad))
        v_bias_corr = v / (1 - beta1 ** hyperparams['t'])
        s_bias_corr = s / (1 - beta2 ** hyperparams['t'])
        p[:].assign(p - hyperparams['lr'] * v_bias_corr
                    / tf.math.sqrt(s_bias_corr) + eps)
```

```PaddlePaddle
%matplotlib inline
import warnings
from d2l import paddle as d2l

warnings.filterwarnings("ignore")
import paddle

def init_adam_states(feature_dim):
    v_w, v_b = paddle.zeros((feature_dim, 1)), paddle.zeros((1, ))
    s_w, s_b = paddle.zeros((feature_dim, 1)), paddle.zeros((1, ))
    return ((v_w, s_w), (v_b, s_b))

def adam(params, states, hyperparams):
    beta1, beta2, eps = 0.9, 0.999, 1e-6
    a = []
    for p, (v, s) in zip(params, states):
        with paddle.no_grad():
            v[:] = beta1 * v + (1 - beta1) * p.grad
            s[:] = beta2 * s + (1 - beta2) * paddle.square(p.grad)
            v_bias_corr = v / (1 - beta1 ** hyperparams['t'])
            s_bias_corr = s / (1 - beta2 ** hyperparams['t'])
            p[:] -= hyperparams['lr'] * v_bias_corr / (paddle.sqrt(s_bias_corr)
                                                       + eps)
        p.grad.zero_()
        a.append(p)
    hyperparams['t'] += 1
    return a
```

--------------------------------

### Execute RNN Model Training

Source: https://zh.d2l.ai/chapter_recurrent-neural-networks/rnn-scratch

This snippet demonstrates how to initiate the training process for the recurrent neural network model by setting hyperparameters like the number of epochs and learning rate, and then calling the `train_ch8` function with the prepared network, data iterators, vocabulary, and device.

```Python
num_epochs, lr = 500, 1
train_ch8(net, train_iter, vocab, lr, num_epochs, d2l.try_gpu())
```

--------------------------------

### Translate Sentences and Calculate BLEU Score with Transformer

Source: https://zh.d2l.ai/chapter_attention-mechanisms/transformer

This snippet demonstrates how to use a trained Transformer model for machine translation inference. It translates example English sentences into French and then calculates the BLEU score to evaluate the translation quality using d2l utilities.

```Python
engs = ['go .', "i lost .", 'he\'s calm .', 'i\'m home .']
fras = ['va !', 'j\'ai perdu .', 'il est calme .', 'je suis chez moi .']
for eng, fra in zip(engs, fras):
    translation, dec_attention_weight_seq = d2l.predict_seq2seq(
        net, eng, src_vocab, tgt_vocab, num_steps, device, True)
    print(f'{eng} => {translation}, ',
          f'bleu {d2l.bleu(translation, fra, k=2):.3f}')
```

--------------------------------

### Load SNLI Dataset with PaddlePaddle DataLoader

Source: https://zh.d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-bert

This snippet demonstrates how to initialize the `SNLIBERTDataset` and create `DataLoader` instances for training and testing using PaddlePaddle's `paddle.io.DataLoader`. It configures batch size and uses `return_list=True` for data handling.

```Python
# 如果出现显存不足错误，请减少“batch_size”。在原始的BERT模型中，max_len=512
batch_size, max_len, num_workers = 512, 128, d2l.get_dataloader_workers()
data_dir = d2l.download_extract('SNLI')
train_set = SNLIBERTDataset(d2l.read_snli(data_dir, True), max_len, vocab)
test_set = SNLIBERTDataset(d2l.read_snli(data_dir, False), max_len, vocab)
train_iter = paddle.io.DataLoader(train_set, batch_size=batch_size, shuffle=True, return_list=True)
test_iter = paddle.io.DataLoader(test_set, batch_size=batch_size, return_list=True)
```

--------------------------------

### Generate Predictions and Create Kaggle Submission File

Source: https://zh.d2l.ai/_sources/chapter_computer-vision/kaggle-dog.rst

This snippet outlines the process of training the model on the combined training and validation datasets, then using the trained model to generate predictions for the test set. The predictions are processed and written to a `submission.csv` file, formatted for submission to a Kaggle competition, including class labels.

```MXNet
net = get_net(devices)
net.hybridize()
train(net, train_valid_iter, None, num_epochs, lr, wd, devices, lr_period,
          lr_decay)

preds = []
for data, label in test_iter:
    output_features = net.features(data.as_in_ctx(devices[0]))
    output = npx.softmax(net.output_new(output_features))
    preds.extend(output.asnumpy())
ids = sorted(os.listdir(
    os.path.join(data_dir, 'train_valid_test', 'test', 'unknown')))
with open('submission.csv', 'w') as f:
    f.write('id,' + ','.join(train_valid_ds.synsets) + '\n')
    for i, output in zip(ids, preds):
        f.write(i.split('.')[0] + ',' + ','.join(
            [str(num) for num in output]) + '\n')
```

```PyTorch
net = get_net(devices)
train(net, train_valid_iter, None, num_epochs, lr, wd, devices, lr_period,
          lr_decay)

preds = []
for data, label in test_iter:
    output = torch.nn.functional.softmax(net(data.to(devices[0])), dim=1)
    preds.extend(output.cpu().detach().numpy())
ids = sorted(os.listdir(
    os.path.join(data_dir, 'train_valid_test', 'test', 'unknown')))
with open('submission.csv', 'w') as f:
    f.write('id,' + ','.join(train_valid_ds.classes) + '\n')
    for i, output in zip(ids, preds):
        f.write(i.split('.')[0] + ',' + ','.join(
            [str(num) for num in output]) + '\n')
```

--------------------------------

### RNN Model Training Function (`train_ch8`)

Source: https://zh.d2l.ai/chapter_recurrent-neural-networks/rnn-scratch

This function trains a recurrent neural network model for a specified number of epochs. It initializes the network, defines the optimizer/updater, and iteratively calls `train_epoch_ch8` to perform training steps. It also includes periodic perplexity logging and prediction examples.

```MXNet
def train_ch8(net, train_iter, vocab, lr, num_epochs, device,  #@save
              use_random_iter=False):
    """训练模型（定义见第8章）"""
    loss = gluon.loss.SoftmaxCrossEntropyLoss()
    animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',
                            legend=['train'], xlim=[10, num_epochs])
    # 初始化
    if isinstance(net, gluon.Block):
        net.initialize(ctx=device, force_reinit=True,
                         init=init.Normal(0.01))
        trainer = gluon.Trainer(net.collect_params(),
                                'sgd', {'learning_rate': lr})
        updater = lambda batch_size: trainer.step(batch_size)
    else:
        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)
    predict = lambda prefix: predict_ch8(prefix, 50, net, vocab, device)
    # 训练和预测
    for epoch in range(num_epochs):
        ppl, speed = train_epoch_ch8(
            net, train_iter, loss, updater, device, use_random_iter)
        if (epoch + 1) % 10 == 0:
            animator.add(epoch + 1, [ppl])
    print(f'困惑度 {ppl:.1f}, {speed:.1f} 词元/秒 {str(device)}')
    print(predict('time traveller'))
    print(predict('traveller'))
```

```PyTorch
#@save
def train_ch8(net, train_iter, vocab, lr, num_epochs, device,
              use_random_iter=False):
    """训练模型（定义见第8章）"""
    loss = nn.CrossEntropyLoss()
    animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',
                            legend=['train'], xlim=[10, num_epochs])
    # 初始化
    if isinstance(net, nn.Module):
        updater = torch.optim.SGD(net.parameters(), lr)
    else:
        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)
    predict = lambda prefix: predict_ch8(prefix, 50, net, vocab, device)
    # 训练和预测
    for epoch in range(num_epochs):
        ppl, speed = train_epoch_ch8(
            net, train_iter, loss, updater, device, use_random_iter)
        if (epoch + 1) % 10 == 0:
            print(predict('time traveller'))
            animator.add(epoch + 1, [ppl])
    print(f'困惑度 {ppl:.1f}, {speed:.1f} 词元/秒 {str(device)}')
    print(predict('time traveller'))
    print(predict('traveller'))
```

```TensorFlow
#@save
def train_ch8(net, train_iter, vocab, lr, num_epochs, strategy,
              use_random_iter=False):
    """训练模型（定义见第8章）"""
    with strategy.scope():
        loss = tf.keras.losses.SparseCategoricalCrossentropy(
            from_logits=True)
        updater = tf.keras.optimizers.SGD(lr)
    animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',
                            legend=['train'], xlim=[10, num_epochs])
    predict = lambda prefix: predict_ch8(prefix, 50, net, vocab)
    # 训练和预测
    for epoch in range(num_epochs):
        ppl, speed = train_epoch_ch8(net, train_iter, loss, updater,
                                     use_random_iter)
        if (epoch + 1) % 10 == 0:
            print(predict('time traveller'))
            animator.add(epoch + 1, [ppl])
    device = d2l.try_gpu()._device_name
    print(f'困惑度 {ppl:.1f}, {speed:.1f} 词元/秒 {str(device)}')
    print(predict('time traveller'))
    print(predict('traveller'))
```

```PaddlePaddle
#@save
def train_ch8(net, train_iter, vocab, lr, num_epochs, device, use_random_iter=False):
    """训练模型（定义见第8章）"""
    loss = nn.CrossEntropyLoss()
    animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',
                            legend=['train'], xlim=[10, num_epochs])
    # 初始化
    if isinstance(net, nn.Layer):
        updater = paddle.optimizer.SGD(
                learning_rate=lr, parameters=net.parameters())
    else:
        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)
    predict = lambda prefix: predict_ch8(prefix, 50, net, vocab, device)
    # 训练和预测
    for epoch in range(num_epochs):
        ppl, speed = train_epoch_ch8(
            net, train_iter, loss, updater, device, use_random_iter)
        if (epoch + 1) % 10 == 0:
            print(predict('time traveller'))
            animator.add(epoch + 1, [ppl])
    print(f'困惑度 {ppl:.1f}, {speed:.1f} 词元/秒 {str(device)}')
    print(predict('time traveller'))
    print(predict('traveller'))
```

--------------------------------

### Check Individual Layer Weight Gradient Status (MXNet, PyTorch, PaddlePaddle)

Source: https://zh.d2l.ai/chapter_deep-learning-computation/parameters

Illustrates how to check the gradient status of a specific layer's weight parameter. Before backpropagation, gradients are typically in their initial (often zero or None) state, as shown in these examples.

```MXNet
net[1].weight.grad()
```

```PyTorch
net[2].weight.grad == None
```

```PaddlePaddle
net[2].weight.grad == None
```

--------------------------------

### Masked Softmax Function Usage in PyTorch, TensorFlow, and PaddlePaddle

Source: https://zh.d2l.ai/chapter_attention-mechanisms/attention-scoring-functions

Demonstrates the application of a `masked_softmax` function across different deep learning frameworks. Each example shows the function call with random input tensors and specific mask lengths, followed by the resulting masked softmax output tensor.

```Python
masked_softmax(torch.rand(2, 2, 4), torch.tensor([[1, 3], [2, 4]]))
```

```Python
tensor([[[1.0000, 0.0000, 0.0000, 0.0000],
         [0.4125, 0.3273, 0.2602, 0.0000]],

        [[0.5254, 0.4746, 0.0000, 0.0000],
         [0.3117, 0.2130, 0.1801, 0.2952]]])
```

```Python
masked_softmax(tf.random.uniform(shape=(2, 2, 4)), tf.constant([[1, 3], [2, 4]]))
```

```Python
<tf.Tensor: shape=(2, 2, 4), dtype=float32, numpy=
array([[[1.        , 0.        , 0.        , 0.        ],
        [0.40658087, 0.38102806, 0.21239112, 0.        ]],

       [[0.3735564 , 0.6264436 , 0.        , 0.        ],
        [0.3183936 , 0.22352162, 0.18998112, 0.26810366]]], dtype=float32)>
```

```Python
masked_softmax(paddle.rand((2, 2, 4)), paddle.to_tensor([[1, 3], [2, 4]]))
```

```Python
Tensor(shape=[2, 2, 4], dtype=float32, place=Place(cpu), stop_gradient=True,
       [[[1.        , 0.00000000, 0.00000000, 0.00000000],
         [0.35218778, 0.39164177, 0.25617039, 0.00000000]],

        [[0.53993517, 0.46006480, 0.00000000, 0.00000000],
         [0.18509592, 0.33311754, 0.25049153, 0.23129500]]])
```

--------------------------------

### Demonstrate Masked Softmax with 1D Valid Lengths

Source: https://zh.d2l.ai/chapter_attention-mechanisms/attention-scoring-functions

This example shows how to use the `masked_softmax` function with a 3D input tensor and a 1D array for `valid_lens`. The `valid_lens` array specifies the effective length for each sample in the batch, causing elements beyond these lengths to be masked to zero in the softmax output.

```Python (MXNet)
masked_softmax(np.random.uniform(size=(2, 2, 4)), np.array([2, 3]))
```

```Python (PyTorch)
masked_softmax(torch.rand(2, 2, 4), torch.tensor([2, 3]))
```

```Python (TensorFlow)
masked_softmax(tf.random.uniform(shape=(2, 2, 4)), tf.constant([2, 3]))
```

```Python (PaddlePaddle)
masked_softmax(paddle.rand((2, 2, 4)), paddle.to_tensor([2, 3]))
```
Tokens:
Raw