MindSpore

https://www.mindspore.cn/docs/en/
Admin
MindSpore is a full-scenario AI framework that integrates core framework, models, and development...
Tokens:3,336,149
Snippets:18,617
Trust Score:8.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### SyncBatchNorm Initialization and Usage Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/mint/mindspore.mint.nn.SyncBatchNorm.html

Demonstrates how to initialize and use SyncBatchNorm with multi-device communication setup. This example requires a distributed environment and specific communication configurations.

```python
import numpy as np
import mindspore
mindspore.communication.init()
mindspore.context.set_context(mode=mindspore.context.PYNATIVE_MODE, device_target="Ascend")
group = "0-1"
rank_ids = [0, 1]
mindspore.communication.create_group(group, rank_ids)
sync_batch_norm = mindspore.mint.nn.SyncBatchNorm(
    num_features=2, process_group=group, dtype=mindspore.float32
)
sync_batch_norm.set_train(False)
input_x = mindspore.tensor(np.linspace(0, 5, 2*2*2*2), mindspore.float32).reshape(2, 2, 2, 2)
output_data = sync_batch_norm(input_x)
```

--------------------------------

### MindSpore Installation Verification Example

Source: https://www.mindspore.cn/docs/en/stable/_sources/faq/installation.md.txt

This Python script demonstrates a basic MindSpore operation to verify installation. It performs an element-wise addition of two tensors.

```python
import numpy as np
import mindspore as ms
import mindspore.ops as ops

ms.set_device(device_target="Ascend")
x = ms.Tensor(np.ones([1,3,3,4]).astype(np.float32))
y = ms.Tensor(np.ones([1,3,3,4]).astype(np.float32))
print(ops.add(x,y))
```

--------------------------------

### All Gather Into Tensor Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.communication.all_gather_into_tensor.html

Demonstrates how to use all_gather_into_tensor to gather tensors across processes. Requires distributed environment setup and initialization.

```python
import numpy as np
import mindspore as ms
from mindspore import ops
from mindspore.ops.communication import init_process_group
from mindspore.ops.communication import all_gather_into_tensor
from mindspore import Tensor

ms.set_device(device_target="Ascend")
init_process_group()
input_tensor = Tensor(np.ones([2, 8]).astype(np.float32))
out_tensor = Tensor(np.zeros([4, 8]).astype(np.float32))
output = all_gather_into_tensor(out_tensor, input_tensor)
print(out_tensor)
```

--------------------------------

### PolynomialDecayLR Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.PolynomialDecayLR.html

Demonstrates the usage of PolynomialDecayLR to create a learning rate scheduler. This example shows how to initialize the scheduler with specific parameters and then use it to get the learning rate for a given global step. It also shows how to integrate this learning rate with an optimizer.

```python
>>> import mindspore
>>> from mindspore import Tensor, nn
>>> 
>>> learning_rate = 0.1
>>> end_learning_rate = 0.01
>>> decay_steps = 4
>>> power = 0.5
>>> global_step = Tensor(2, mindspore.int32)
>>> polynomial_decay_lr = nn.PolynomialDecayLR(learning_rate, end_learning_rate, decay_steps, power)
>>> lr = polynomial_decay_lr(global_step)
>>> net = nn.Dense(2, 3)
>>> optim = nn.SGD(net.trainable_params(), learning_rate=lr)

```

--------------------------------

### Get Process Group Ranks Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.communication.get_process_group_ranks.html

Demonstrates how to initialize a process group and retrieve the ranks of all processes within that group using get_process_group_ranks. This example requires a distributed environment with 4 devices.

```Python
import mindspore as ms
from mindspore.ops.communication import init_process_group, get_process_group_ranks
# Launch 4 processes.
ms.set_device(device_target="Ascend")
init_process_group()
output = get_process_group_ranks()
print(output)
```

--------------------------------

### Get Peak Memory Allocated by Tensors

Source: https://www.mindspore.cn/docs/en/stable/api_python/hal/mindspore.hal.max_memory_allocated.html

This example demonstrates how to use mindspore.hal.max_memory_allocated to retrieve the peak memory size occupied by Tensors since the process started. Note that this API is deprecated.

```python
>>> import mindspore
>>> a = mindspore.tensor(mindspore.ops.ones([1, 2]), mindspore.float32)
>>> b = mindspore.tensor(mindspore.ops.ones([1, 2]), mindspore.float32)
>>> c = mindspore.ops.add(a, b).asnumpy()
>>> print(mindspore.hal.max_memory_allocated())
1536
```

--------------------------------

### Get Multiprocessing Start Method

Source: https://www.mindspore.cn/docs/en/stable/api_python/dataset/mindspore.dataset.config.get_multiprocessing_start_method.html

Retrieves the global configuration of the multiprocessing start method. This is useful when the main process needs to get data from subprocesses. The default value is 'fork' if `set_multiprocessing_start_method` has not been called.

```python
>>> # Get the global configuration of multiprocessing start method when main process gets data
>>> # from subprocesses.
>>> # default ``fork`` will be returned.
>>> import mindspore.dataset as ds
>>> multiprocessing_start_method = ds.config.get_multiprocessing_start_method()
```

--------------------------------

### WarmUpLR Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.WarmUpLR.html

Demonstrates how to initialize and use the WarmUpLR scheduler to adjust the learning rate during training. This example shows setting up the scheduler and applying it to a global step.

```python
>>> import mindspore
>>> from mindspore import Tensor, nn
>>> 
>>> learning_rate = 0.1
>>> warmup_steps = 2
>>> global_step = Tensor(2, mindspore.int32)
>>> warmup_lr = nn.WarmUpLR(learning_rate, warmup_steps)
>>> lr = warmup_lr(global_step)
>>> net = nn.Dense(2, 3)
>>> optim = nn.SGD(net.trainable_params(), learning_rate=lr)
```

--------------------------------

### Get Device Capability Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/hal/mindspore.hal.get_device_capability.html

Demonstrates how to get the capability of a specified device using mindspore.hal.get_device_capability. This example retrieves the device target from the current context and then calls the function with device ID 0.

```python
>>> import mindspore
>>> device_target = mindspore.context.get_context("device_target")
>>> print(mindspore.hal.get_device_capability(0, device_target))
```

--------------------------------

### AlltoAllV Basic Usage

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.AlltoAllV.html

Demonstrates the basic usage of AlltoAllV for distributed communication. This example requires a distributed environment with at least two devices and should be run using the msrun startup method. It shows how to initialize communication, define a network cell with AlltoAllV, and perform uneven data scattering and gathering.

```Python
import mindspore.nn as nn
from mindspore.communication import init, get_rank
from mindspore.ops import AlltoAllV
from mindspore import Tensor

init()
rank = get_rank()
class Net(nn.Cell):
    def __init__(self):
        super(Net, self).__init__()
        self.all_to_all = AlltoAllV()

    def construct(self, x, send_numel_list, recv_numel_list):
        return self.all_to_all(x, send_numel_list, recv_numel_list)
send_numel_list = []
recv_numel_list = []
if rank == 0:
   send_tensor = Tensor([0, 1, 2.])
   send_numel_list = [1, 2]
   recv_numel_list = [1, 2]
elif rank == 1:
   send_tensor = Tensor([3, 4, 5.])
   send_numel_list = [2, 1]
   recv_numel_list = [2, 1]
net = Net()
output = net(send_tensor, send_numel_list, recv_numel_list)
print(output)
```

--------------------------------

### SyncBatchNorm Usage Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.SyncBatchNorm.html

Demonstrates how to initialize and use SyncBatchNorm with multiple devices. Ensure communication environment variables are configured and run with multiple devices using msrun.

```python
>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore.communication import init
>>> 
>>> ms.set_context(mode=ms.GRAPH_MODE)
>>> init()
>>> ms.reset_auto_parallel_context()
>>> ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.DATA_PARALLEL)
>>> sync_bn_op = ms.nn.SyncBatchNorm(num_features=3, process_groups=[[0, 1], [2, 3]])
>>> x = ms.Tensor(np.ones([1, 3, 2, 2]), ms.float32)
>>> output = sync_bn_op(x)
>>> print(output)
[[[[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]]]

```

--------------------------------

### mindspore.dataset.config.get_multiprocessing_start_method

Source: https://www.mindspore.cn/docs/en/stable/api_python/dataset/mindspore.dataset.config.get_multiprocessing_start_method.html

Get the global configuration of multiprocessing start method. If set_multiprocessing_start_method is never called before, the default start method is 'fork'.

```APIDOC
## mindspore.dataset.config.get_multiprocessing_start_method

### Description
Get the global configuration of multiprocessing start method. If `set_multiprocessing_start_method` is never called before, the default start method is `'fork'`.

### Returns
- **str**: multiprocessing start method.

### Examples
```python
>>> # Get the global configuration of multiprocessing start method when main process gets data
>>> # from subprocesses.
>>> # default ``fork`` will be returned.
>>> import mindspore.dataset as ds
>>> multiprocessing_start_method = ds.config.get_multiprocessing_start_method()
```
```

--------------------------------

### Initialize and Use TCPStore

Source: https://www.mindspore.cn/docs/en/stable/api_python/mint/mindspore.mint.distributed.TCPStore.html

Demonstrates how to initialize a TCPStore as a master node, set a key-value pair, and retrieve the value.

```python
from mindspore.mint.distributed import TCPStore
store = TCPStore("127.0.0.1", 1234, is_master=True)
store.set("first_key", "first_value")
data = store.get("first_key")
print(data)
```

--------------------------------

### MultiClassDiceLoss Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.MultiClassDiceLoss.html

Demonstrates how to initialize and use the MultiClassDiceLoss for multi-class classification. This example shows the basic setup with logits and labels tensors.

```python
>>> import mindspore
>>> from mindspore import Tensor, nn
>>> import numpy as np
>>> loss = nn.MultiClassDiceLoss(weights=None, ignore_indiex=None, activation="softmax")
>>> logits = Tensor(np.array([[0.2, 0.5, 0.7], [0.3, 0.1, 0.5], [0.9, 0.6, 0.3]]), mindspore.float32)
>>> labels = Tensor(np.array([[0, 1, 0], [1, 0, 0], [0, 0, 1]]), mindspore.float32)
>>> output = loss(logits, labels)
>>> print(output)
0.54958105
```

--------------------------------

### TransformerEncoder Usage Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.TransformerEncoder.html

Demonstrates how to initialize and use the TransformerEncoder module with a TransformerEncoderLayer. This example shows the basic setup for processing input sequences.

```python
import mindspore as ms
import numpy as np
encoder_layer = ms.nn.TransformerEncoderLayer(d_model=512, nhead=8)
transformer_encoder = ms.nn.TransformerEncoder(encoder_layer, num_layers=6)
src = ms.Tensor(np.random.rand(10, 32, 512), ms.float32)
out = transformer_encoder(src)
print(out.shape)
```

--------------------------------

### P2POp Initialization Examples

Source: https://www.mindspore.cn/docs/en/stable/api_python/mint/mindspore.mint.distributed.P2POp.html

Demonstrates initializing P2POp objects for both 'isend' and 'irecv' operations using string literals and function references. Ensure you have launched multiple processes for these examples to function correctly.

```python
import numpy as np
import mindspore
from mindspore.mint.distributed import P2POp, isend, irecv
from mindspore import Tensor
# Launch 2 processes.
send_tensor = Tensor(1.)
send_op = P2POp('isend', send_tensor, 1)
send_op = P2POp(isend, send_tensor, 1)
recv_tensor = Tensor(0.)
recv_op = P2POp('irecv', recv_tensor, 0)
recv_op = P2POp(irecv, recv_tensor, 0)
```

--------------------------------

### AllGather Operation Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.AllGather.html

Demonstrates how to use the AllGather operation in a distributed environment. This example requires at least 2 devices and should be run using the msrun startup method. Ensure communication environment variables are configured.

```Python
import numpy as np
import mindspore as ms
from mindspore import ops
import mindspore.nn as nn
from mindspore.communication import init
from mindspore import Tensor

ms.set_context(mode=ms.GRAPH_MODE)
init()
class Net(nn.Cell):
    def __init__(self):
        super(Net, self).__init__()
        self.allgather = ops.AllGather()

    def construct(self, x):
        return self.allgather(x)

input_x = Tensor(np.ones([2, 8]).astype(np.float32))
net = Net()
output = net(input_x)
print(output)
```

--------------------------------

### Create Process and Execute Task with 'spawn'

Source: https://www.mindspore.cn/docs/en/stable/_sources/api_python/mindspore.multiprocessing.md.txt

Illustrates creating a child process using the 'spawn' start method. The child process explicitly sets its device target to 'CPU', while the parent uses 'Ascend'. Results are retrieved via a queue.

```python
from mindspore import Tensor, ops, context
import mindspore.multiprocessing as multiprocessing

def child_process(q):
    # Child process may not be able to acquire resources when using the same resources as parent process.
    context.set_context(device_target="CPU")
    y = ops.log(Tensor(2.0))
    q.put(y)
    return

if __name__ == '__main__':
    context.set_context(device_target="Ascend")
    multiprocessing.set_start_method("spawn", force=True)
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=child_process, args=(q,))
    p.start()
    print("child process:ops.log(Tensor(2.0))=", q.get())
    p.join()
    # Child process may not be able to acquire resources when this compute task is executed before the child process is created.
    print("parent process:ops.log(Tensor(2.0))=", ops.log(Tensor(2.0)))
```

--------------------------------

### LSTM Operation Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.LSTM.html

Demonstrates how to initialize and use the LSTM operation with sample input tensors. This example shows the basic setup for a single-layer LSTM.

```Python
>>> import numpy as np
>>> from mindspore import Tensor, ops
>>> input_size = 10
>>> hidden_size = 2
>>> num_layers = 1
>>> seq_len = 5
>>> batch_size = 2
>>>
>>> net = ops.LSTM(input_size, hidden_size, num_layers, True, False, 0.0)
>>> input_tensor = Tensor(np.ones([seq_len, batch_size, input_size]).astype(np.float32))
>>> h0 = Tensor(np.ones([num_layers, batch_size, hidden_size]).astype(np.float32))
>>> c0 = Tensor(np.ones([num_layers, batch_size, hidden_size]).astype(np.float32))
>>> w = Tensor(np.ones([112, 1, 1]).astype(np.float32))
>>> output, hn, cn, _, _ = net(input_tensor, h0, c0, w)
>>> print(output)
[[[0.9640267  0.9640267 ]
  [0.9640267  0.9640267 ]]
 [[0.9950539  0.9950539 ]
  [0.9950539  0.9950539 ]]
 [[0.99932843 0.99932843]
  [0.99932843 0.99932843]]
 [[0.9999084  0.9999084 ]
  [0.9999084  0.9999084 ]]
 [[0.9999869  0.9999869 ]
  [0.9999869  0.9999869 ]]]
```

--------------------------------

### Get Tensor Strides

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/Tensor/mindspore.Tensor.strides.html

Demonstrates how to get the strides of a tensor. The strides indicate the memory layout of the tensor. This example creates a 2x2 tensor and prints its strides.

```Python
from mindspore import Tensor
from mindspore import dtype as mstype
import numpy as np
x = Tensor(np.array([[1, 2], [3, 4]]), dtype=mstype.int64)
output = x.strides
print(output)
```

--------------------------------

### AllReduce SUM Operation Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.AllReduce.html

Demonstrates how to use the AllReduce operation with the SUM reduction operator. This example requires a distributed environment with at least two devices and should be run using the msrun startup method. Ensure communication environment variables are configured.

```python
import numpy as np
from mindspore.communication import init
from mindspore import Tensor
from mindspore.ops import ReduceOp
import mindspore.nn as nn
from mindspore import ops

init()
class Net(nn.Cell):
    def __init__(self):
        super(Net, self).__init__()
        self.allreduce_sum = ops.AllReduce(ReduceOp.SUM)

    def construct(self, x):
        return self.allreduce_sum(x)

input_ = Tensor(np.ones([2, 8]).astype(np.float32))
net = Net()
output = net(input_)
print(output)
```

--------------------------------

### ASGD Optimizer Initialization and Training Step

Source: https://www.mindspore.cn/docs/en/stable/api_python/experimental/optim/mindspore.experimental.optim.ASGD.html

Demonstrates how to initialize the ASGD optimizer with a network's trainable parameters and set up a basic training step using value_and_grad. This example requires a pre-defined LeNet5 network and a loss function.

```python
import mindspore
from mindspore import nn
from mindspore.experimental import optim
# Define the network structure of LeNet5. Refer to
# https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/lenet.py
net = LeNet5()
loss_fn = nn.SoftmaxCrossEntropyWithLogits(sparse=True)
optimizer = optim.ASGD(net.trainable_params(), lr=0.1)
def forward_fn(data, label):
    logits = net(data)
    loss = loss_fn(logits, label)
    return loss, logits
grad_fn = mindspore.value_and_grad(forward_fn, None, optimizer.parameters, has_aux=True)
def train_step(data, label):
    (loss, _), grads = grad_fn(data, label)
    optimizer(grads)
    return loss

```

--------------------------------

### Get Image Channels

Source: https://www.mindspore.cn/docs/en/stable/api_python/dataset_vision/mindspore.dataset.vision.get_image_num_channels.html

Demonstrates how to get the number of channels from a PIL Image object using the get_image_num_channels function. Ensure the PIL library is installed and the image path is valid.

```python
import mindspore.dataset.vision as vision
from PIL import Image
image = Image.open("/path/to/image_file")
num_channels = vision.get_image_num_channels(image)
```

--------------------------------

### Build and Train a Model

Source: https://www.mindspore.cn/docs/en/stable/api_python/train/mindspore.train.Model.html

Demonstrates how to build a model with a specified dataset and then train it for a set number of epochs. This example assumes the existence of `create_dataset` and `LeNet5` functions/classes.

```python
>>> from mindspore import nn
>>> from mindspore.train import Model
>>> from mindspore.amp import FixedLossScaleManager
>>> 
>>> # Create the dataset taking MNIST as an example. Refer to
>>> # https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/mnist.py
>>> dataset = create_dataset()
>>> # Define the network structure of LeNet5. Refer to
>>> # https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/lenet.py
>>> net = LeNet5()
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> loss_scale_manager = FixedLossScaleManager()
>>> optim = nn.Momentum(params=net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> model = Model(net, loss_fn=loss, optimizer=optim, metrics=None,
>>>                  loss_scale_manager=loss_scale_manager)
>>> model.build(dataset, epoch=2)
>>> model.train(2, dataset)
```

--------------------------------

### Configure Boost Directly

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore.boost.html

Demonstrates how to directly configure boost settings using a Python dictionary.

```python
from mindspore.boost import AutoBoost
#1) when configuring the dict directly:
boost_config_dict = {"boost": {"mode": "auto"}}
boost = AutoBoost("O1", boost_config_dict)
```

--------------------------------

### Get Memory Reserved (CPU)

Source: https://www.mindspore.cn/docs/en/stable/api_python/hal/mindspore.hal.memory_reserved.html

This example demonstrates how to get the total amount of memory managed by the memory pool for the CPU device. For CPU, this function always returns 0.

```python
>>> import mindspore
>>> a = mindspore.tensor(mindspore.ops.ones([1, 2]), mindspore.float32)
>>> b = mindspore.tensor(mindspore.ops.ones([1, 2]), mindspore.float32)
>>> c = mindspore.ops.add(a, b).asnumpy()
>>> print(mindspore.hal.memory_reserved())
```

--------------------------------

### PolynomialLR Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/experimental/optim/mindspore.experimental.optim.lr_scheduler.PolynomialLR.html

Demonstrates how to initialize and use PolynomialLR to adjust learning rates over several epochs. It shows the learning rate decay and how it reaches zero.

```python
from mindspore import nn
from mindspore.experimental import optim
class Net(nn.Cell):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Dense(16 * 5 * 5, 120)
    def construct(self, x):
        return self.fc(x)
net = Net()
optimizer = optim.Adam(net.trainable_params(), 0.01)
scheduler = optim.lr_scheduler.PolynomialLR(optimizer)
for i in range(6):
    scheduler.step()
    current_lr = scheduler.get_last_lr()
    print(current_lr)
```

--------------------------------

### Get Image Size using PIL Image

Source: https://www.mindspore.cn/docs/en/stable/api_python/dataset_vision/mindspore.dataset.vision.get_image_size.html

Demonstrates how to get the size of an image using a PIL Image object. Ensure the PIL library is installed and the image path is valid.

```python
import mindspore.dataset.vision as vision
from PIL import Image
image = Image.open("/path/to/image_file")
image_size = vision.get_image_size(image)
```

--------------------------------

### Get non-zero positions as a 2D Tensor

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/Tensor/mindspore.Tensor.nonzero.html

This example demonstrates how to use `nonzero()` to get the positions of non-zero elements. The output is a 2D tensor where each row represents the indices of a non-zero element.

```python
import mindspore
import numpy as np
from mindspore import Tensor
x = Tensor(np.array([[[1,  0], [-5, 0]]]), mindspore.int32)
output = x.nonzero()
print(output)
```

--------------------------------

### Get Device Name Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/hal/mindspore.hal.get_device_name.html

Demonstrates how to get the name of a specified device using mindspore.hal.get_device_name. This function is deprecated and will be removed in future versions. It always returns 'CPU' for CPU devices.

```python
>>> import mindspore
>>> device_target = mindspore.context.get_context("device_target")
>>> print(mindspore.hal.get_device_name(0, device_target))
```

--------------------------------

### Initialize Model with Network, Loss, Optimizer, and Metrics

Source: https://www.mindspore.cn/docs/en/stable/api_python/train/mindspore.train.Model.html

This snippet demonstrates how to initialize the Model class with a defined network, loss function, optimizer, and metrics. It also shows how to access the training, prediction, and evaluation networks after initialization. The example assumes the existence of a LeNet5 network definition and a dataset creation function.

```python
>>> from mindspore import nn
>>> from mindspore.train import Model
>>>
>>> # Define the network structure of LeNet5. Refer to
>>> # https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/lenet.py
>>> net = LeNet5()
>>> loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True)
>>> optim = nn.Momentum(params=net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> model = Model(net, loss_fn=loss, optimizer=optim, metrics=None)
>>> model.train_network
>>> model.predict_network
>>> model.eval_network
>>> # Create the dataset taking MNIST as an example. Refer to
>>> # https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/mnist.py
>>> dataset = create_dataset()
>>> model.train(2, dataset)
```

--------------------------------

### Get Tensor Memory Allocation

Source: https://www.mindspore.cn/docs/en/stable/api_python/runtime/mindspore.runtime.memory_allocated.html

Demonstrates how to use mindspore.runtime.memory_allocated to get the memory size occupied by Tensors. This example requires setting the device to Ascend or GPU. For CPU, it will always return 0.

```python
>>> import mindspore as ms
>>> import numpy as np
>>> from mindspore import Tensor, ops
>>> ms.set_device("Ascend", 0)
>>> a = Tensor(np.ones([1, 2]), ms.float32)
>>> b = Tensor(np.ones([1, 2]), ms.float32)
>>> c = ops.add(a, b).asnumpy()
>>> print(ms.runtime.memory_allocated())
1024
```

--------------------------------

### Using BackupAndRestore Callback

Source: https://www.mindspore.cn/docs/en/stable/api_python/train/mindspore.train.BackupAndRestore.html

Demonstrates how to initialize and use the BackupAndRestore callback during model training. This snippet shows the setup for a typical training loop with checkpoint backup.

```python
from mindspore import nn
from mindspore.train import Model, BackupAndRestore, RunContext

# Define the network structure of LeNet5. Refer to
# https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/lenet.py
net = LeNet5()
loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')
optim = nn.Momentum(net.trainable_params(), 0.01, 0.9)
model = Model(net, loss_fn=loss, optimizer=optim)
# Create the dataset taking MNIST as an example. Refer to
# https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/mnist.py
dataset = create_dataset()
backup_ckpt = BackupAndRestore("backup")
model.train(10, dataset, callbacks=backup_ckpt)
```

--------------------------------

### Get non-zero positions as a tuple of Tensors

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/Tensor/mindspore.Tensor.nonzero.html

This example demonstrates using `nonzero(as_tuple=True)` to get non-zero element positions as a tuple of tensors. Each tensor in the tuple corresponds to the indices along one dimension.

```python
x = Tensor(np.array([[[1,  0], [-5, 0]]]), mindspore.int32)
output = x.nonzero(as_tuple=True)
print(output)
```

--------------------------------

### AlltoAllVC Basic Usage

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.AlltoAllVC.html

Demonstrates the basic usage of AlltoAllVC for distributed communication. This example requires a distributed environment with at least 2 devices and should be run using the msrun startup method. Ensure communication environment variables are configured.

```python
from mindspore.ops import AlltoAllVC
import mindspore.nn as nn
from mindspore.communication import init, get_rank
from mindspore import Tensor

init()
rank = get_rank()
class Net(nn.Cell):
    def __init__(self):
        super(Net, self).__init__()
        self.all_to_all_v_c = AlltoAllVC()

    def construct(self, x, send_count_matrix):
        return self.all_to_all_v_c(x, send_count_matrix)
send_count_matrix = Tensor([[0, 3], [3, 0]])
send_tensor = Tensor([0, 1, 2.]) * rank
net = Net()
output = net(send_tensor, send_count_matrix)
print(output)
```

--------------------------------

### InverseDecayLR Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.InverseDecayLR.html

Demonstrates how to initialize and use the InverseDecayLR scheduler to calculate learning rates. This example shows setting up the scheduler with specific decay parameters and applying it to a global step.

```python
>>> import mindspore
>>> from mindspore import Tensor, nn
>>> 
>>> learning_rate = 0.1
>>> decay_rate = 0.9
>>> decay_steps = 4
>>> global_step = Tensor(2, mindspore.int32)
>>> inverse_decay_lr = nn.InverseDecayLR(learning_rate, decay_rate, decay_steps, True)
>>> lr = inverse_decay_lr(global_step)
>>> net = nn.Dense(2, 3)
>>> optim = nn.SGD(net.trainable_params(), learning_rate=lr)
```

--------------------------------

### AdaptiveMaxPool3d Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.AdaptiveMaxPool3d.html

Demonstrates the usage of AdaptiveMaxPool3d with a specified output size and return_indices set to True. This example shows how to initialize the layer and pass an input tensor to get the pooled output and the indices of the maximum values.

```python
>>> import mindspore as ms
>>> import numpy as np
>>> input = ms.Tensor(np.arange(0,36).reshape((1, 3, 3, 4)).astype(np.float32))
>>> output_size = (1, 1, 2)
>>> net = ms.nn.AdaptiveMaxPool3d(output_size, True)
>>> output = net(input)
>>> print(output[0].asnumpy())
[[[[33. 35.]]]]
>>> print(output[1].asnumpy())
[[[[33 35]]]]

```

--------------------------------

### Custom ConstantLR Scheduler Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/experimental/optim/mindspore.experimental.optim.lr_scheduler.LRScheduler.html

Demonstrates how to define and use a custom ConstantLR scheduler. This example shows the initialization of a custom scheduler, its integration with an optimizer, and how to step through epochs to observe learning rate changes.

```python
from mindspore import nn
from mindspore.experimental import optim

class ConstantLR(optim.lr_scheduler.LRScheduler):
    def __init__(self, optimizer, factor=0.5, total_iters=3, last_epoch=-1):
        self.factor = factor
        self.total_iters = total_iters
        super(ConstantLR, self).__init__(optimizer, last_epoch)

    def get_lr(self):
        if self.last_epoch == 0:
            return [lr * self.factor for lr in self._last_lr]
        if self.last_epoch != self.total_iters:
            return [lr * 1. for lr in self._last_lr]
        return [lr / self.factor for lr in self._last_lr]

net = nn.Dense(8, 2)
optimizer = optim.SGD(net.trainable_params(), 0.01)
scheduler = ConstantLR(optimizer)
for i in range(4):
    scheduler.step()
    current_lr = scheduler.get_last_lr()
    print(current_lr)
```

--------------------------------

### Get Overall Maximum Value of a 2D Tensor

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/Tensor/mindspore.Tensor.max.html

This example illustrates retrieving the single maximum value from a 2D tensor when no axis is specified. The `return_indices=True` parameter can be used to also get the index of this maximum value.

```python
>>> import numpy as np
>>> from mindspore import Tensor
>>> a = Tensor(np.arange(4).reshape((2, 2)).astype('float32'))
>>> output = a.max()
>>> print(output)
3.0
```

--------------------------------

### Get the number of samples from a sampler

Source: https://www.mindspore.cn/docs/en/stable/api_python/dataset/mindspore.dataset.PKSampler.html

This example demonstrates how to get the total number of samples that will be drawn by a sampler, considering its own num_samples and any child samplers. The logic for combining num_samples from parent and child samplers is detailed in the documentation.

```python
import mindspore.dataset as ds

sampler = ds.SequentialSampler(start_index=0, num_samples=3)
num_samplers = sampler.get_num_samples()
```

--------------------------------

### Create Process and Execute Task with 'fork'

Source: https://www.mindspore.cn/docs/en/stable/_sources/api_python/mindspore.multiprocessing.md.txt

Demonstrates creating a child process using the 'fork' start method to execute a task and retrieve results via a queue. The parent process sets the device target to 'Ascend'.

```python
from mindspore import Tensor, ops, context
import mindspore.multiprocessing as multiprocessing

def child_process(q):
    y = ops.log(Tensor(2.0))
    q.put(y)
    return

if __name__ == '__main__':
    context.set_context(device_target="Ascend")
    # Child process may not be able to acquire resources when start_method is set to 'spawn' . 
    multiprocessing.set_start_method("fork", force=True)
    print("parent process:ops.log(Tensor(2.0))=", ops.log(Tensor(2.0)))
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=child_process, args=(q,))
    p.start()
    print("child process:ops.log(Tensor(2.0))=", q.get())
    p.join()
```

--------------------------------

### Generate a tensor with a step of 2

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.range.html

This example demonstrates how to use mindspore.ops.range to create a tensor with a specified start, end, and step value. The output tensor contains elements starting from 0, incrementing by 2, up to (but not including) 6.

```python
>>> import mindspore
>>> mindspore.ops.range(0, 6, 2)
Tensor(shape=[3], dtype=Int64, value= [0, 2, 4])
```

--------------------------------

### Configure Boost from JSON File

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore.boost.html

Shows how to load boost configurations from a JSON file. Ensure the JSON file exists and is correctly formatted.

```python
import json
boost_json = "/path/boost_config.json"
with open(boost_json, 'r') as fp:
    boost_config_dict = json.load(fp)
boost = AutoBoost("O1", boost_config_dict)
```

--------------------------------

### Get Local Rank ID

Source: https://www.mindspore.cn/docs/en/stable/api_python/communication/mindspore.communication.get_local_rank.html

This example demonstrates how to initialize the communication environment, get the global rank, and then retrieve the local rank ID using `get_local_rank`. This method is intended for Ascend devices and requires the communication environment to be initialized.

```python
>>> import mindspore as ms
>>> from mindspore.communication import init, get_rank, get_local_rank
>>> ms.set_device(device_target="Ascend")
>>> init()
>>> world_rank = get_rank()
>>> local_rank = get_local_rank()
>>> print("local_rank is: {}, world_rank is {}".format(local_rank, world_rank))
local_rank is: 1, world_rank is 9
```

--------------------------------

### Get Loss Scale Value

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.DynamicLossScaleUpdateCell.html

This example shows how to retrieve the current loss scale value from an initialized `DynamicLossScaleUpdateCell` instance.

```python
>>> from mindspore import nn
>>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=212, scale_factor=2, scale_window=1000)
>>> output = manager.get_loss_scale()
>>> print(output)
212
```

--------------------------------

### NaturalExpDecayLR Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/nn/mindspore.nn.NaturalExpDecayLR.html

Demonstrates how to initialize and use NaturalExpDecayLR to calculate a decayed learning rate. This example also shows how to integrate the calculated learning rate with an optimizer.

```python
import mindspore
from mindspore import Tensor, nn

learning_rate = 0.1
decay_rate = 0.9
decay_steps = 4
global_step = Tensor(2, mindspore.int32)
natural_exp_decay_lr = nn.NaturalExpDecayLR(learning_rate, decay_rate, decay_steps, True)
lr = natural_exp_decay_lr(global_step)
net = nn.Dense(2, 3)
optim = nn.SGD(net.trainable_params(), learning_rate=lr)
```

--------------------------------

### Model Initialization and Usage

Source: https://www.mindspore.cn/docs/en/stable/api_python/train/mindspore.train.Model.html

Demonstrates how to initialize a Model object with a network, loss function, optimizer, and metrics, and then use the fit method for training with provided datasets.

```APIDOC
## Model Class

### Description

The `Model` class provides a high-level API for training and evaluating neural networks. It simplifies the process by encapsulating the network, loss function, optimizer, and metrics.

### Parameters

*   **epoch** (_int_) – Total training epochs. Generally, train network will be trained on complete dataset per epoch. If dataset_sink_mode is set to True and sink_size is greater than 0, each epoch will train sink_size steps instead of total steps of dataset. If epoch used with initial_epoch, it is to be understood as "final epoch".
*   **train_dataset** (_Dataset_) – A training dataset iterator. If loss_fn is defined, the data and label will be passed to the network and the loss_fn respectively, so a tuple (data, label) should be returned from dataset. If there is multiple data or labels, set loss_fn to None and implement calculation of loss in network, then a tuple (data1, data2, data3, …) with all data returned from dataset will be passed to the network.
*   **valid_dataset** (_Dataset_ _,__optional_) – Dataset to evaluate the model. If valid_dataset is provided, evaluation process will be performed on the end of training process. Default: `None` .
*   **valid_frequency** (_int_ _,__list_ _,__optional_) – Only relevant if valid_dataset is provided. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. valid_frequency=2 runs validation every 2 epochs. If a list, specifies the epochs on which to run validation, e.g. valid_frequency=[1, 5] runs validation at the end of the 1st, 5th epochs. Default: `1` .
*   **callbacks** (_[__list_ _[__Callback_ _]__,__Callback_ _]__,__optional_) – List of callback objects or callback object, which should be executed while training. Default: `None` .
*   **dataset_sink_mode** (_bool_ _,__optional_) – Determines whether to pass the train data through dataset channel. Configure pynative mode or CPU, the training process will be performed with dataset not sink. Default: `False` .
*   **valid_dataset_sink_mode** (_bool_ _,__optional_) – Determines whether to pass the validation data through dataset channel. Default: `False` .
*   **sink_size** (_int_ _,__optional_) – Control the number of steps for each sinking. sink_size is invalid if dataset_sink_mode is False. If sink_size = -1, sink the complete dataset for each epoch. If sink_size > 0, sink sink_size data for each epoch. Default: `-1` .
*   **initial_epoch** (_int_ _,__optional_) – Epoch at which to start train, it useful for resuming a previous training run. Default: `0` .

### Example

```python
>>> from mindspore import nn
>>> from mindspore.train import Model
>>>
>>> # Create the dataset taking MNIST as an example. Refer to
>>> # https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/mnist.py
>>> train_dataset = create_dataset("train")
>>> valid_dataset = create_dataset("test")
>>> # Define the network structure of LeNet5. Refer to
>>> # https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/lenet.py
>>> net = LeNet5()
>>> loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True)
>>> optim = nn.Momentum(params=net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> model = Model(net, loss_fn=loss, optimizer=optim, metrics={"accuracy"})
>>> model.fit(2, train_dataset, valid_dataset)
```
```

--------------------------------

### Accessing Parameter's Slice Status

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/mindspore.Parameter.html

Get the slice status of the parameter. This example shows how to set and retrieve the sliced attribute.

```python
>>> from mindspore import Tensor, Parameter
>>> import numpy as np
>>> x = Parameter(Tensor(np.array([1, 2], dtype=np.float32)), name="param")
>>> x.sliced = True
>>> x.sliced
True
```

--------------------------------

### Initialize and Set Mode for SummaryRecord

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/mindspore.SummaryRecord.html

Demonstrates how to initialize SummaryRecord with a specified log directory, file prefix, and file suffix, and then set the recording mode to 'eval'. Ensure the log directory exists or can be created.

```python
import mindspore as ms
if __name__ == '__main__':
    with ms.SummaryRecord(log_dir="./summary_dir", file_prefix="xx_", file_suffix="_yy") \
            as summary_record:
        summary_record.set_mode('eval')
```

--------------------------------

### Accessing and Modifying Parameter's Name

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/mindspore.Parameter.html

Get the name of the parameter. This example demonstrates how to set and retrieve the parameter's name.

```python
>>> from mindspore import Tensor, Parameter
>>> import numpy as np
>>> x = Parameter(Tensor(np.array([1, 2], dtype=np.float32)), name="param")
>>> x.name = "param1"
>>> x.name
'param1'
```

--------------------------------

### Loading Parameters into a Network

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/mindspore.load_param_into_net.html

This example demonstrates how to load parameters from a checkpoint file into a network using `mindspore.load_param_into_net`. It shows how to filter parameters during loading and prints the names of parameters that were not loaded.

```python
import mindspore as ms

# Define the network structure of LeNet5. Refer to
# https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/lenet.py
net = LeNet5()
ckpt_file_name = "./checkpoint/LeNet5-1_32.ckpt"
param_dict = ms.load_checkpoint(ckpt_file_name, filter_prefix="conv1")
param_not_load, _ = ms.load_param_into_net(net, param_dict)
print(param_not_load)
```

--------------------------------

### MultiStepLR Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/experimental/optim/mindspore.experimental.optim.lr_scheduler.MultiStepLR.html

Demonstrates how to use MultiStepLR to adjust learning rates based on epoch milestones. This example initializes a Dense network, an Adam optimizer, and then sets up the MultiStepLR scheduler with specific milestones and gamma. It iterates through epochs, steps the scheduler, and prints the current learning rate for each group.

```python
from mindspore import nn
from mindspore.experimental import optim
net = nn.Dense(2, 3)
optimizer = optim.Adam(net.trainable_params(), 0.05)
# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.05     if epoch < 2
# lr = 0.005    if 2 <= epoch < 4
# lr = 0.0005   if epoch >= 4
scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[2,4], gamma=0.1)
for i in range(6):
    scheduler.step()
    current_lr = scheduler.get_last_lr()
    print(current_lr)

```

--------------------------------

### Using jit with backend and jit_level

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/mindspore.jit.html

This example demonstrates how to use the `jit` decorator with specified `backend` and `jit_level` for graph compilation.

```APIDOC
## @jit(backend="ms_backend", jit_level="O1")

### Description
Decorates a Python function to create a callable MindSpore graph with a specified backend and optimization level.

### Parameters
#### backend
- **backend** (str) - The backend to use for graph compilation. Example: "ms_backend".
#### jit_level
- **jit_level** (str) - The optimization level for JIT compilation. Example: "O1".

### Request Example
```python
from mindspore import jit

@jit(backend="ms_backend", jit_level="O1")
def tensor_add_by_trace(x, y):
    z = x + y
    return z
```

### Response
Returns a callable MindSpore graph.
```

--------------------------------

### ReduceScatter Example

Source: https://www.mindspore.cn/docs/en/stable/api_python/ops/mindspore.ops.communication.reduce_scatter.html

Demonstrates how to use reduce_scatter to reduce and scatter tensors across devices. This example requires at least 2 devices and proper communication environment setup. The output tensor will contain the sum of the input tensors divided by the number of devices.

```python
from mindspore import Tensor
from mindspore.ops.communication import init_process_group
from mindspore.ops.communication import reduce_scatter
import numpy as np

init_process_group()
input_tensors = [Tensor(np.ones([4, 8]).astype(np.float32)), Tensor(np.ones([4, 8]).astype(np.float32))]
output_tensor = Tensor(np.zeros([4, 8]).astype(np.float32))
output = reduce_scatter(output_tensor ,input_tensors)
print(output_tensor)
```

--------------------------------

### Using jit with fullgraph=True

Source: https://www.mindspore.cn/docs/en/stable/api_python/mindspore/mindspore.jit.html

This example demonstrates how to use the `jit` decorator with the `fullgraph=True` option to capture the entire function into the graph.

```APIDOC
## @jit(fullgraph=True)

### Description
Decorates a Python function to compile it into a callable MindSpore graph, capturing the entire function into the graph.

### Parameters
#### fullgraph
- **fullgraph** (bool) - If True, captures the entire function into the graph. Defaults to False.

### Request Example
```python
from mindspore import jit

@jit(fullgraph=True)
def tensor_add_fullgraph(x, y):
    z = x + y
    return z
```

### Response
Returns a callable MindSpore graph.
```
Tokens:
Raw