PopTorch

https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/
Admin
PopTorch is a set of PyTorch extensions enabling PyTorch models to run directly on Graphcore IPU...
Tokens:77,811
Snippets:750
Trust Score:9.9
Update:5 months ago
Show doc for...
Tokens:
Raw
### Precompilation of Training and Validation Models

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Example setup for precompiling both training and validation models. Note that exported models lose their connections to other models, requiring manual weight synchronization if needed.

```python
model = ExampleModelWithLoss()

opts = poptorch.Options()

# Some dummy inputs.
input = torch.randn(10)
target = torch.randn(10)

# Wrap the model in our PopTorch annotation wrapper.
training_model = poptorch.trainingModel(model, opts)

```

--------------------------------

### Run Simple Adder Example

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/installation.html

This Python script demonstrates compiling a model to add two tensors using the IPU. It serves as a basic validation of your Poplar SDK setup.

```python
#!/usr/bin/env python3
# Copyright (c) 2020 Graphcore Ltd. All rights reserved.

import torch
import torch.nn as nn
import poptorch

# This simple example demonstrates compiling a model to add
# two tensors together using the IPU.


class SimpleAdder(nn.Module):
    def forward(self, x, y):
        return x + y


model = SimpleAdder()
inference_model = poptorch.inferenceModel(model)

t1 = torch.tensor([1.])
t2 = torch.tensor([2.])

assert inference_model(t1, t2) == 3.0
print("Success")

```

--------------------------------

### Install torchvision

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/example.html

Install the required version of torchvision for the MNIST example.

```bash
$ python3 -m pip install torchvision==0.11.1

```

--------------------------------

### Example of using deviceIterations

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

This example demonstrates how to set `deviceIterations` to 100, causing the IPU to execute the training loop 100 times per host call. It also shows how to configure a `DataLoader` and a `trainingModel` with these options.

```python
from functools import reduce
from operator import mul

import torch
import poptorch


class ExampleModelWithLoss(torch.nn.Module):
    def __init__(self, data_shape, num_classes):
        super().__init__()

        self.fc = torch.nn.Linear(reduce(mul, data_shape), num_classes)
        self.loss = torch.nn.CrossEntropyLoss()

    def forward(self, x, target=None):
        reshaped = x.reshape([x.shape[0], -1])
        fc = self.fc(reshaped)

        if target is not None:
            return fc, self.loss(fc, target)
        return fc


class ExampleDataset(torch.utils.data.Dataset):
    def __init__(self, shape, length):
        super().__init__()
        self._shape = shape
        self._length = length

        self._all_data = []
        self._all_labels = []

        torch.manual_seed(0)
        for _ in range(length):
            label = 1 if torch.rand(()) > 0.5 else 0
            data = torch.rand(self._shape) + label
            data[0] = -data[0]
            self._all_data.append(data)
            self._all_labels.append(label)

    def __len__(self):
        return self._length

    def __getitem__(self, index):
        return self._all_data[index], self._all_labels[index]


def device_iterations_example():
    # Set the number of samples for which activations/gradients are computed
    # in parallel on a single IPU
    model_batch_size = 2

    # Create a poptorch.Options instance to override default options
    opts = poptorch.Options()

    # Run a 100 iteration loop on the IPU, fetching a new batch each time
    opts.deviceIterations(100)

    # Set up the DataLoader to load that much data at each iteration
    training_data = poptorch.DataLoader(opts,
                                        dataset=ExampleDataset(shape=[3, 2],
                                                               length=10000),
                                        batch_size=model_batch_size,
                                        shuffle=True,
                                        drop_last=True)

    model = ExampleModelWithLoss(data_shape=[3, 2], num_classes=2)
    # Wrap the model in a PopTorch training wrapper
    poptorch_model = poptorch.trainingModel(model, options=opts)

    # Run over the training data, 100 batches at a time (specified in
    # opts.deviceIterations())
    for batch_number, (data, labels) in enumerate(training_data):
        # Execute the device with a 100 iteration loop of batchsize 2.
        # "output" and "loss" will be the respective output and loss of the
        # final batch (the default OutputMode).

        output, loss = poptorch_model(data, labels)
        print(f"{labels[-1]}, {output}, {loss}")

```

--------------------------------

### Create and Activate Virtual Environment and Install PopTorch

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/installation.html

Recommended method for installing PopTorch to isolate it from the system Python environment. This includes creating a virtual environment, activating it, updating pip, and installing the PopTorch wheel file.

```bash
$ virtualenv -p python3 poptorch_test
$ source poptorch_test/bin/activate
$ pip install -U pip
$ pip install <sdk_path>/poptorch_x.x.x.whl

```

--------------------------------

### Instantiate PyTorch Training Components

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/pytorch_to_poptorch.html

Standard PyTorch setup for model, training mode, and optimizer.

```python
model = ExampleModelWithLoss()
model.train()

optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)
```

--------------------------------

### Example config file for PopTorch options

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Use this format for a text file to set PopTorch options. Each line corresponds to an option in the Options class, omitting the 'options.' prefix.

```text
1deviceIterations(1)
2setExecutionStrategy(poptorch.ShardedExecution())
3replicationFactor(1)
4enableSyntheticData(True)
```

--------------------------------

### PyTorch DataLoader Example

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/pytorch_to_poptorch.html

Standard PyTorch DataLoader instantiation for training data.

```python
training_data = torch.utils.data.DataLoader(ExampleDataset(shape=[1],
                                                           length=20000),
                                            batch_size=10,
                                            shuffle=True,
                                            drop_last=True)
```

--------------------------------

### Load Options from Multiple Config Files

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Load options from configuration files. Options from later files override those from earlier files if they conflict. This example shows how deviceIterations is updated.

```python
options.loadFromFile('c1.cfg')
options.loadFromFile('c2.cfg')
```

--------------------------------

### PyTorch Optimizer Update Example

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/pytorch_to_poptorch.html

Example of conditionally updating a PyTorch optimizer within a training loop.

```python
    if momentum_loss is None:
        momentum_loss = loss
    else:
        momentum_loss = momentum_loss * 0.95 + loss * 0.05

    if momentum_loss < 0.1:
        optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
```

--------------------------------

### Configure PopTorch Options for Training with OutputMode.Sum

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

Configures PopTorch options for training, setting device iterations, gradient accumulation, and output mode to Sum for efficient metric calculation. This example demonstrates how to wrap a model and iterate over training data.

```python
opts = poptorch.Options()

opts.deviceIterations(5)
opts.Training.gradientAccumulation(10)
opts.outputMode(poptorch.OutputMode.Sum)

training_data = poptorch.DataLoader(opts,
                                    dataset=ExampleClassDataset(
                                        NUM_CLASSES, VEC_LENGTH, 2000),
                                    batch_size=5,
                                    shuffle=True,
                                    drop_last=True)


model = MulticlassPerceptron(VEC_LENGTH, NUM_CLASSES)
model.train()

# Wrap the model in a PopTorch training wrapper
poptorch_model = poptorch.trainingModel(model,
                                        options=opts,
                                        optimizer=torch.optim.Adam(
                                            model.parameters()))

# Run over the training data, 5 batches at a time.
for batch_number, (data, labels) in enumerate(training_data):
    # Execute the device with a 5 iteration loop of batchsize 5 with 10
    # gradient accumulations (global batchsize = 5 * 10 = 50). "loss" and
    # "accuracy" will be the sum across all device iterations and gradient
    # accumulations but not across the model batch size.
    _, accuracy = poptorch_model(data, labels)

    # Correct for iterations
    # Do not divide by batch here, as this is already accounted for in the
    # PyTorch Model.
    accuracy /= (opts.device_iterations * opts.Training.gradient_accumulation)
    print(f"Accuracy: {float(accuracy)*100:.2f}%")
```

--------------------------------

### Run PyTorch model inference on IPU using poptorch.inferenceModel

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Wraps a PyTorch model for inference on an IPU. This example demonstrates inference with a pre-trained MobileNetV2 and also shows how to cast model parameters to half-precision for inference.

```python
1import torch
2import torchvision
3import poptorch
4
5# Some dummy imagenet sized input.
6picture_of_a_cat_here = torch.randn([1, 3, 224, 224])
7
8# The model, in this case a MobileNet model with pretrained weights that comes
9# canned with PyTorch.
10model = torchvision.models.mobilenet_v2(pretrained=True)
11model.train(False)
12
13# Wrap in the PopTorch inference wrapper
14inference_model = poptorch.inferenceModel(model)
15
16# Execute on IPU.
17out_tensor = inference_model(picture_of_a_cat_here)
18
19# Get the top 5 ImageNet classes.
20top_five_classes = torch.topk(torch.softmax(out_tensor, 1), 5)
21print(top_five_classes)
22
23# Try the same on native PyTorch
24native_out = model(picture_of_a_cat_here)
25
26native_top_five_classes = torch.topk(torch.softmax(native_out, 1), 5)
27
28# Models should be very close to native output although some operations are
29# numerically different and floating point differences can accumulate.
30assert any(top_five_classes[1][0] == native_top_five_classes[1][0])
31# inference_half_start
32model = torch.nn.Linear(1, 10)
33
34# Cast the parameters (weights) to half.
35model.half()
36
37t1 = torch.tensor([1.]).half()
38
39opts = poptorch.Options()
40
41inference_model = poptorch.inferenceModel(model, opts)
42out = inference_model(t1)
43
44assert out.dtype == torch.half
45# inference_half_end
```

--------------------------------

### Configure Tensor Storage Location

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Defines where tensors are stored. This example shows how to configure activations to use off-chip storage.

```python
opts.TensorLocations.setActivationLocation(
    poptorch.TensorLocationSettings().useOnChipStorage(False))
```

--------------------------------

### Access Training Options

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Access options specific to model training. These options should not be set for inference models. Example shows how to set gradient accumulation.

```python
>>> opts = poptorch.Options()
>>> opts.Training.gradientAccumulation(4)

```

--------------------------------

### Gradient Accumulation Example in PopTorch

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

Demonstrates setting gradient accumulation to 8 for training. This means gradients from 8 micro-batches (batch size 2) are accumulated before a single model update, resulting in a global batch size of 16.

```python
 1def gradient_accumulation_example():
 2    # Set the number of samples for which activations/gradients are computed
 3    # in parallel on a single IPU
 4    model_batch_size = 2
 5    # Create a poptorch.Options instance to override default options
 6    opts = poptorch.Options()
 7
 8    # Run a 400 iteration loop on the IPU, fetching a new batch each time
 9    opts.deviceIterations(400)
10
11    # Accumulate the gradient 8 times before applying it.
12    opts.Training.gradientAccumulation(8)
13
14    training_data = poptorch.DataLoader(opts,
15                                        dataset=ExampleDataset(shape=[3, 2],
16                                                               length=100000),
17                                        batch_size=model_batch_size,
18                                        shuffle=True,
19                                        drop_last=True)
20
21    # Wrap the model in a PopTorch training wrapper
22    poptorch_model = poptorch.trainingModel(model, options=opts)
23
24    # Run over the training data, 400 batches at a time (specified in
25    # opts.deviceIterations())
26    for batch_number, (data, labels) in enumerate(training_data):
27        # Execute the device with a 100 iteration loop of model batchsize 2
28        # with gradient updates every 8 iterations (global batchsize = 2 * 8 = 16).
29        # "output" and "loss" will be the respective output and loss of the
30        # final batch of each replica (the default OutputMode).
31        output, loss = poptorch_model(data, labels)
32        print(f"{labels[-1]}, {output}, {loss}")
```

--------------------------------

### Configure SerialPhasedExecution Strategy

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Use `SerialPhasedExecution` to run phases sequentially on a single group of IPUs. This example demonstrates setting up three phases and assigning them to IPUs 0 and 1.

```python
strategy = poptorch.SerialPhasedExecution(
    poptorch.Phase(poptorch.Stage("A"), poptorch.Stage("A2")),
    poptorch.Phase(poptorch.Stage("B"), poptorch.Stage("B2")),
    poptorch.Phase(poptorch.Stage("C"), poptorch.Stage("C2")))

strategy.phase(0).ipus(0, 1)
strategy.phase(1).ipus(0, 1)
strategy.phase(2).ipus(0, 1)

opts.setExecutionStrategy(strategy)
```

--------------------------------

### Run Model with Half Precision (float16)

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Demonstrates how to run a model using half-precision floating-point format for increased memory efficiency. This example casts model parameters to `float16` and uses `float16` input tensors.

```python
model = torch.nn.Linear(1, 10)

# Cast the parameters (weights) to half.
model.half()

t1 = torch.tensor([1.]).half()

opts = poptorch.Options()

inference_model = poptorch.inferenceModel(model, opts)
out = inference_model(t1)

assert out.dtype == torch.half

```

--------------------------------

### Correct Persistent Tensor Handling with PyTorch Buffers in PopTorch

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

This example shows the correct method for creating a persistent tensor that increments on each iteration by registering it as a buffer. This ensures the tensor's value is maintained across runs on the IPU.

```python
class CounterModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer("i", torch.tensor([0.], dtype=torch.float))

    def forward(self):
        self.i += 1
        return self.i


model = CounterModel()
poptorch_model = poptorch.inferenceModel(model)

print(poptorch_model())  # tensor([1.])
print(poptorch_model())  # tensor([2.])

```

--------------------------------

### Update Pip

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/installation.html

Ensure pip is updated to the latest version to properly install PopTorch dependencies. Run this command before installing PopTorch.

```bash
$ pip install -U pip

```

--------------------------------

### Initialize Training Model with Options and Optimizer

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

Initializes the `poptorch.trainingModel` with the loss model, configured options, and an AdamW optimizer. This is the final step before running the training loop.

```python
training_model = poptorch.trainingModel(
    model_with_loss,
    model_opts,
    optimizer=optim.AdamW(model.parameters(), lr=opts.lr))
```

--------------------------------

### modelName()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Sets or gets the name of the model.

```APIDOC
## modelName()

### Description
Assigns a name to the model or retrieves the currently set name. This name can be used for identification in logs or profiling outputs.

### Method
`poptorch.Options.modelName(name: str)`

### Parameters
- **name** (str) - Optional - The name to assign to the model.
```

--------------------------------

### Load PopTorch options from a config file

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Instantiate PopTorch Options and load settings from a specified configuration file.

```python
1opts = poptorch.Options()
2opts.loadFromFile("tmp/poptorch.conf")
```

--------------------------------

### Enable PopVision Trace Instrumentation (PVTI)

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Set PVTI_OPTIONS to enable the PopVision Trace Instrumentation library, which is necessary for capturing pvti reports for the PopVision System Analyser.

```bash
export PVTI_OPTIONS='{"enable":"true"}'
```

--------------------------------

### PyTorch Custom Loss Function

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/pytorch_to_poptorch.html

Example of a custom loss function in PyTorch that combines multiple loss calculations.

```python
def custom_loss(output, target)
    loss1 = torch.nn.functional.nll_loss(x, target)
    loss2 = torch.nn.functional.nll_loss(x, target) * 5.0
    return loss1 + loss2
```

--------------------------------

### Instantiate PopTorch Training Model

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/pytorch_to_poptorch.html

Wrap your PyTorch model with PopTorch's training wrapper, providing options and the optimizer.

```python
model = ExampleModelWithLoss()
model.train()

optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)

# Wrap the model in a PopTorch training wrapper
poptorch_model = poptorch.trainingModel(model,
                                        options=opts,
                                        optimizer=optimizer)
```

--------------------------------

### Get IPU Hardware Version

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Retrieves the IPU hardware version of the system. An exception is raised if no hardware is detected.

```python
poptorch.ipuHardwareVersion()
```

--------------------------------

### PopTorch DataLoader with Options

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/pytorch_to_poptorch.html

PopTorch DataLoader instantiation, requiring PopTorch Options for IPU-specific configurations like deviceIterations.

```python
# Set up the PyTorch DataLoader to load that much data at each iteration
opts = poptorch.Options()
opts.deviceIterations(10)
training_data = poptorch.DataLoader(options=opts,
                                    dataset=ExampleDataset(shape=[1],
                                                           length=20000),
                                    batch_size=10,
                                    shuffle=True,
                                    drop_last=True)
```

--------------------------------

### Configure I/O Tiles and Execution Strategy

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/hostio_optimisation.html

Specify the number of I/O tiles and select an execution strategy to enable compute and I/O overlap. This impacts compute performance due to fewer tiles available for computation.

```python
opts.TensorLocations.numIOTiles(64)
opts.setExecutionStrategy(poptorch.ShardedExecution())
```

--------------------------------

### Retrieve Tensor Names in PopTorch

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/debugging.html

Call getTensorNames() on a compiled PopTorch model to get a list of all tensor names. The model must be compiled before this method can be used.

```python
input = torch.rand(10, 10)
label = torch.rand(10, 10)

model = Model()
poptorch_model = poptorch.trainingModel(model)
poptorch_model(input, label)

tensor_names = poptorch_model.getTensorNames()

```

--------------------------------

### poptorch.trainingModel

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Creates a PopTorch training model from a PyTorch model for IPU hardware in training mode. It wraps the original model to facilitate weight synchronization.

```APIDOC
## poptorch.trainingModel

### Description
Create a PopTorch training model, from a PyTorch model, to run on IPU hardware in training mode.
Note
PopTorch makes a shallow copy of the model and wraps the original model to facilitate weight synchronisation. Changes to the parameters in the returned training model affect the original model and vice versa. However, primitive variable types are not synced. For example calling `model.train()` on the original model, which changes the `training` bool of the model instance, will not alter the model returned by this function. You may need to call `model.train()` on your model before you call this function for correct behaviour.

### Parameters
* **model** (_Union_ _[__torch.nn.Module_ _,__poptorch.PoplarExecutor_ _]_) – The PyTorch model to wrap.
* **options** (_Optional_ _[__poptorch.Options_ _]_) – The IPU specific options
* **optimizer** (_Optional_ _[__torch.optim.Optimizer_ _]_) – 
The optimizers to apply during training.
Supported PyTorch optimizers: `optim.SGD`, `optim.Adam`, `optim.AdamW`, `optim.RMSprop`.
Supported PopTorch optimizers: `SGD`, `Adam`, `AdamW`, `RMSprop`. `LAMB`.

### Returns
The `PoplarExecutor` wrapper to use in place of `model`.

### Return type
poptorch.PoplarExecutor
```

--------------------------------

### loadFromFile()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Loads poptorch options from a file.

```APIDOC
## loadFromFile()

### Description
Loads poptorch.Options from a configuration file. This allows for persistent storage and easy reuse of complex option configurations.

### Method
`poptorch.Options.loadFromFile(filepath: str)`

### Parameters
- **filepath** (str) - Required - The path to the configuration file.
```

--------------------------------

### Options Class

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Main class for configuring poptorch behavior.

```APIDOC
## Options (class in poptorch)

### Description
The central class for configuring various aspects of poptorch, including compilation, execution, debugging, and distributed training.

### Usage
```python
options = poptorch.Options()
```
```

--------------------------------

### appendToLocationExcludes

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

When printing the IR all the frames containing one of the excluded strings will be ignored. This is helpful to get the IR to trace back to user code rather than some function inside a framework.

```APIDOC
## appendToLocationExcludes(_* excludes_)

When printing the IR all the frames containing one of the excluded strings will be ignored. This is helpful to get the IR to trace back to user code rather than some function inside a framework.

### Parameters

*   **excludes** (_str_) – Append these exclusions to the existing list of exclusions.

### Return type

poptorch.Options
```

--------------------------------

### Jit.traceModel

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Options related to PyTorch’s JIT compiler. Can be accessed via `poptorch.Options.Jit`.

```APIDOC
## Jit.traceModel(trace_model)

### Description
Options related to PyTorch’s JIT compiler.

### Parameters
#### Path Parameters
- **trace_model** (bool) - Required - 

### Return type
poptorch.options._JitOptions
```

--------------------------------

### Dynamic Update Slice Operation

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Enables dynamic update slicing in PopTorch applications. Requires specifying the input tensor, source tensor, dimension, start index, and size.

```python
poptorch.dynamic_update(input_tensor, src_tensor, dim=1, start=start_index_tensor, size=5)
```

--------------------------------

### Apply Custom Operation with PopART

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Applies a custom operation, implemented within PopART, to the inputs. Requires specifying inputs, name, domain, domain version, example outputs, and optional attributes.

```python
poptorch.custom_op(inputs=(x, y), name="MyCustomOp", domain="com.example", domain_version=1, example_outputs=(torch.zeros_like(x),), attributes={'scale': 2.0})
```

--------------------------------

### from_json()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Loads poptorch options from a JSON configuration.

```APIDOC
## from_json()

### Description
Initializes or updates poptorch.Options object by parsing configuration settings from a JSON string or file.

### Method
`poptorch.Options.from_json(json_config: str)`

### Parameters
- **json_config** (str) - Required - A JSON string containing the configuration options.

### Usage
```python
options = poptorch.Options.from_json('{"enable_profiling": true}')
```
```

--------------------------------

### Configure Model Replication Factor in PopTorch

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

Use replicationFactor to duplicate the model across multiple IPUs for automatic data parallelism. This example sets the replication factor to 4 and configures device iterations.

```python
 1def replication_factor_example():
 2    # Set the number of samples for which activations/gradients are computed
 3    # in parallel on a single IPU
 4    model_batch_size = 2
 5    # replication_start
 6    # Create a poptorch.Options instance to override default options
 7    opts = poptorch.Options()
 8
 9    # Run a 100 iteration loop on the IPU, fetching a new batch each time
10    opts.deviceIterations(100)
11
12    # Duplicate the model over 4 replicas.
13    opts.replicationFactor(4)
14
15    training_data = poptorch.DataLoader(opts,
16                                        dataset=ExampleDataset(shape=[3, 2],
17                                                               length=100000),
18                                        batch_size=model_batch_size,
19                                        shuffle=True,
20                                        drop_last=True)
21
22    model = ExampleModelWithLoss(data_shape=[3, 2], num_classes=2)
23    # Wrap the model in a PopTorch training wrapper
24    poptorch_model = poptorch.trainingModel(model, options=opts)
25
26    # Run over the training data, 100 batches at a time (specified in
27    # opts.deviceIterations())
28    for batch_number, (data, labels) in enumerate(training_data):
29        # Execute the device with a 100 iteration loop of model batchsize 2
30        # across 4 IPUs (global batchsize = 2 * 4 = 8). "output" and "loss"
31        # will be the respective output and loss of the final batch of each
32        # replica (the default OutputMode).
33        output, loss = poptorch_model(data, labels)
34        print(f"{labels[-1]}, {output}, {loss}")
```

--------------------------------

### Configure Serial Phased Execution

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

Sets up `poptorch.SerialPhasedExecution` where all stages run on the same IPU. This strategy is used when `opts.strategy` is 'phased'.

```python
strategy = poptorch.SerialPhasedExecution("B1", "B2", "B3", "B4")
strategy.stage("B1").ipu(0)
strategy.stage("B2").ipu(0)
strategy.stage("B3").ipu(0)
strategy.stage("B4").ipu(0)
model_opts.setExecutionStrategy(strategy)
```

--------------------------------

### execute()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Executes a model on the CPU.

```APIDOC
## execute()

### Description
Runs the model's operations on the CPU. This is typically used for debugging, testing, or when IPU execution is not available or desired.

### Method
`poptorch.CPU.execute(model, ...)`

### Parameters
- **model** (torch.nn.Module) - Required - The model to execute.

### Usage
```python
cpu_runner = poptorch.CPU()
output = cpu_runner.execute(model, input_data)
```
```

--------------------------------

### Dynamic Slice Operation

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Enables dynamic slicing in PopTorch applications, as native Torch dynamic slices cannot be properly intercepted by backends. Requires specifying the tensor, dimension, start index, size, and step.

```python
poptorch.dynamic_slice(tensor, dim=0, start=start_index_tensor, size=10, step=1)
```

--------------------------------

### Instantiate PopTorch Inference Model

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/pytorch_to_poptorch.html

Wrap your PyTorch model for inference using PopTorch's inference wrapper.

```python
poptorch_model = poptorch.inferenceModel(model)
```

--------------------------------

### Call Custom Op in PopTorch Inference Model

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Invoke a custom operation within a PopTorch inference model using `poptorch.custom_op`. Specify the operation name, vendor, version, and provide example outputs as a template.

```python
def test_inference():
    class BasicNetwork(nn.Module):
        def forward(self, x, bias):
            x, y = poptorch.custom_op([x, bias],
                                      "Cube",
                                      "com.acme",
                                      1,
                                      example_outputs=[x, x])
            return x, y
```

--------------------------------

### poptorch.ipuHardwareVersion

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Retrieves the IPU hardware version available in the system. An exception is raised if no hardware is found.

```APIDOC
## poptorch.ipuHardwareVersion

### Description
Indicates what IPU hardware version is available in the system.
Raise an exception if no hardware is available.

### Returns
- **int** - The IPU hardware version or -1 if unknown.
```

--------------------------------

### Load Precompiled Model and Run on Specific IPU

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Load a precompiled model and specify a particular IPU device to use at runtime using an `edit_opts_fn`. Only runtime options are applied when loading.

```python
def setIpuDevice(opts):
    opts.useIpuId(1)  # always use IPU 1


poptorch_model = poptorch.load(filename, edit_opts_fn=setIpuDevice)
poptorch_model(input, target)  # Run on IPU 1
```

--------------------------------

### Handling PopTorch Recoverable and Unrecoverable Errors

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Provides an example of how to catch and handle PopTorch's specific error types, including RecoverableError and UnrecoverableError. It demonstrates checking the recovery_action attribute for specific recovery steps.

```python
1    try:
2        m = PytorchModel(model_param)
3        inference_model = poptorch.inferenceModel(m)
4        t1 = torch.tensor([1.])
5        t2 = torch.tensor([2.])
6        assert inference_model(t1, t2) == 3.0
7    except poptorch.RecoverableError as e:
8        print(e)
9        if e.recovery_action == "FULL_RESET":
10            reboot_server()
11        elif e.recovery_action == "IPU_RESET":
12            print("Need to reset the IPU")
13        elif e.recovery_action == "PARITION_RESET":
14            print("Need to reset the partition")
15    except poptorch.UnrecoverableError as e:
16        print(f"Unrecoverable error: machine needs to be taken offline: {e}")
17        shutdown_system()
18    except poptorch.Error as e:
19        print(f"Received {e.message} from component {e.type}, "
20              f"location: {e.location}")
21        # Or you could just print all the information at once:
22        print(e)
23    except Exception as e:
24        print(e)

```

--------------------------------

### loadExecutable()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Loads a pre-compiled executable.

```APIDOC
## loadExecutable()

### Description
Loads a previously compiled Poplar executable from a file or memory. This bypasses the compilation step, allowing for faster model loading.

### Method
`poptorch.PoplarExecutor.loadExecutable(executable_path: str)`

### Parameters
- **executable_path** (str) - Required - The path to the compiled executable file.
```

--------------------------------

### ipuHardwareIsAvailable()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Checks if IPU hardware is available.

```APIDOC
## ipuHardwareIsAvailable()

### Description
Determines whether IPU hardware is detected and available for use by poptorch on the current system.

### Usage
```python
if poptorch.ipuHardwareIsAvailable():
    print('IPU hardware is available.')
```
```

--------------------------------

### loadFromFile

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Load options from a config file where each line in the file corresponds to a single option being set.

```APIDOC
## loadFromFile(_filepath_)

### Description
Load options from a config file where each line in the file corresponds to a single option being set. To set an option, simply specify how you would set the option within a Python script, but omit the `options.` prefix.
This method can be called multiple times on the same `Options` object. The options will not be reset to their defaults in between.

### Parameters
#### Path Parameters
- **filepath** (_str_) –

### Return type
poptorch.Options
```

--------------------------------

### Configure ParallelPhasedExecution Strategy

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Use `ParallelPhasedExecution` to execute phases in parallel, alternating between two groups of IPUs. This example sets up three phases, assigning even phases to IPUs 0 and 2, and odd phases to IPUs 1 and 3.

```python
strategy = poptorch.ParallelPhasedExecution(
    poptorch.Phase(poptorch.Stage("0"), poptorch.Stage("1")),
    poptorch.Phase(poptorch.Stage("2"), poptorch.Stage("3")),
    poptorch.Phase(poptorch.Stage("4"), poptorch.Stage("5")))

strategy.phase(0).ipus(0, 2)
strategy.phase(1).ipus(1, 3)
strategy.phase(2).ipus(0, 2)

opts.setExecutionStrategy(strategy)
```

--------------------------------

### Illustrating Data Loss with Iterable Datasets

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

This example demonstrates how naive division of data among workers can lead to significant data loss, especially with larger batch sizes and worker counts, when using PyTorch's default iterable dataset behavior.

```python
num_tensors = 100
num_workers = 7
batch_size = 4

per_worker_tensors = ceil(100 / num_workers) = 15
last_worker_tensors = 100 - (num_workers - 1) * per_worker_tensors = 10

num_tensors_used = batch_size * (floor(per_worker_tensors / batch_size) * (num_workers - 1) + floor(last_worker_tensors / batch_size))
                 = 80
```

--------------------------------

### Set PopTorch Options

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Instantiate and configure PopTorch options for model compilation and execution. Use `poptorch.Options()` to create an options object and then set specific parameters like `deviceIterations()` and training-specific options such as `gradientAccumulation()`.

```python
opts = poptorch.Options()
opts.deviceIterations(10)
opts.Training.gradientAccumulation(4)
```

--------------------------------

### poptorch.load()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Load a PopTorch model from a file previously created using compileAndExport().

```APIDOC
## poptorch.load()

### Description
Load a PopTorch model from a file previously created using `compileAndExport()`.

### Parameters
- **filename** (_str_) - Required - The path to the compiled executable file.
- **edit_opts_fn** (_Optional[Callable[[poptorch.Options], None]]_) - Optional - Function to edit the options before the model is restored. For example to attach to a specific IPU device.

### Returns
- **poptorch.PoplarExecutor**: The loaded PopTorch model executor.

### Example
```python
>>> model = poptorch.inferenceModel(model)
>>> model.compileAndExport("my_model.poptorch")
...
>>> model = poptorch.load("my_model.poptorch")
>>> model(my_input)
```
```

--------------------------------

### Set Host Overlap for Input Tensors

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Sets host overlap for input tensors to potentially increase performance by overlapping data copying from host to IPUs with computation. This requires IPU tiles to be set aside as IO tiles and should be used at the start of the model's forward method.

```python
poptorch.set_overlap_for_input(_input_tensors_ , _mode_)
```

--------------------------------

### Declare Poptorch Phases with Stages

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Declares Poptorch Phases, grouping Poptorch Stages for execution. Stages can be assigned to specific IPUs. This example shows two ways to define the 'second' phase: one grouping blocks 'B' and 'C' into a single stage, and another defining them as separate stages.

```python
first = poptorch.Phase(poptorch.Stage("A").ipu(0))
# Regrouped in a single stage
second = poptorch.Phase(poptorch.Stage("B", "C").ipu(1))
# 2 separate stages
second = poptorch.Phase(poptorch.Stage("B").ipu(1), poptorch.Stage("C").ipu(3))
```

--------------------------------

### Jit Property

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Accesses Just-In-Time (JIT) compilation options.

```APIDOC
## Jit (property in poptorch.Options)

### Description
Provides access to configuration settings related to Just-In-Time (JIT) compilation, which can affect how the model is optimized and compiled for the IPU.

### Usage
```python
options.Jit. ...
```
```

--------------------------------

### Inspect Optimizer State

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/debugging.html

Instantiate an optimizer, compile the training model, and then call state_dict() on the optimizer to retrieve its internal state. Note that PopTorch's optimizer state_dict may differ from PyTorch's.

```python
optim = poptorch.optim.SGD(model.parameters(), lr=0.01)
poptorch_model = poptorch.trainingModel(model, opts, optim)
poptorch_model(input, label)

state = optim.state_dict()

```

--------------------------------

### Define Execution Phase with Stages

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Shows how to define an execution Phase by explicitly specifying Stages, each potentially containing multiple blocks and assigned to specific IPUs.

```python
>>> p = Phase(poptorch.Stage("A").ipu(0))
>>> # 2 stages made of one block each
>>> p = Phase(poptorch.Stage("A").ipu(0), poptorch.Stage("B").ipu(1))
```

--------------------------------

### enableConvDithering()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Enables dithering for convolutional operations.

```APIDOC
## enableConvDithering()

### Description
Activates dithering, a technique to reduce quantization error, specifically for convolutional layers in the model.

### Method
`poptorch.MultiConv.enableConvDithering()`

### Usage
```python
multi_conv.enableConvDithering()
```
```

--------------------------------

### Enable Profiling with Debug Info

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Enable profiling report generation and include debug information. Specify the directory for output. Values from the environment variable POPLAR_ENGINE_OPTIONS take precedence.

```bash
POPLAR_ENGINE_OPTIONS={"autoReport.directory":"/profile/output",\
"autoReport.all":"true"}
```

```bash
POPLAR_ENGINE_OPTIONS={"autoReport.directory":"/profile/output",\
"autoReport.outputDebugInfo":"true"}
```

--------------------------------

### Precompile Model with Offline IPU Target

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Wrap a PyTorch model with PopTorch, configure offline IPU targeting, and compile and export it to a file. Ensure PopTorch versions match on both machines.

```python
import torch
import poptorch


class ExampleModelWithLoss(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = torch.nn.Linear(10, 10)
        self.loss = torch.nn.MSELoss()

    def forward(self, x, target=None):
        fc = self.fc(x)
        if self.training:
            return fc, self.loss(fc, target)
        return fc


torch.manual_seed(0)
model = ExampleModelWithLoss()

opts = poptorch.Options()
# You don't need a real IPU to compile the executable.
opts.useOfflineIpuTarget(ipu_target_version)

# Wrap the model in our PopTorch annotation wrapper.
poptorch_model = poptorch.trainingModel(model, opts)

# Some dummy inputs.
input = torch.randn(10)
target = torch.randn(10)

poptorch_model.compileAndExport(filename, input, target)
```

--------------------------------

### Load Precompiled Executable

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Load a precompiled executable and re-create the PopTorch wrapper on the target machine. This method is used when only the executable was exported.

```python
model = ExampleModelWithLoss()

opts = poptorch.Options()

# Wrap the model in our PopTorch annotation wrapper.
poptorch_model = poptorch.trainingModel(model, opts)
poptorch_model.loadExecutable(filename)

# Some dummy inputs.
input = torch.randn(10)
target = torch.randn(10)

poptorch_model(input, target)  # Run on IPU
```

--------------------------------

### Use PopTorch IPU Model

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Set POPTORCH_IPU_MODEL to 1 to use the IPU model instead of attaching to a physical IPU.

```bash
export POPTORCH_IPU_MODEL=1
```

--------------------------------

### Define Execution Phase with Blocks

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Illustrates creating an execution Phase using block IDs. Blocks within the same Phase are executed in parallel.

```python
>>> with poptorch.Block("A"):
...     layer()
>>> with poptorch.Block("B"):
...     layer()
>>> p = Phase("A","B") # One Stage made of 2 blocks
```

--------------------------------

### Conditional IPU/CPU Execution with isRunningOnIpu

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Shows how to use poptorch.isRunningOnIpu() to create code paths that execute differently on the IPU versus the CPU. This is useful for implementing custom operators or fallback logic.

```python
class Network(torch.nn.Module):
  def forward(self, x, y):
      if poptorch.isRunningOnIpu():
          # IPU path
          return my_custom_operator(x, y)
      else:
          # CPU path
          return my_torch_implementation(x,y)

```

--------------------------------

### useIpuModel

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Determines whether to use the IPU Model for simulation or physical IPU hardware. The IPU Model simulates IPU hardware behavior but has limitations.

```APIDOC
## useIpuModel(_use_model_)

### Description

Whether to use the IPU Model or physical hardware (default)
The IPU model simulates the behaviour of IPU hardware but does not offer all the functionality of an IPU. Please see the Poplar and PopLibs User Guide for further information.
This setting takes precedence over the `POPTORCH_IPU_MODEL` environment variable.

### Parameters

*   **use_model** (_bool_) – 
  * True: Use the IPU Model.
  * False: Use IPU hardware.

### Return type

poptorch.Options
```

--------------------------------

### ipuHardwareVersion()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Retrieves the version of the IPU hardware.

```APIDOC
## ipuHardwareVersion()

### Description
Returns the specific version or model identifier of the available IPU hardware.

### Usage
```python
version = poptorch.ipuHardwareVersion()
print(f'IPU Hardware Version: {version}')
```
```

--------------------------------

### poptorch.inferenceModel

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Creates a PopTorch inference model from a PyTorch model for IPU hardware in inference mode. It makes a shallow copy of the model.

```APIDOC
## poptorch.inferenceModel

### Description
Create a PopTorch inference model, from a PyTorch model, to run on IPU hardware in inference mode.
Note
PopTorch makes a shallow copy of the model. Changes to the parameters in the returned inference model affect the original model and vice versa. However, primitive variable types are not synced: for example calling `model.eval()` on the original model will not alter the model returned by this function. You may need to call `model.eval()` on your model before you call this function for correct behaviour.

### Parameters
* **model** (_Union_ _[__torch.nn.Module_ _,__poptorch.PoplarExecutor_ _]_) – The PyTorch model to wrap.
* **options** (_Optional_ _[__poptorch.Options_ _]_) – The IPU specific options

### Returns
The `PoplarExecutor` wrapper to use in place of `model`.

### Return type
poptorch.PoplarExecutor
```

--------------------------------

### load()

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/genindex.html

Loads a model or state.

```APIDOC
## load()

### Description
Loads a pre-trained model or its state dictionary. This function is typically used to initialize a model with learned weights.

### Usage
```python
poptorch.load(model, filepath)
```
```

--------------------------------

### Defining Network Blocks with PopTorch

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html

Illustrates defining a neural network with distinct blocks using `poptorch.Block`. This is useful for organizing layers and potentially applying different execution strategies to subsets of the model.

```python
 1class Network(nn.Module):
 2    def __init__(self):
 3        super(Network, self).__init__()
 4        self.layer1 = nn.Linear(784, 784)
 5        self.layer2 = nn.Linear(784, 784)
 6        self.layer3 = nn.Linear(784, 128)
 7        self.layer4 = nn.Linear(128, 10)
 8        self.softmax = nn.Softmax(1)
 9
10    def forward(self, x):
11        x = x.view(-1, 784)
12        with poptorch.Block("B1"):
13            x = self.layer1(x)
14        with poptorch.Block("B2"):
15            x = self.layer2(x)
16        with poptorch.Block("B3"):
17            x = self.layer3(x)
18        with poptorch.Block("B4"):
19            x = self.layer4(x)
20            x = self.softmax(x)
21        return x
22
23
24class TrainingModelWithLoss(torch.nn.Module):
25    def __init__(self, model):
26        super().__init__()
27        self.model = model
28        self.loss = torch.nn.CrossEntropyLoss()
29
30    def forward(self, args, loss_inputs=None):
31        output = self.model(args)
32        if loss_inputs is None:
33            return output
34        with poptorch.Block("B4"):
35            loss = self.loss(output, loss_inputs)
36        return output, loss
37
38
```

--------------------------------

### Combining Convolutions with PopTorch.MultiConv

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

The poptorch.MultiConv context manager combines multiple data-independent convolutions into a single operation for parallel execution, potentially increasing throughput. Use its methods to configure individual convolution parameters.

```python
with poptorch.MultiConv():
    y = self.convA(x)
    v = self.convB(u)
```

```python
multi_conv.availableMemoryProportions(value)
```

```python
multi_conv.cycleBackOff(value)
```

```python
multi_conv.enableConvDithering(value)
```

```python
multi_conv.partialsTypes(value)
```

```python
multi_conv.perConvReservedTiles(value)
```

```python
multi_conv.planType(value)
```

--------------------------------

### clone

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/reference.html

Create an unfrozen deep copy of the current options.

```APIDOC
## clone()

Create an unfrozen deep copy of the current options.

### Return type

poptorch.Options
```

--------------------------------

### Explicit Weight Copying with PoplarExecutor

Source: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/overview.html

Demonstrates when explicit weight copying to the device is necessary, such as after training on the CPU or altering wrapped model parameters. This ensures the IPU model has the latest weights.

```python
1model = Model()
2model.eval()
3
4poptorch_inf = poptorch.inferenceModel(model)
5
6# Switch for "poptorch.trainingModel": poptorch_inf will remain in "eval" mode
7model.train()
8poptorch_train = poptorch.trainingModel(model)
9
10# train on IPU
11train(poptorch_train)
12torch.save(model.state_dict(), "model.save")  # OK
13
14# Aready in "eval" mode
15validate(poptorch_inf)  # OK
16
17# switch to "eval" mode for CPU
18model.eval()
19validate(model)  # OK
20
21# train on CPU
22model.train()
23train_on_cpu(model)
24
25# Explicit copy needed
26poptorch_inf.copyWeightsToDevice()
27validate(poptorch_inf)

```
Tokens:
Raw