### Setup Python Environment for Docs
Source: https://github.com/apple/coremltools/blob/main/docs/README.md
Installs necessary dependencies and sets up the Python environment for building documentation. Ensure you are in your own fork of the repository.
```shell
scripts/build.sh --python=3.10
source scripts/env_activate.sh --python=3.10
pip reqs/docs.pip
pip install -r reqs/docs.pip
pip install -e .
```
--------------------------------
### InnerProductLayer Usage Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example of how to instantiate an InnerProductLayer in a default configuration.
```default
y = InnerProductLayer(x)
```
--------------------------------
### Install Core ML Tools and Dependencies
Source: https://github.com/apple/coremltools/blob/main/docs/source/convert-a-pytorch-segmentation-model.md
Install PyTorch, Torchvision, and coremltools using pip. Ensure these are installed before proceeding with model conversion.
```shell
pip install torch
pip install torchvision
pip install -U coremltools
```
--------------------------------
### Install PyTorch and Core ML Tools
Source: https://github.com/apple/coremltools/blob/main/docs/source/model-exporting.md
Install the necessary libraries for PyTorch model exporting and Core ML conversion.
```shell
pip install torch
pip install coremltools
```
--------------------------------
### BatchnormLayer Usage Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example of how to instantiate a BatchnormLayer in a default configuration.
```default
y = BatchnormLayer(x)
```
--------------------------------
### Run Example Script
Source: https://github.com/apple/coremltools/blob/main/docs/source/introductory-quickstart.md
Execute the example script using Python. The output will show the conversion process, the prediction result, and confirmation of model save and load.
```shell
python getting-started.py
```
--------------------------------
### EmbeddingLayer Usage Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example of how to instantiate an EmbeddingLayer in a default configuration.
```default
y = EmbeddingLayer(x)
```
--------------------------------
### Initialize NeuralNetworkBuilder for Classifier
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.models.neural_network.md
Example demonstrating the initialization of a NeuralNetworkBuilder for a classifier. This setup is for a neural network classifier with specific input dimensions.
```python
# Construct a builder that builds a neural network classifier with a 299 x 299 x 3
```
--------------------------------
### Install Core ML Tools and Dependencies
Source: https://github.com/apple/coremltools/blob/main/docs/source/convert-nlp-model.md
Install PyTorch, Transformers, and coremltools using pip. These are the essential libraries for this conversion process.
```shell
pip install torch
pip install transformers
pip install -U coremltools
```
--------------------------------
### Resize Bilinear Example Inputs
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.converters.mil.mil.ops.defs.md
Defines example input dimensions for demonstrating resizing behavior.
```python
Xin = 2
input_interval = [0,1]
```
--------------------------------
### Instantiate PoolingLayer
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example of how to instantiate a PoolingLayer. Requires 1 input and produces 1 output.
```default
y = PoolingLayer(x)
```
--------------------------------
### Import MIL Ops Definitions
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.converters.mil.md
Importing the ops definitions triggers the installation of all MIL ops into the Builder. This is a common setup step before defining operations.
```python
# Importing ops triggers installation of all ops into Builder.
>>> from .ops import defs as _ops
```
--------------------------------
### Create Development Installation Target
Source: https://github.com/apple/coremltools/blob/main/CMakeLists.txt
Sets up a development installation of the project using pip in editable mode. Depends on the 'coremlpython' target.
```cmake
add_custom_target(
pip_install_dev
COMMAND pip install -e ${PROJECT_SOURCE_DIR}
DEPENDS "coremlpython"
)
```
--------------------------------
### Install Core ML Tools Wheel
Source: https://github.com/apple/coremltools/blob/main/docs/source/installing-coremltools.md
Install a specific version of coremltools by providing the path to a downloaded Python wheel (.whl) file. This is useful for installing from source or specific builds.
```shell
pip install coremltools-4.0-cp38-none-macosx_10_12_intel.whl
```
--------------------------------
### EmbeddingNDLayer Usage Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example of how to instantiate an EmbeddingNDLayer in a default configuration.
```default
y = EmbeddingNDLayer(x)
```
--------------------------------
### CoreMLWeightMetaData Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/source/coremltools.optimize.coreml.utilities.md
Demonstrates how to create and print CoreMLWeightMetaData for a given numpy array.
```APIDOC
## CoreMLWeightMetaData Example
### Description
This example shows how to instantiate `CoreMLWeightMetaData` with a NumPy array and print its details, including shape, sparsity, and unique values.
### Method
Instantiate `CoreMLWeightMetaData`
### Endpoint
N/A (Python Class)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
import numpy as np
from coremltools.optimize.coreml import CoreMLWeightMetaData
data = np.array([[1.0, 0.0], [0.0, 6.0]], dtype=np.float32)
meta_data = CoreMLWeightMetaData(data)
print(meta_data)
```
### Response
#### Success Response (200)
Prints the metadata of the weight data.
#### Response Example
```default
[
val: np.ndarray(shape=(2, 2), dtype=float32)
sparsity: 0.5
unique_values: 3
]
```
```
--------------------------------
### ArgSortLayer Example (Ascending)
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example demonstrating the ArgSort layer with ascending order. The output provides the indices that would sort the input tensor.
```default
input shape = (5,)
axis = 0
input values = [3.1, 5.4, 32.9, 3.2, 77.0]
output shape = (5,)
output values = [0, 3, 1, 2, 4], descending = False
```
--------------------------------
### Install TensorFlow 1 Environment
Source: https://github.com/apple/coremltools/blob/main/docs/source/convert-a-tensorflow-1-image-classifier.md
Sets up a Miniconda environment for TensorFlow 1.15 and installs necessary packages.
```shell
conda create -n tensorflow1-env python=3.7
conda activate tensorflow1-env
conda install tensorflow==1.15
```
--------------------------------
### Install Core ML Tools and Dependencies
Source: https://github.com/apple/coremltools/blob/main/docs/source/convert-openelm.md
Install PyTorch, Transformers, and Core ML Tools using pip. Ensure you are using compatible versions.
```shell
pip install torch
pip install transformers
pip install coremltools
```
--------------------------------
### ConcatLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Illustrates the basic usage of ConcatLayer for concatenating multiple input blobs.
```default
y = ConcatLayer(x1,x2,....)
```
--------------------------------
### Sparse to Dense Conversion Example (iOS 16)
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.converters.mil.mil.ops.defs.md
Demonstrates byte packing for sparse to dense conversion with `shape = (5,)`, resulting in 1 byte. The example shows how elements are packed from MSB to LSB within the byte.
```python
shape = (5,) => M = 1 bytes
> MSB LSB
> :
> |
> ```
> ```
>
mask =
```
|
```
x x x 0 1 1 0 0 | <== packed elements
:
|--|
```
–
```
|--|i4|i3|i2|i1|i0|
```
<== tagged element ids
| byte 0 | <== tagged bytes
```
--------------------------------
### LRNLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Shows the basic syntax for applying LRNLayer to an input blob.
```default
y = LRNLayer(x)
```
--------------------------------
### Example of PIXEL_SHUFFLE Reorganization
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Demonstrates the output tensor shape and data arrangement for PIXEL_SHUFFLE mode with a blockSize of 2.
```default
[[[ 1 3 2 4]
[ 5 7 6 8]]
[[ 9 11 10 12]
[13 15 14 16]]]
```
--------------------------------
### Get Connected iPhone Devices
Source: https://github.com/apple/coremltools/blob/main/docs/source/mlmodel-debugging-perf-utilities.md
Retrieve a list of connected iPhone devices using the Device.get_connected_devices method. Ensure Xcode and Command Line Tools are installed and the device is connected and unlocked.
```python
from coremltools.models.ml_program.experimental.remote_device import (
AppSigningCredentials,
Device,
DeviceType,
)
# Get a list of connected iPhone devices
connected_devices = Device.get_connected_devices(device_type=DeviceType.IPHONE)
# This will display information about each connected iPhone, which may include device name, os version, and other relevant details
print(connected_devices)
```
--------------------------------
### Palettize Torch Model Weights (4-bit)
Source: https://github.com/apple/coremltools/blob/main/docs/source/opt-palettization-api.md
Applies 4-bit palettization to a Torch model with grouped channel granularity. This example requires a utility function to get the Torch model and specific palettization configuration classes.
```python
from model_utilities import get_torch_model
from coremltools.optimize.torch.palettization import PostTrainingPalettizer, \
PostTrainingPalettizerConfig
```
--------------------------------
### Work with MLModel Specification
Source: https://github.com/apple/coremltools/blob/main/docs/source/mlmodel.md
Provides examples for retrieving the model specification, printing its description, identifying the model type, and saving the spec. It also shows how to compile a spec into an MLModel and load a spec from a file.
```python
# Get the spec from the MLModel
spec = mlmodel.get_spec()
# Print the input/output description for the MLModel
print(spec.description)
# Get the type of MLModel (NeuralNetwork, SupportVectorRegressor, Pipeline etc)
print(spec.WhichOneof('Type'))
# Save out the MLModel directly from the spec
ct.models.utils.save_spec(spec, 'path/to/the/saved/model.mlmodel')
# Convert spec to MLModel. This step also compiles the model.
mlmodel = ct.models.MLModel(spec)
# Load the spec from the saved .mlmodel file directly
spec = ct.models.utils.load_spec('path/to/the/model.mlmodel')
```
--------------------------------
### Apply Joint Sparsity and Palettization to PyTorch Model
Source: https://github.com/apple/coremltools/blob/main/docs-guides/source/opt-joint-compression.md
This example shows how to apply magnitude pruning followed by data-free palettization to a PyTorch model. It includes model preparation, training iterations with pruning steps, finalization, palettization, and conversion to MLPackage. Ensure you have torchvision, torch, and coremltools installed.
```python
import torchvision
import torch
import coremltools as ct
from coremltools.optimize.torch.pruning import ModuleMagnitudePrunerConfig,
MagnitudePrunerConfig,
MagnitudePruner
from coremltools.optimize.torch.palettization import PostTrainingPalettizer,
PostTrainingPalettizerConfig,
ModulePostTrainingPalettizerConfig
# Apply pruning
# Initialize model and optimizer
# e.g. Resnet50
model = torchvision.models.resnet50(weights="IMAGENET1K_V2")
model.train()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
# Prepare model for pruning
prune_config = MagnitudePrunerConfig(
global_config=ModuleMagnitudePrunerConfig(
target_sparsity=0.8,
)
)
pruner = MagnitudePruner(model, prune_config)
pruned_model = pruner.prepare()
# run a couple of training iterations with random data
n_classes = 1000
batch_size = 5
for i in range(2):
inputs = torch.randn(batch_size, 3, 256, 256) # Batch of samples
targets = torch.randint(0, n_classes, (batch_size,)) # Target labels
logits = pruned_model(inputs)
out = torch.nn.LogSoftmax(dim=1)(logits)
loss = torch.nn.functional.nll_loss(out, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
pruner.step()
# finalize model
pruned_model = pruner.finalize(pruned_model, inplace=True)
# Apply palettization
palettization_config = PostTrainingPalettizerConfig(
global_config=ModulePostTrainingPalettizerConfig(
n_bits=4,
)
)
palettizer = PostTrainingPalettizer(pruned_model, palettization_config)
joint_compressed_model = palettizer.compress()
# convert the compressed model
joint_compressed_model.eval()
traced_model = torch.jit.trace(joint_compressed_model, torch.rand(1, 3, 256, 256))
mlmodel = ct.convert(traced_model,
inputs=[ct.TensorType(shape=(1, 3, 256, 256))],
minimum_deployment_target=ct.target.macOS15,
)
mlmodel.save("model_torch_pruned_and_palettized.mlpackage")
```
--------------------------------
### Initialize and Configure MultiFunctionDescriptor
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.models.md
Shows how to initialize a MultiFunctionDescriptor from an existing mlpackage or construct one from scratch by adding functions from different model paths.
```python
from coremltools.utils import MultiFunctionDescriptor, save_multifunction
# Initialize a MultiFunctionDescriptor instance with functions in an existing mlpackage.
# desc will contain all functions in "my_model.mlpackage"
desc = MultiFunctionDescriptor("my_model.mlpackage")
# Construct a MultiFunctionDescriptor instance from scratch.
# The below code inserts the "main" function from "my_model.mlpackage" as "main_1",
# and inserts the "main" function from "my_model_2.mlpackage" as "main_2".
desc = MultiFunctionDescriptor()
desc.add_function(
model_path="my_model.mlpackage",
source_function_name="main",
target_function_name="main_1",
)
desc.add_function(
model_path="my_model_2.mlpackage",
source_function_name="main",
target_function_name="main_2",
)
# Each MultiFunctionDescriptor instance must have a default function name
# so it can be saved as a multifunction mlpackage on disk.
desc.default_function_name = "main_1"
save_multifunction(desc, "my_multifunction_model.mlpackage")
```
--------------------------------
### Build Documentation Locally
Source: https://github.com/apple/coremltools/blob/main/docs/README.md
Cleans previous builds and generates HTML documentation locally. Use this to preview changes before committing.
```shell
make clean
make html
```
--------------------------------
### Install Pip in Conda Environment
Source: https://github.com/apple/coremltools/blob/main/docs/source/installing-coremltools.md
Install the pip package manager within your activated Conda environment. This is necessary for installing coremltools if not already present.
```shell
conda install pip
```
--------------------------------
### PaddingLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Demonstrates the application of PaddingLayer with different padding types (Constant, Reflection, Replication) on a sample input.
```default
y = PaddingLayer(x)
```
```default
H_out = H_in + topPaddingAmount + bottomPaddingAmount
W_out = W_in + leftPaddingAmount + rightPaddingAmount
topPaddingAmount == Height startEdgeSize == borderAmounts[0].startEdgeSize
bottomPaddingAmount == Height endEdgeSize == borderAmounts[0].endEdgeSize
leftPaddingAmount == Width startEdgeSize == borderAmounts[1].startEdgeSize
rightPaddingAmount == Width endEdgeSize == borderAmounts[1].endEdgeSize
```
```default
[1, 3, 4] : 1 2 3 4
5 6 7 8
9 10 11 12
```
```default
[1, 5, 6] : 0 0 0 0 0 0
0 0 0 0 0 0
0 0 1 2 3 4
0 0 5 6 7 8
0 0 9 10 11 12
```
```default
[1, 5, 6] : 11 10 9 10 11 12
7 6 5 6 7 8
3 2 1 2 3 4
7 6 5 6 7 8
11 10 9 10 11 12
```
```default
[1, 5, 6] : 1 1 1 2 3 4
1 1 1 2 3 4
1 1 1 2 3 4
5 5 5 6 7 8
9 9 9 10 11 12
```
--------------------------------
### FlattenLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example usage of the FlattenLayer. This layer flattens the input.
```default
y = FlattenLayer(x)
```
--------------------------------
### Load and Save MLModel
Source: https://github.com/apple/coremltools/blob/main/docs/source/mlmodel.md
Demonstrates how to load an existing MLModel from a file and save an MLModel to a file.
```python
# Load the MLModel
mlmodel = ct.models.MLModel('path/to/the/model.mlmodel')
# Save the MLModel
mlmodel.save('path/to/the/saved/model.mlmodel')
```
--------------------------------
### OpMagnitudePrunerConfig: Target Sparsity Example
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.optimize.coreml.pruning.md
Illustrates pruning by setting a target sparsity percentage. The lowest absolute weight values are zeroed out to achieve the desired sparsity.
```python
weight = [0.3, -0.2, -0.01, 0.05]
target_sparsity = 0.75
# sparsified weight would be [0.3, 0, 0, 0]
```
--------------------------------
### Construct Optimization Configuration for Weight Palettization
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.optimize.coreml.utilities.md
Create an OptimizationConfig object to define fine-grained compression strategies. This example shows how to set global, op-type, and op-name specific configurations for weight palettization using different modes and bit depths.
```python
from coremltools.optimize.coreml import OpPalettizerConfig, OptimizationConfig
# The default global configuration is 8 bits palettization with kmeans
global_config = OpPalettizerConfig(mode="kmeans", nbits=8)
# We use 2 bits palettization for convolution layers, and skip the compression for linear layers
op_type_configs = {
"conv": OpPalettizerConfig(mode="kmeans", nbits=2),
"linear": None,
}
# We want a convolution layer named "conv_1" to have a 4 bits palettization with a different mode
op_name_configs = {
"conv_1": OpPalettizerConfig(mode="uniform", nbits=4),
}
# Now we can put all configuration across three levels to construct an OptimizationConfig object
config = OptimizationConfig(
global_config=global_config,
op_type_configs=op_type_configs,
op_name_configs=op_name_configs,
)
```
--------------------------------
### Install TensorFlow 1 and Core ML Tools
Source: https://github.com/apple/coremltools/blob/main/docs/source/typed-execution-example.md
Installs TensorFlow 1.15 and coremltools within a conda environment. Pillow and matplotlib are also installed for image handling and visualization.
```shell
conda create -n tensorflow1-env python=3.7
conda activate tensorflow1-env
conda install tensorflow==1.15
```
```shell
pip install -U coremltools
pip install pillow
conda install matplotlib
```
--------------------------------
### Quantization Aware Training (QAT) Implementation
Source: https://github.com/apple/coremltools/blob/main/docs/source/opt-quantization-api.md
Prepare the model for QAT by initializing LinearQuantizer with a YAML configuration and inserting FakeQuantize layers. Integrate quantizer.step() into the training loop after optimizer.step() to update quantization statistics.
```python
# Initialize the quantizer
config = LinearQuantizerConfig.from_yaml("/path/to/yaml/config.yaml")
quantizer = LinearQuantizer(model, config)
# Prepare the model to insert FakeQuantize layers for QAT
example_input = torch.rand(1, 1, 20, 20)
model = quantizer.prepare(example_inputs=example_input, inplace=True)
# Use quantizer in your PyTorch training loop
for inputs, labels in data:
output = model(inputs)
loss = loss_fn(output, labels)
loss.backward()
optimizer.step()
quantizer.step()
```
--------------------------------
### Initialize MLModel from .mlmodel or .mlpackage
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.models.md
Construct an MLModel object from a model file. Supports both '.mlmodel' and '.mlpackage' formats.
```python
loaded_model = MLModel("my_model.mlmodel")
```
```python
loaded_model = MLModel("my_model.mlpackage")
```
--------------------------------
### LoadConstantLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example usage of the LoadConstantLayer. This layer loads data as a parameter and provides it as an output.
```default
y = LoadConstantLayer()
```
--------------------------------
### Resize Bilinear Grid Point Examples
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.converters.mil.mil.ops.defs.md
Shows calculated grid points for different output sizes and sampling modes, illustrating the impact of each mode.
```python
[0., 0.1, 0.5, 0.9, 1.] (Xout = 5, UNALIGN_CORNERS)
```
```python
[0., 0.25, 0.5, 0.75, 1.] (Xout = 5, "STRICT_ALIGN_CORNERS" / "ALIGN_CORNERS")
```
```python
[0., 0.4, 0.8, 1., 1.] (Xout = 5, "DEFAULT")
```
```python
[0.1, 0.3, 0.5, 0.7, 0.9] (Xout = 5, "OFFSET_CORNERS")
```
```python
[0., 0., 0.33, 0.67, 1., 1.] (Xout = 6, UNALIGN_CORNERS)
```
```python
[0., 0.2, 0.4, 0.6, 0.8, 1.] (Xout = 6, "STRICT_ALIGN_CORNERS" / "ALIGN_CORNERS")
```
```python
[0., 0.33, 0.67, 1., 1., 1.] (Xout = 6, "DEFAULT")
```
```python
[0.08, 0.25, 0.42, 0.58, 0.75, 0.92] (Xout = 6, "OFFSET_CORNERS")
```
--------------------------------
### Install deepspeech package
Source: https://github.com/apple/coremltools/blob/main/docs/source/convert-a-tensorflow-1-deepspeech-model.md
Install the deepspeech package using pip. This is a prerequisite for exporting the TensorFlow model.
```shell
pip install deepspeech
```
--------------------------------
### Install coremltools
Source: https://github.com/apple/coremltools/blob/main/README.md
Installs the latest stable version of coremltools using pip.
```shell
pip install coremltools
```
--------------------------------
### CropResizeLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example usage of the CropResizeLayer. This layer extracts cropped spatial patches or RoIs from the input and resizes them.
```default
y = CropResizeLayer(x)
```
--------------------------------
### Example Grid Points for Resize Bilinear
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.converters.mil.mil.ops.defs.md
Illustrates grid point calculations for ResizeBilinear with align_corners set to False and True, showing variations for different output sizes.
```python
Xin = 2
input_interval = [0,1]
```
```text
[0., 0.1, 0.5, 0.9, 1.] (Xout = 5, align_corners=False)
[0., 0.25, 0.5, 0.75, 1.] (Xout = 5, align_corners=True)
[0., 0., 0.33, 0.67, 1., 1.] (Xout = 6, align_corners=False)
[0., 0.2, 0.4, 0.6, 0.8, 1.] (Xout = 6, align_corners=True)
```
--------------------------------
### Instantiate and Trace a Model
Source: https://github.com/apple/coremltools/blob/main/docs/source/model-tracing.md
Instantiate the network and trace it using torch.jit.trace with an example input tensor to generate TorchScript. The input shape can be variable and specified later during conversion.
```python
model = SimpleNet() # Instantiate the network.
example = torch.rand(1, 3, 224, 224) # Example input, needed by jit tracer.
traced = torch.jit.trace(model, example) # Generate TorchScript by tracing.
```
--------------------------------
### ScaleLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example usage of the ScaleLayer. This layer performs elementwise multiplication by a scale factor and optionally adds a bias.
```default
y = ScaleLayer(x)
```
--------------------------------
### Quantization Aware Training (QAT) Configuration
Source: https://github.com/apple/coremltools/blob/main/docs/source/opt-quantization-api.md
Configure QAT using a YAML file to specify quantization parameters like scheme and milestones. This setup is used to initialize the LinearQuantizer for training.
```yaml
global_config:
quantization_scheme: symmetric
milestones:
- 0
- 100
- 400
- 200
module_name_configs:
first_layer: null
final_layer: null
```
--------------------------------
### BiasLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example usage of the BiasLayer. This layer performs elementwise addition of a bias, which is broadcasted to match the input shape.
```default
y = BiasLayer(x)
```
--------------------------------
### Import Necessary Libraries
Source: https://github.com/apple/coremltools/blob/main/docs/source/convert-a-pytorch-segmentation-model.md
Import PyTorch, Torchvision, PIL, and coremltools. This setup is required for loading models, handling images, and performing conversions.
```python
import urllib
import warnings
warnings.simplefilter(action="ignore", category=FutureWarning)
import torch
import torch.nn as nn
import torchvision
import json
from torchvision import transforms
from PIL import Image
import coremltools as ct
```
--------------------------------
### Install Core ML Tools and Dependencies
Source: https://github.com/apple/coremltools/blob/main/docs/source/convert-a-tensorflow-1-image-classifier.md
Installs coremltools, Pillow for image handling, and other required packages using pip and conda.
```shell
pip install -U coremltools
pip install pillow
conda install requests
conda install matplotlib
```
--------------------------------
### Build Core ML Tools from Source
Source: https://github.com/apple/coremltools/blob/main/docs/source/installing-coremltools.md
Execute the build script to create a 'build' folder with the coremltools distribution and a 'dist' folder with Python wheel files. Ensure CMake is installed.
```shell
zsh -i scripts/build.sh
```
--------------------------------
### Upgrade Pip
Source: https://github.com/apple/coremltools/blob/main/docs/source/installing-coremltools.md
Ensure you have the latest version of pip installed before creating a virtual environment. This command installs or upgrades pip for the current user.
```shell
python -m pip install --user --upgrade pip
```
--------------------------------
### Prepare Device for Model Debugging
Source: https://github.com/apple/coremltools/blob/main/docs/source/mlmodel-debugging-perf-utilities.md
Prepare a connected device for model debugging by specifying app signing credentials. This action installs the ModelRunner application on the device, which is necessary for executing Core ML models remotely. Ensure the bundle identifier matches your provisioning profile.
```python
connected_device = connected_devices[0]
# Define the app signing credentials
credentials = AppSigningCredentials(
development_team="", # Your Apple Developer Team ID
bundle_identifier="com.example.modelrunnerd", # Unique identifier for your app
provisioning_profile_uuid=None # UUID of provisioning profile (if applicable)
)
# Prepare the device for model debugging
# This installs the application on the device
prepared_device = await connected_device.prepare_for_model_debugging(credentials=credentials)
```
--------------------------------
### Post-Training Palettization Configuration
Source: https://github.com/apple/coremltools/blob/main/docs/source/opt-palettization-api.md
Configure post-training palettization with global settings. Use this for basic quantization without calibration data.
```python
torch_model = get_torch_model()
palettization_config_dict = {
"global_config": {"n_bits": 4, "granularity": "per_grouped_channel", "group_size": 4},
}
palettization_config = PostTrainingPalettizerConfig.from_dict(palettization_config_dict)
palettizer = PostTrainingPalettizer(torch_model, palettization_config)
palettized_torch_model = palettizer.compress()
```
--------------------------------
### Prepare and Test Image with Model
Source: https://github.com/apple/coremltools/blob/main/docs/source/image-inputs.md
Opens, resizes, and saves a sample image. Then, it uses the `predict()` method to process the image with the Core ML model and displays/saves the output image. Ensure the input dictionary key matches the model's input name.
```python
img = Image.open("palmtrees.jpg")
img = img.resize((256, 256), Image.ANTIALIAS)
display(img)
img.save("palmtrees_256_by_256.jpg")
```
```python
output = coreml_model.predict({"colorImage" : img})["colorOutput"]
display(output)
output.save("palmtrees_result.png")
```
--------------------------------
### GatherNDLayerParams Definition and Examples
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Defines parameters for a GatherND layer, which gathers elements using multi-indices. Includes formula for output shape and examples.
```default
'params' = input[0], 'indices' = input[1]
'indices' is a rank K+1 tensor of shape [I_0, I_1, .., I_(K-1), I_K]
which is viewed as a collection of indices of (I_0 * I_1 * ... * I_(K-1))
points in the I_K dimensional space. For instance, the multi-index of the first
point is indices[0,0,...,0,:].
The following shows how the output is constructed:
$$
for i = 0,1,...,(I_0-1)
...
for j = 0,1,....,(I_(K-1)-1)
output[i,....,j,:,:,..,:] = params[indices[i,...,j,:], :,:,..,:]
$$
Hence, output shape is [I_0, I_1,...,I(K-1)] + params.shape[I_K:].
```
```default
output.rank = indices.rank - 1 + params.rank - indices.shape[-1]
```
```default
input[0] shape = (4, 2, 3, 4)
input[1] shape = (6, 2)
output shape = (6,) + (3, 4) = (6, 3, 4)
input[0] shape = (3, 3, 3, 4, 7)
input[1] shape = (3, 5)
output shape = (3,) + () = (3,)
input[0] shape = (5, 3, 2, 5)
input[1] shape = (2, 7, 3, 2)
output shape = (2, 7, 3) + (2, 5) = (2, 7, 3, 2, 5)
```
```proto
message GatherNDLayerParams {
}
```
--------------------------------
### L2NormalizeLayer Example
Source: https://github.com/apple/coremltools/blob/main/mlmodel/docs/Format/NeuralNetwork.md
Example usage of the L2NormalizeLayer. This layer performs L2 normalization, dividing by the square root of the sum of squares of all elements of the input.
```default
y = L2NormalizeLayer(x)
```
--------------------------------
### OpMagnitudePrunerConfig: Block Sparsity Example
Source: https://github.com/apple/coremltools/blob/main/docs/source/coremltools.optimize.coreml.pruning.md
Demonstrates block sparsity where weights are pruned in structured blocks. This can improve performance on-device for linear and conv layers.
```python
# Given a 4 x 2 weight with the following value, and block_size = 2, dim = 0.
[
[1, 3],
[-6, -7],
[0, 3],
[-9, 2],
]
# We first flatten the matrix along axis = 0.
[1, -6, 0, -9, 3, -7, 3, 2]
# For block size 2, the L2 norm will be compute of first 2 elements, then the second and 3rd element and so on.
[6.08, 9.00, 7.62, 3.61]
# Then the smallest values will be picked to prune. So if target_sparsity = 0.5, then the blocks that will be
# pruned will be with ones with L2 norm value of 6.08 and 3.61. And hence, the elements in the first and third
# block are pruned. Resulting in the following flatten pruned tensor:
[0, 0, 0, -9, 3, -7, 0, 0]
# The final pruned tensor is:
[
[0, 3],
[0, -7],
[0, 0],
[-9, 0],
]
```