ONNX Runtime

https://onnxruntime.ai/docs/
Admin
ONNX Runtime is a cross-platform machine-learning model accelerator that optimizes inference and...
Tokens:1,438,314
Snippets:13,503
Trust Score:9.8
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Setup emsdk environment (non-Windows)

Source: https://onnxruntime.ai/docs/build/web.html

Installs and activates the emsdk environment. Skip on Windows.

```bash
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh

```

--------------------------------

### Install onnx from source

Source: https://onnxruntime.ai/docs/build/inferencing

Install onnx from source after installing protobuf. Sets ONNX_ML=1, builds a wheel, and installs it. Uninstall protobuf before building ONNX Runtime to avoid version conflicts.

```bash
  export ONNX_ML=1
  python3 setup.py bdist_wheel
  pip3 install --upgrade dist/*.whl
```

--------------------------------

### Install Python wheel from build output

Source: https://onnxruntime.ai/docs/performance/mobile-performance-tuning.html

Install the Python wheel from the build output directory on Windows. The path shown is an example; the actual path and package name depend on your build configuration and platform.

```bash
      pip install -U build\Windows\RelWithDebIfo\RelWithDebIfo\dist\onnxruntime_noopenmp-1.7.0-cp37-cp37m-win_amd64.whl

```

--------------------------------

### Install VCPKG prerequisites on Ubuntu

Source: https://onnxruntime.ai/docs/build/dependencies.md

Install the required packages on Ubuntu before using VCPKG. The command installs git, curl, zip, unzip, pkgconfig, and ninja-build.

```bash
apt-get install git curl zip unzip pkgconfig ninja-build
```

--------------------------------

### Install other dependencies

Source: https://onnxruntime.ai/docs/genai/tutorials/finetune.html

Installs the optimum and peft dependencies.

```bash
pip install optimum peft

```

--------------------------------

### Initialize CANN environment

Source: https://onnxruntime.ai/docs/build/eps.html

Initializes the CANN environment by sourcing the setup script. The default path is shown; change it if your installation is elsewhere.

```bash
# Default path, change it if needed.
source /usr/local/Ascend/ascend-toolkit/set_env.sh

```

--------------------------------

### Install onnxruntime-training-cpu for on-device training

Source: https://onnxruntime.ai/docs/install

Install the required dependencies and the onnxruntime-training-cpu package from the ORT package index. The first command installs general Python packages needed for training; the second installs the pre-built CPU training package.

```bash
python -m pip install cerberus flatbuffers h5py numpy>=1.16.6 onnx packaging protobuf sympy setuptools>=41.4.0
```

```bash
pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT/pypi/simple/ onnxruntime-training-cpu
```

--------------------------------

### Install and Import

Source: https://onnxruntime.ai/docs/genai/api/python.html

Install the onnxruntime-genai package and import it in Python.

```APIDOC
## Install and Import

### Install
```bash
pip install onnxruntime-genai
```

### Import
```python
import onnxruntime_genai
```
```

--------------------------------

### Install dependencies from requirements.txt

Source: https://onnxruntime.ai/docs/tutorials/iot-edge/rasp-pi-cv.html

Navigates into the cloned repository and installs the Python dependencies listed in requirements.txt.

```bash
cd onnxruntime-raspberrypi
pip install -r requirements.txt

```

--------------------------------

### Install Linux prerequisites for TVM EP

Source: https://onnxruntime.ai/docs/execution-providers/community-maintained/TVM-ExecutionProvider.html

Install the minimal pre-requisites on Ubuntu/Debian-like Linux systems. Requires gcc/g++ 8.2 or higher and nasm 2.14.02 or higher; on Ubuntu LTS 18, apt-get install nasm provides 2.13.02, so install from sources instead.

```bash
apt-get install -y python3 python3-dev python3-pip python3-setuptools gcc libtinfo-dev zlib1g-dev build-essential cmake libedit-dev libxml2-dev llvm-12
pip3 install numpy decorator attrs nasm

```

--------------------------------

### Example output for CUDA

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-v.html

Example output from the model when prompted with 'Describe the image'.

```text
The image shows a cup of coffee with a latte art design on top. The coffee is a light brown color,
and the art is white with a leaf-like pattern. The cup is white and has a handle on one side.</s>

```

--------------------------------

### Install onnxruntime-genai for CPU

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-v.html

Installs the ONNX Runtime generate() API for CPU.

```bash
pip install onnxruntime-genai

```

--------------------------------

### Install dependencies and build the project

Source: https://onnxruntime.ai/docs/tutorials/web/excel-addin-bert-js.html

Run these commands to install the required packages and build the project. This is the first step after cloning the project.

```bash
npm install
npm run build

```

--------------------------------

### Output of aarch64-linux-gnu-gcc -v

Source: https://onnxruntime.ai/docs/build/inferencing

Example output from running `aarch64-linux-gnu-gcc -v` on a cross-compilation setup. It shows the compiler's target, build configuration, and version, confirming the toolchain is correctly installed.

```text
 Using built-in specs.
 COLLECT_GCC=/usr/bin/aarch64-linux-gnu-gcc
 COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-linux-gnu/9/lto-wrapper
 Target: aarch64-linux-gnu
 Configured with: ../gcc-9.2.1-20190827/configure --bindir=/usr/bin --build=x86_64-redhat-linux-gnu --datadir=/usr/share --disable-decimal-float --disable-dependency-tracking --disable-gold --disable-libgcj --disable-libgomp --disable-libmpx --disable-libquadmath --disable-libssp --disable-libunwind-exceptions --disable-shared --disable-silent-rules --disable-sjlj-exceptions --disable-threads --with-ld=/usr/bin/aarch64-linux-gnu-ld --enable-__cxa_atexit --enable-checking=release --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++ --enable-linker-build-id --enable-lto --enable-nls --enable-obsolete --enable-plugin --enable-targets=all --exec-prefix=/usr --host=x86_64-redhat-linux-gnu --includedir=/usr/include --infodir=/usr/share/info --libexecdir=/usr/libexec --localstatedir=/var --mandir=/usr/share/man --prefix=/usr --program-prefix=aarch64-linux-gnu- --sbindir=/usr/sbin --sharedstatedir=/var/lib --sysconfdir=/etc --target=aarch64-linux-gnu --with-bugurl=https://bugzilla.redhat.com/bugzilla/ --with-gcc-major-version-only --with-isl --with-newlib --with-plugin-ld=/usr/bin/aarch64-linux-gnu-ld --with-sysroot=/usr/aarch64-linux-gnu/sys-root --with-system-libunwind --with-system-zlib --without-headers --enable-gnu-indirect-function --with-linker-hash-style=gnu
 Thread model: single
 gcc version 9.2.1 20190827 (Red Hat Cross 9.2.1-3) (GCC)

```

--------------------------------

### Compile the sample

Source: https://onnxruntime.ai/docs/tutorials/csharp/yolov3_object_detection_csharp.html

Command to build the C# sample project.

```bash
dotnet build

```

--------------------------------

### Run ONNX model and get a specific output by name

Source: https://onnxruntime.ai/docs/api/python/sources/tutorial.rst.txt

Similar to the previous example, but demonstrates how to retrieve a specific output by specifying its name in the list passed to sess.run. The code includes the same setup and inference steps.

```python
# hide: start
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
clr = LogisticRegression(max_iter=200, random_state=42)
clr.fit(X_train, y_train)
initial_type = [('float_input', FloatTensorType([None, 4]))]
onx = convert_sklearn(clr, initial_types=initial_type)
with open("logreg_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())

# Force GetPciBusId errors to be piped to null. When generating the logs
# This is because we get some output during initialization in the Sphinx generator
import os
_stderr_fd = os.dup(2)
_devnull = open(os.devnull, "w")
os.dup2(_devnull.fileno(), 2)
# hide: stop

import numpy
import onnxruntime as rt

# hide: start
os.dup2(_stderr_fd, 2)
os.close(_stderr_fd)
_devnull.close()
# hide: stop
```

--------------------------------

### OrtCustomOp::GetStartVersion

Source: https://onnxruntime.ai/docs/api/c/onnxruntime__c__api_8h_source.html

Gets the start version of the custom operator.

```APIDOC
## [C] OrtCustomOp::GetStartVersion

### Description
Gets the start version of the custom operator.

### Method
C function pointer

### Endpoint
OrtCustomOp::GetStartVersion

### Parameters
#### Function Pointer Signature
- **op** (const struct OrtCustomOp*) - Required - Pointer to the custom operator.

### Returns
- **int** - Start version.

### Example
```c
int start_version = op->GetStartVersion(op);
```
```

--------------------------------

### Run the sample

Source: https://onnxruntime.ai/docs/tutorials/csharp/yolov3_object_detection_csharp.html

Command to run the sample with model, image, and output image paths as arguments.

```bash
dotnet run [path-to-model] [path-to-image] [path-to-output-image]

```

--------------------------------

### OrtEpApi::ProfilingEvent_GetTimestampUs

Source: https://onnxruntime.ai/docs/api/c/onnxruntime__ep__c__api_8h_source.html

Get the start timestamp of a profiling event in microseconds.

```APIDOC
## OrtEpApi::ProfilingEvent_GetTimestampUs

### Description
Get the start timestamp of a profiling event in microseconds.

### Method
ProfilingEvent_GetTimestampUs

### Signature
OrtStatus * ProfilingEvent_GetTimestampUs(const OrtProfilingEvent *event, int64_t *out)

### Parameters
- **event** (const OrtProfilingEvent*) - Required - The profiling event.
- **out** (int64_t*) - Required - Pointer to the output timestamp.

### Returns
OrtStatus* - Status of the operation.
```

--------------------------------

### GetStartVersion

Source: https://onnxruntime.ai/docs/api/c/struct_ort_custom_op.html

Function pointer to get the start version of the custom operator.

```APIDOC
## GetStartVersion

### Description
Returns the start version of the custom operator.

### Signature
```c
int (*OrtCustomOp::GetStartVersion)(const struct OrtCustomOp *op);
```

### Parameters
- **op** (const struct OrtCustomOp *) - Pointer to the custom operator.

### Returns
- **int** - The start version.
```

--------------------------------

### Set up Arm NN build environment

Source: https://onnxruntime.ai/docs/build/eps.html

Set up the build environment for Arm NN on i.MX8QM. Requires installing the i.MX8QM BSP first.

```bash
source /opt/fsl-imx-xwayland/4.*/environment-setup-aarch64-poky-linux
alias cmake="/usr/bin/cmake -DCMAKE_TOOLCHAIN_FILE=$OECORE_NATIVE_SYSROOT/usr/share/cmake/OEToolchainConfig.cmake"

```

--------------------------------

### Example run command

Source: https://onnxruntime.ai/docs/tutorials/csharp/fasterrcnn_csharp.html

Example invocation using the FasterRCNN-10.onnx model on demo.jpg, saving the result to out.jpg.

```bash
dotnet run ~/Downloads/FasterRCNN-10.onnx ~/Downloads/demo.jpg ~/Downloads/out.jpg
```

--------------------------------

### Install nightly onnxruntime-qnn package

Source: https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider

Install the nightly build of the ONNX Runtime QNN Python package from the ORT-Nightly feed. Use this to get the latest features and fixes.

```bash
python -m pip install --pre --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple onnxruntime-qnn
```

--------------------------------

### Cross-compile for Android on Windows (example paths)

Source: https://onnxruntime.ai/docs/build/android.html

Example invocation of the Windows build command using placeholder SDK/NDK paths. The ABI is set to arm64-v8a and API level 27.

```bash
./build.bat --android --android_sdk_path .../Android --android_ndk_path .../Android/ndk/<NDK version> --android_abi arm64-v8a --android_api 27 --cmake_generator Ninja

```

--------------------------------

### Tokenized output tensor example

Source: https://onnxruntime.ai/docs/tutorials/csharp/stable-diffusion-csharp.html

Example output tensor from the CLIP tokenizer for the sample prompt. Shows the token IDs (starting with 49406) and padding with 49407 to reach the 77-token max length.

```text
tensor([[49406,  1078,   320,  1674,   539,  1901,  2677,   593,  4023,  1630,
           585,   537,   320,   736,  2390, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
         49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407])

```

--------------------------------

### Initialize OpenVINO Environment on Linux

Source: https://onnxruntime.ai/docs/build/eps.html

Initialize the OpenVINO environment on Linux. Required before building.

```bash
 $ source <openvino_install_directory>/setupvars.sh
```

--------------------------------

### Clone the ONNX Runtime inference examples repository

Source: https://onnxruntime.ai/docs/tutorials/mobile/superres.html

Clones the onnxruntime-inference-examples repo and changes into the iOS super resolution example directory. Run these commands in a terminal to get the sample app source code.

```bash
git clone https://github.com/microsoft/onnxruntime-inference-examples
cd onnxruntime-inference-examples/mobile/examples/super_resolution/ios

```

--------------------------------

### Install OpenVINO and dependencies

Source: https://onnxruntime.ai/docs/tutorials/accelerate-pytorch/resnet-inferencing.html

Install OpenVINO, ONNX, and onnxconverter_common, then add OpenVINO libraries to the path using openvino.utils.

```python
import sys

if sys.platform in ['linux', 'win32']: # Linux or Windows
    !{sys.executable} -m pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
else: # Mac
    print("PyTorch 1.9 MacOS Binaries do not support CUDA, install from source instead")

!{sys.executable} -m pip install onnxruntime-openvino onnx onnxconverter_common==1.8.1 pillow

import openvino.utils as utils
utils.add_openvino_libs_to_path()
```

--------------------------------

### Install ONNX Runtime training wheel

Source: https://onnxruntime.ai/docs/build/training.html

Installs the built ONNX Runtime training wheel using pip.

```bash
python -m pip install build/Linux/RelWithDebInfo/dist/*.whl
```

--------------------------------

### Create GPU environment with Conda

Source: https://onnxruntime.ai/docs/tutorials/accelerate-pytorch/resnet-inferencing.html

Commands to create and activate a Conda environment named 'gpu_env_demo' with Python 3.8, install ipykernel and ipywidgets, register the kernel, and start Jupyter Notebook.

```bash
conda create -n gpu_env_demo python=3.8
conda activate gpu_env_demo 
conda install -c anaconda ipykernel
conda install -c conda-forge ipywidgets
python -m ipykernel install --user --name=gpu_env_demo 
jupyter notebook
```

--------------------------------

### QDQ model optimizer help

Source: https://onnxruntime.ai/docs/tutorials/mobile/helpers

Shows the command-line help for the QDQ model optimizer. It updates a QDQ format ONNX model for optimal performance with ONNX Runtime. If no optimizations are applied, the output model will be identical to the input and can be discarded.

```bash
python -m onnxruntime.tools.qdq_helpers.optimize_qdq_model --help
usage: optimize_qdq_model.py [-h] input_model output_model

Update a QDQ format ONNX model to ensure optimal performance when executed using ONNX Runtime.

positional arguments:
  input_model   Provide path to ONNX model to update.
  output_model  Provide path to write updated ONNX model to.

optional arguments:
  -h, --help    show this help message and exit

```

--------------------------------

### Initialize CocoaPods Podfile

Source: https://onnxruntime.ai/docs/tutorials/on-device-training/ios-app.html

Run this command in the project directory to create a Podfile for CocoaPods.

```bash
pod init

```

--------------------------------

### Create CPU environment with Conda

Source: https://onnxruntime.ai/docs/tutorials/accelerate-pytorch/resnet-inferencing.html

Commands to create and activate a Conda environment named 'cpu_env_demo' with Python 3.8, install ipykernel and ipywidgets, register the kernel, and start Jupyter Notebook.

```bash
conda create -n cpu_env_demo python=3.8
conda activate cpu_env_demo
conda install -c anaconda ipykernel
conda install -c conda-forge ipywidgets
python -m ipykernel install --user --name=cpu_env_demo
jupyter notebook
```

--------------------------------

### Example output for CPU

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-v.html

Example output from the model when prompted with 'Convert this image to markdown format'.

```text
| Product             | Qtr 1      | Qtr 2      | Grand Total |
|---------------------|------------|------------|-------------|
| Chocolade           | $744.60    | $162.56    | $907.16     |
| Gummibarchen        | $5,079.60  | $1,249.20  | $6,328.80   |
| Scottish Longbreads | $1,267.50  | $1,062.50  | $2,330.00   |
| Sir Rodney's Scones | $1,418.00  | $756.00    | $2,174.00   |
| Tarte au sucre      | $4,728.00  | $4,547.92  | $9,275.92   |
| Chocolate Biscuits  | $943.89    | $349.60    | $1,293.49   |
| Total               | $14,181.59 | $8,127.78  | $22,309.37  |

The table lists various products along with their sales figures for Qtr 1, Qtr 2, and the Grand Total.
The products include Chocolade, Gummibarchen, Scottish Longbreads, Sir Rodney's Scones, Tarte au sucre,
and Chocolate Biscuits. The Grand Total column sums up the sales for each product across the two quarters.</s>

```

--------------------------------

### Load and run the ONNX model with ONNX Runtime to get predictions

Source: https://onnxruntime.ai/docs/api/python/sources/tutorial.rst.txt

Loads the saved ONNX model using rt.InferenceSession with the CPUExecutionProvider, runs inference on the test data, and prints the predictions. The code includes setup steps to suppress stderr output during Sphinx generation.

```python
# hide: start
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
clr = LogisticRegression(max_iter=200, random_state=42)
clr.fit(X_train, y_train)
initial_type = [('float_input', FloatTensorType([None, 4]))]
onx = convert_sklearn(clr, initial_types=initial_type)
with open("logreg_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())

# Force GetPciBusId errors to be piped to null. When generating the logs
# This is because we get some output during initialization in the Sphinx generator
import os
_stderr_fd = os.dup(2)
_devnull = open(os.devnull, "w")
os.dup2(_devnull.fileno(), 2)
# hide: stop

import numpy
import onnxruntime as rt

# hide: start
os.dup2(_stderr_fd, 2)
os.close(_stderr_fd)
_devnull.close()
# hide: stop

sess = rt.InferenceSession("logreg_iris.onnx", providers=["CPUExecutionProvider"])
input_name = sess.get_inputs()[0].name
pred_onx = sess.run(None, {input_name: X_test.astype(numpy.float32)})[0]
print(pred_onx)
```

--------------------------------

### Python: Run inference with CUDA and CPU providers

Source: https://onnxruntime.ai/docs/execution-providers

Complete example of using the Python API to run inference with ONNX Runtime, setting provider priority to CUDA then CPU, and processing the output image. Demonstrates how to get input/output names, run inference, and update provider priority.

```python
import onnxruntime as rt

#define the priority order for the execution providers
# prefer CUDA Execution Provider over CPU Execution Provider
EP_list = ['CUDAExecutionProvider', 'CPUExecutionProvider']

# initialize the model.onnx
sess = rt.InferenceSession("model.onnx", providers=EP_list)

# get the outputs metadata as a list of :class:`onnxruntime.NodeArg`
output_name = sess.get_outputs()[0].name

# get the inputs metadata as a list of :class:`onnxruntime.NodeArg`
input_name = sess.get_inputs()[0].name

# inference run using image_data as the input to the model 
detections = sess.run([output_name], {input_name: image_data})[0]

print("Output shape:", detections.shape)

# Process the image to mark the inference points 
image = post.image_postprocess(original_image, input_size, detections)
image = Image.fromarray(image)
image.save("kite-with-objects.jpg")

# Update EP priority to only CPUExecutionProvider
sess.set_providers(['CPUExecutionProvider'])

cpu_detection = sess.run(...)

```

--------------------------------

### Run Phi-3 model with phi3-qa.py

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-python.html

Download the phi3-qa.py example script and run it with the DirectML model. The script loads the model and then prompts for input in a loop, streaming output as it is generated.

```bash
curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py -o phi3-qa.py
python phi3-qa.py -m directml\directml-int4-awq-block-128 -e dml
```

--------------------------------

### Install CocoaPods

Source: https://onnxruntime.ai/docs/tutorials/mobile/deploy-ios.html

Installs CocoaPods, the dependency manager used to install the onnxruntime-objc pod for the iOS app.

```bash
sudo gem install cocoapods

```

--------------------------------

### CreateInstanceWithOptions(ref EnvironmentCreationOptions)

Source: https://onnxruntime.ai/docs/api/csharp/api/Microsoft.ML.OnnxRuntime.OrtEnv.html

Provides a way to create an instance with options. It throws if the instance already exists and the specified options not have effect.

```APIDOC
## CreateInstanceWithOptions(ref EnvironmentCreationOptions)

### Description
Provides a way to create an instance with options. It throws if the instance already exists and the specified options not have effect.

### Method
CreateInstanceWithOptions

### Declaration
public static OrtEnv CreateInstanceWithOptions(ref EnvironmentCreationOptions options)

### Parameters
- **options** (EnvironmentCreationOptions) - Required - The environment creation options.

### Returns
- **OrtEnv** - The created OrtEnv instance.

### Exceptions
- **OnnxRuntimeException** - if the singleton has already been created
```

--------------------------------

### Install CMake on Linux via pip

Source: https://onnxruntime.ai/docs/build/inferencing

Installs CMake via pip on Linux and verifies the installation path.

```bash
python3 -m pip install cmake
which cmake
```

--------------------------------

### Install onnxruntime-extensions with pip

Source: https://onnxruntime.ai/docs/extensions

Installs the ONNX Runtime Extensions package for Python. This is the standard installation method for the stable release.

```bash
pip install onnxruntime-extensions

```

--------------------------------

### Initialize OpenVINO Environment on Windows

Source: https://onnxruntime.ai/docs/build/eps.html

Initialize the OpenVINO environment on Windows. Required before building.

```bat
 C:\<openvino_install_directory>\setupvars.bat
```

--------------------------------

### Install Python wheel

Source: https://onnxruntime.ai/docs/genai/howto/build-from-source.html

Installs the Python wheel. Change to the directory containing the wheel file before running pip install.

```bash
# Change dir to the folder containing the onnxruntime-genai wheel
# Example for Linux: cd build/Linux/Release/wheel/
pip install *.whl

```

--------------------------------

### Enable QNN profiling with basic level (Python)

Source: https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider

Sets the QNNExecutionProvider options to enable basic profiling and writes profiling data to output.csv. Requires setting profiling_level and profiling_file_path. Generates output.csv and optionally output_qnn.log with QAIRT 2.39+.

```python
# Python on Windows on Snapdragon device
import onnxruntime as ort
import numpy as np

provider_options = [
    "backend_path": "path/to/QnnHtp.dll", # Use libQnnHtp.so if on Linux
    "htp_performance_mode": "burst",
    "device_id": "0",
    "htp_graph_finalization_optimization_mode":"3",
    "soc_model": "60",
    "htp_arch": "73",
    "vtcm_mv": "8",
    "profiling_level": "basic",
    "profiling_file_path": "output.csv"
]

sess_options = ort.SessionOptions()

session = ort.InferenceSession(
    "model.onnx",
    sess_options=sess_options,
    providers=["QNNExecutionProvider"],
    provider_options=provider_options
)

input0 = np.ones((1,2,3,4), dtype=np.float32)
result = session.run(None, {"input": input0})

```

--------------------------------

### Install torch-ort and configure onnxruntime-training

Source: https://onnxruntime.ai/docs/get-started/training-pytorch.html

Install the torch-ort and onnxruntime-training Python packages and configure onnxruntime-training to work with the user's PyTorch installation. Note that this installs the default versions mapped to specific CUDA library versions.

```bash
# Installs the torch_ort and onnxruntime-training Python packages
pip install torch-ort
# Configures onnxruntime-training to work with user's PyTorch installation
python -m torch_ort.configure
```

--------------------------------

### Example dotnet run command

Source: https://onnxruntime.ai/docs/tutorials/csharp/resnet50_csharp.html

Example invocation using a ResNet50 ONNX model and a dog image.

```bash
dotnet run ~/Downloads/resnet50-v2-7.onnx ~/Downloads/dog.jpeg

```

--------------------------------

### Example CPU model input and output

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-python.html

Example interaction with phi3-qa.py on CPU, showing a prompt about generative AI and the streamed response.

```text
Input: Tell me a joke about generative AI

Output:  Why did the generative AI go to school?

To improve its "creativity" algorithm!

```

--------------------------------

### Download and run Phi-3.5 mini 128k context CPU

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-python.html

Downloads the Phi-3.5-mini-instruct-onnx CPU model using huggingface-cli and runs the QA script with the CPU execution provider.

```bash
huggingface-cli download microsoft/Phi-3.5-mini-instruct-onnx --include cpu_and_mobile/cpu-int4-awq-block-128-acc-level-4/* --local-dir .
python phi3-qa.py -m cpu_and_mobile/cpu-int4-awq-block-128-acc-level-4 -e cpu

```

--------------------------------

### Verify CUDA installation with nvcc --version

Source: https://onnxruntime.ai/docs/tutorials/csharp/csharp-gpu.html

Run this command in a terminal to verify the CUDA toolkit installation. It should output the installed CUDA version.

```bash
nvcc --version

```

--------------------------------

### Install onnxruntime-genai-cuda

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-v.html

Installs the ONNX Runtime generate() API for CUDA. This package uses CUDA 12; for CUDA 11, build and install from source.

```bash
pip install onnxruntime-genai-cuda

```

--------------------------------

### Install and login to Hugging Face CLI

Source: https://onnxruntime.ai/docs/genai/tutorials/phi2-python.html

Install the Hugging Face Hub CLI and log in to Hugging Face. Required before building the phi-2 model with the -m option.

```bash
pip install huggingface-hub
huggingface-cli login

```

--------------------------------

### Install ONNX Runtime GenAI for CPU

Source: https://onnxruntime.ai/docs/genai/tutorials/deepseek-python.html

Creates a virtual environment and installs ONNX Runtime GenAI, Olive, and dependencies for CPU inference.

```bash
python -m venv .venv && source .venv/bin/activate
pip install requests numpy --pre onnxruntime-genai olive-ai

```

--------------------------------

### Install onnxruntime-qnn Python package

Source: https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider

Install the ONNX Runtime QNN Python package using pip. This is the standard installation command for the QNN Execution Provider.

```bash
pip install onnxruntime-qnn
```

--------------------------------

### Open audio files from paths and bytes

Source: https://onnxruntime.ai/docs/genai/api/python.html

Example of opening an audio file from a path and from bytes read from a file.

```python
audios = onnxruntime_genai.Audios.open("audio1.wav")
with open("audio1.wav", "rb") as f:
    audios_bytes = onnxruntime_genai.Audios.open_bytes(f.read())
```

--------------------------------

### Download and run Phi-3 mini 4k context CPU

Source: https://onnxruntime.ai/docs/genai/tutorials/phi3-python.html

Downloads the Phi-3-mini-4k-instruct-onnx CPU model using huggingface-cli and runs the QA script with the CPU execution provider.

```bash
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir .
python phi3-qa.py -m cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 -e cpu

```

--------------------------------

### Verify CMake installation

Source: https://onnxruntime.ai/docs/build/inferencing

Verifies that CMake is installed and accessible.

```bash
cmake --version
```

--------------------------------

### Python: Using load_config for CPU

Source: https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html

Configures an OpenVINO Execution Provider session for the CPU device with latency performance hint and a single stream. Use this to optimize CPU inference with explicit configuration.

```python
import onnxruntime as ort
import json

# Create CPU config
config = {
    "CPU": {
        "PERFORMANCE_HINT": "LATENCY",
        "NUM_STREAMS": "1"
    }
}
options = {"device_type": "CPU", "load_config": json.dumps(config)}
session = ort.InferenceSession("model.onnx", 
                                providers=[("OpenVINOExecutionProvider", options)])

```
Tokens:
Raw