ONNX Runtime

https://onnxruntime.ai/docs/api/python/
Admin
ONNX Runtime is a performance-focused inference and training engine for Open Neural Network Exchange...
Tokens:56,105
Snippets:504
Trust Score:9.8
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Run ONNX model and get a specific output by name

Source: https://onnxruntime.ai/docs/api/python/sources/tutorial.rst.txt

Similar to the previous example, but demonstrates how to retrieve a specific output by specifying its name in the list passed to sess.run. The code includes the same setup and inference steps.

```python
# hide: start
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
clr = LogisticRegression(max_iter=200, random_state=42)
clr.fit(X_train, y_train)
initial_type = [('float_input', FloatTensorType([None, 4]))]
onx = convert_sklearn(clr, initial_types=initial_type)
with open("logreg_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())

# Force GetPciBusId errors to be piped to null. When generating the logs
# This is because we get some output during initialization in the Sphinx generator
import os
_stderr_fd = os.dup(2)
_devnull = open(os.devnull, "w")
os.dup2(_devnull.fileno(), 2)
# hide: stop

import numpy
import onnxruntime as rt

# hide: start
os.dup2(_stderr_fd, 2)
os.close(_stderr_fd)
_devnull.close()
# hide: stop
```

--------------------------------

### onnxruntime.get_build_info

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Returns build information for the ONNX Runtime installation.

```APIDOC
## onnxruntime.get_build_info

### Description
Returns build information for the ONNX Runtime installation.

### Signature
get_build_info() -> str
```

--------------------------------

### get_profiling_start_time_ns

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Returns the nanoseconds of profiling start time. On some platforms, this timer may not be as precise as nanoseconds (e.g., Windows and MacOS have ~100ns precision).

```APIDOC
## get_profiling_start_time_ns

### Description
Returns the nanoseconds of profiling start time. On some platforms, this timer may not be as precise as nanoseconds (e.g., Windows and MacOS have ~100ns precision).

### Method
`get_profiling_start_time_ns()`

### Parameters
None

### Returns
- **int** - Nanoseconds of profiling start time.
```

--------------------------------

### Output of ONNX Runtime predictions

Source: https://onnxruntime.ai/docs/api/python/tutorial.html

Example output from running the ONNX model with ONNX Runtime. Shows the predicted class labels for the test set.

```text
[1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0 0 0 1 0 0 2 1
 0]

```

--------------------------------

### onnxruntime.get_version_string

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Returns the version string of the ONNX Runtime installation.

```APIDOC
## onnxruntime.get_version_string

### Description
Returns the version string of the ONNX Runtime installation.

### Signature
get_version_string() -> str
```

--------------------------------

### Load and run the ONNX model with ONNX Runtime to get predictions

Source: https://onnxruntime.ai/docs/api/python/sources/tutorial.rst.txt

Loads the saved ONNX model using rt.InferenceSession with the CPUExecutionProvider, runs inference on the test data, and prints the predictions. The code includes setup steps to suppress stderr output during Sphinx generation.

```python
# hide: start
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
clr = LogisticRegression(max_iter=200, random_state=42)
clr.fit(X_train, y_train)
initial_type = [('float_input', FloatTensorType([None, 4]))]
onx = convert_sklearn(clr, initial_types=initial_type)
with open("logreg_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())

# Force GetPciBusId errors to be piped to null. When generating the logs
# This is because we get some output during initialization in the Sphinx generator
import os
_stderr_fd = os.dup(2)
_devnull = open(os.devnull, "w")
os.dup2(_devnull.fileno(), 2)
# hide: stop

import numpy
import onnxruntime as rt

# hide: start
os.dup2(_stderr_fd, 2)
os.close(_stderr_fd)
_devnull.close()
# hide: stop

sess = rt.InferenceSession("logreg_iris.onnx", providers=["CPUExecutionProvider"])
input_name = sess.get_inputs()[0].name
pred_onx = sess.run(None, {input_name: X_test.astype(numpy.float32)})[0]
print(pred_onx)
```

--------------------------------

### onnxruntime.get_available_providers

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Returns a list of available execution providers in this installed version of ONNX Runtime, in default priority order.

```APIDOC
## onnxruntime.get_available_providers

### Description
Returns a list of available execution providers in this installed version of ONNX Runtime, in default priority order.

### Signature
get_available_providers() -> list[str]
```

--------------------------------

### Use OrtValue for CPU data

Source: https://onnxruntime.ai/docs/api/python/sources/api_summary.rst.txt

Create an OrtValue from a numpy array on CPU, inspect its properties, and pass it as input to a session. The example shows device_name(), shape(), data_type(), is_tensor(), and numpy() usage.

```python
# X is numpy array on cpu
ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X)
ortvalue.device_name()  # 'cpu'
ortvalue.shape()        # shape of the numpy array X
ortvalue.data_type()    # 'tensor(float)'
ortvalue.is_tensor()    # 'True'
np.array_equal(ortvalue.numpy(), X)  # 'True'

# ortvalue can be provided as part of the input feed to a model
session = onnxruntime.InferenceSession(
	'model.onnx',
	providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
results = session.run(["Y"], {"X": ortvalue})
```

--------------------------------

### onnxruntime.backend.prepare

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Loads a model and creates an OnnxRuntimeBackendRep instance ready to be used as a backend. Accepts various model formats including file paths, bytes, ModelProto, InferenceSession, or an existing backend rep.

```APIDOC
## onnxruntime.backend.prepare

### Description
Load the model and creates an `onnxruntime.backend.backend_rep.OnnxRuntimeBackendRep` ready to be used as a backend.

### Method
N/A (Python function)

### Endpoint
N/A

### Parameters
- **model** (str, bytes, onnx.ModelProto, onnxruntime.InferenceSession, or OnnxRuntimeBackendRep) - Required - The model to prepare. Accepts a file path, serialized model bytes, ModelProto, InferenceSession, or an existing backend rep (returned as-is).
- **device** (str or None) - Optional - Requested device for computation. None means the default device depending on compilation settings.
- **kwargs** - Optional - Only a safe subset of `onnxruntime.SessionOptions` attributes are accepted; see `_ALLOWED_SESSION_OPTIONS` for the list.

### Returns
- **onnxruntime.backend.backend_rep.OnnxRuntimeBackendRep** - The prepared backend representation.
```

--------------------------------

### Backend.prepare()

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Prepares the backend for execution with the given model.

```APIDOC
## Backend.prepare()

### Description
Prepares the backend for execution with the given model.

### Method
N/A (Python method)

### Endpoint
N/A

### Parameters
No parameters documented.

### Request Example
N/A

### Response
#### Success Response
- Returns a prepared model object.

#### Response Example
N/A
```

--------------------------------

### Run ONNX model and get specific output by name

Source: https://onnxruntime.ai/docs/api/python/tutorial.html

Runs the ONNX model and retrieves only the first output (label) by specifying its name in the list passed to sess.run. This demonstrates how to get a specific output.

```python
import numpy
import onnxruntime as rt

sess = rt.InferenceSession("logreg_iris.onnx", providers=["CPUExecutionProvider"])
input_name = sess.get_inputs()[0].name
label_name = sess.get_outputs()[0].name
pred_onx = sess.run([label_name], {input_name: X_test.astype(numpy.float32)})[0]
print(pred_onx)

```

--------------------------------

### SessionOptions.graph_optimization_level

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets the graph optimization level for this session.

```APIDOC
## SessionOptions.graph_optimization_level

### Description
Gets or sets the graph optimization level for this session.

### Property
graph_optimization_level

### Type
GraphOptimizationLevel
```

--------------------------------

### AdapterFormat: Create and manage adapter files

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Provides methods to read, export, and manipulate adapter files in ONNX Runtime adapter format, including LoRA parameters. Includes getters and setters for format, adapter, and model versions.

```python
class AdapterFormat:
    """
    This class is used to create adapter files from python structures
    """

    def __init__(self, adapter=None) -> None:
        if adapter is None:
            self._adapter = C.AdapterFormat()
        else:
            self._adapter = adapter

    @staticmethod
    def read_adapter(file_path: os.PathLike) -> AdapterFormat:
        return AdapterFormat(C.AdapterFormat.read_adapter(file_path))

    def export_adapter(self, file_path: os.PathLike):
        """
        This function writes a file at the specified location
        in onnxrunitme adapter format containing Lora parameters.

        :param file_path: absolute path for the adapter
        """
        self._adapter.export_adapter(file_path)

    def get_format_version(self) -> int:
        return self._adapter.format_version

    def set_adapter_version(self, adapter_version: int) -> None:
        self._adapter.adapter_version = adapter_version

    def get_adapter_version(self) -> int:
        return self._adapter.adapter_version

    def set_model_version(self, model_version: int) -> None:
        self._adapter.model_version = model_version

    def get_model_version(self) -> int:
        return self._adapter.model_version
```

--------------------------------

### SessionOptions.execution_mode

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets the execution mode. Default is sequential.

```APIDOC
## SessionOptions.execution_mode

### Description
Gets or sets the execution mode. Default is sequential.

### Property
execution_mode

### Type
ExecutionMode
```

--------------------------------

### IOBinding

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Class providing API to bind input/output to a specified device, e.g., GPU.

```APIDOC
## IOBinding

### Description
This class provides API to bind input/output to a specified device, e.g., GPU.

### Usage
Instantiate IOBinding to manage device-specific input/output bindings for inference sessions.

### Example
```python
from onnxruntime import IOBinding
# Typically obtained from an InferenceSession
io_binding = session.io_binding()
```
```

--------------------------------

### IOBinding.bind_output

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Binds an output to a specified device.

```APIDOC
## bind_output

### Description
Binds an output to a specified device.

### Method
bind_output

### Parameters
- **name** (str) - Required - Output name.
- **device_type** (str) - Optional - Device type, e.g., 'cpu', 'cuda', 'cann'. Default is 'cpu'.
- **device_id** (int) - Optional - Device id, e.g., 0. Default is 0.
- **element_type** (type or int) - Optional - Output element type. Can be numpy type (e.g., numpy.float32) or an integer for onnx type (e.g., onnx.TensorProto.BFLOAT16).
- **shape** (list) - Optional - Output shape.
- **buffer_ptr** (int) - Optional - Memory pointer to output data.

### Response
None
```

--------------------------------

### SessionOptions.execution_order

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets the execution order. Default is basic topological order.

```APIDOC
## SessionOptions.execution_order

### Description
Gets or sets the execution order. Default is basic topological order.

### Property
execution_order

### Type
ExecutionOrder
```

--------------------------------

### Enable profiling with SessionOptions

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Creates a SessionOptions object, enables profiling, and passes it to InferenceSession via sess_options. The providers list is also specified.

```python
options = onnxruntime.SessionOptions()
options.enable_profiling=True
session = onnxruntime.InferenceSession(
        'model.onnx',
        sess_options=options,
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)

```

--------------------------------

### SessionOptions.enable_profiling

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets whether to enable profiling for this session. Default is false.

```APIDOC
## SessionOptions.enable_profiling

### Description
Gets or sets whether to enable profiling for this session. Default is false.

### Property
enable_profiling

### Type
bool
```

--------------------------------

### RunOptions.log_severity_level

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets the log severity level for a particular Run() invocation.

```APIDOC
## RunOptions.log_severity_level

### Description
Gets or sets the log severity level for a particular Run() invocation. 0: Verbose, 1: Info, 2: Warning, 3: Error, 4: Fatal. Default is 2.

### Property
log_severity_level

### Type
int
```

--------------------------------

### Use IOBinding for device-resident data

Source: https://onnxruntime.ai/docs/api/python/sources/api_summary.rst.txt

Replace run() with run_with_iobinding() to keep data on a device like CUDA. Bind CPU input and output, then copy results back to CPU.

```python
# X is numpy array on cpu
session = onnxruntime.InferenceSession(
	'model.onnx',
	providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
# OnnxRuntime will copy the data over to the CUDA device if 'input' is consumed by nodes on the CUDA device
io_binding.bind_cpu_input('input', X)
io_binding.bind_output('output')
session.run_with_iobinding(io_binding)
Y = io_binding.copy_outputs_to_cpu()[0]
```

--------------------------------

### SessionOptions.get_session_config_entry

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets a single session configuration value using the given configuration key.

```APIDOC
## SessionOptions.get_session_config_entry

### Description
Gets a single session configuration value using the given configuration key.

### Method
get_session_config_entry

### Signature
get_session_config_entry(_self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions_, _arg0: str_) -> str

### Parameters
- **key** (str) - Required - The configuration key.

### Returns
- **str** - The configuration value.
```

--------------------------------

### SessionOptions.enable_mem_reuse

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets whether to enable the memory reuse optimization. Default is true.

```APIDOC
## SessionOptions.enable_mem_reuse

### Description
Gets or sets whether to enable the memory reuse optimization. Default is true.

### Property
enable_mem_reuse

### Type
bool
```

--------------------------------

### OrtDevice.make

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Static factory method to create an OrtDevice instance from a device name, device ID, and optional vendor ID.

```APIDOC
## OrtDevice.make

### Description
Creates an OrtDevice instance representing an ONNX Runtime device. Supports predefined device names and vendor-specific device types.

### Method
Static factory method

### Signature
`OrtDevice.make(ort_device_name: str, device_id: int, vendor_id: int = -1) -> OrtDevice`

### Parameters
- **ort_device_name** (str) - Required - Name of the device (e.g., 'cpu', 'cuda', 'dml').
- **device_id** (int) - Required - Device identifier.
- **vendor_id** (int) - Optional - Vendor identifier; defaults to -1.

### Returns
- **OrtDevice** - The created device object.

### Notes
- For predefined device names, vendor ID is automatically resolved.
- For generic device names, vendor ID is used directly.
```

--------------------------------

### Backend.is_compatible()

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Checks whether the backend is compatible with the given model or device.

```APIDOC
## Backend.is_compatible()

### Description
Checks whether the backend is compatible with the given model or device.

### Method
N/A (Python method)

### Endpoint
N/A

### Parameters
No parameters documented.

### Request Example
N/A

### Response
#### Success Response
- Returns a boolean indicating compatibility.

#### Response Example
N/A
```

--------------------------------

### SessionOptions.enable_mem_pattern

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets whether to enable the memory pattern optimization. Default is true.

```APIDOC
## SessionOptions.enable_mem_pattern

### Description
Gets or sets whether to enable the memory pattern optimization. Default is true.

### Property
enable_mem_pattern

### Type
bool
```

--------------------------------

### Load and run an ONNX model with InferenceSession

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Creates an InferenceSession from 'model.onnx' and runs it with a list of output names and an input feed. The session.run() call returns the outputs as a list.

```python
session = onnxruntime.InferenceSession('model.onnx')

outputs = session.run([output names], inputs)

```

--------------------------------

### SessionOptions.enable_cpu_mem_arena

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets whether to enable memory arena on CPU. Default is true.

```APIDOC
## SessionOptions.enable_cpu_mem_arena

### Description
Gets or sets whether to enable memory arena on CPU. Default is true.

### Property
enable_cpu_mem_arena

### Type
bool
```

--------------------------------

### RunOptions.get_run_config_entry

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets a single run configuration value using the given configuration key.

```APIDOC
## RunOptions.get_run_config_entry

### Description
Gets a single run configuration value using the given configuration key.

### Method
get_run_config_entry

### Signature
get_run_config_entry(_self: onnxruntime.capi.onnxruntime_pybind11_state.RunOptions_, _arg0: str_) -> str

### Parameters
- **key** (str) - Required - The configuration key.

### Returns
- **str** - The configuration value.
```

--------------------------------

### Enable profiling with SessionOptions

Source: https://onnxruntime.ai/docs/api/python/sources/api_summary.rst.txt

Create a SessionOptions object, enable profiling, and pass it to InferenceSession via sess_options. This configures the session for profiling.

```python
options = onnxruntime.SessionOptions()
options.enable_profiling=True
session = onnxruntime.InferenceSession(
	'model.onnx',
	sess_options=options,
	providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
```

--------------------------------

### RunOptions.only_execute_path_to_fetches

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets a flag indicating whether to only execute the nodes needed by the fetch list.

```APIDOC
## RunOptions.only_execute_path_to_fetches

### Description
Gets or sets a flag indicating whether to only execute the nodes needed by the fetch list.

### Property
only_execute_path_to_fetches

### Type
bool
```

--------------------------------

### IOBinding.bind_output

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Binds an output to a specific device, optionally with pre-allocated memory.

```APIDOC
## IOBinding.bind_output

### Description
Binds an output to a specific device, optionally with pre-allocated memory.

### Method
bind_output

### Parameters
- **name** (str) - Required - Output name.
- **device_type** (str) - Required - Device type (e.g., 'cpu', 'cuda').
- **device_id** (int) - Required - Device ID.
- **element_type** (numpy dtype or int) - Optional - Element type, required if pre-allocated memory is provided.
- **shape** (list) - Optional - Output shape, required if pre-allocated memory is provided.
- **buffer_ptr** (memory pointer) - Optional - Memory pointer to output data.

### Description
If buffer_ptr is None, ORT allocates memory for the user. Otherwise, element_type and shape must be provided.
```

--------------------------------

### RunOptions.logid

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets the log ID to identify logs generated by a particular Run() invocation.

```APIDOC
## RunOptions.logid

### Description
Gets or sets the log ID to identify logs generated by a particular Run() invocation.

### Property
logid

### Type
str
```

--------------------------------

### Use IOBinding to copy CPU input to GPU and run

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Binds a CPU numpy array as input and an output, then runs with run_with_iobinding. ONNX Runtime copies the input to the CUDA device if needed, and the output is copied back to CPU.

```python
# X is numpy array on cpu
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
# OnnxRuntime will copy the data over to the CUDA device if 'input' is consumed by nodes on the CUDA device
io_binding.bind_cpu_input('input', X)
io_binding.bind_output('output')
session.run_with_iobinding(io_binding)
Y = io_binding.copy_outputs_to_cpu()[0]

```

--------------------------------

### SessionOptions.add_external_initializers_from_files_in_memory

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Provide external initializer file contents from memory.

```APIDOC
## SessionOptions.add_external_initializers_from_files_in_memory

### Description
Provide external initializer file contents from memory.

### Method
add_external_initializers_from_files_in_memory

### Signature
add_external_initializers_from_files_in_memory(_self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions_, _arg0: collections.abc.Sequence[str]_, _arg1: collections.abc.Sequence[collections.abc.Buffer]_, _arg2: collections.abc.Sequence[SupportsInt | SupportsIndex]_) -> None

### Parameters
- **names** (sequence[str]) - Required - Sequence of external file names (as referenced by the model's external_data locations).
- **buffers** (sequence[bytes-like]) - Required - Sequence of bytes-like objects exposing the buffer protocol (e.g., bytes, bytearray, memoryview, numpy uint8 array) containing the corresponding file contents.
- **lengths** (sequence[int]) - Required - Sequence of sizes in bytes for each buffer.

### Notes
- Keep the provided buffers alive until after session creation completes. ONNX Runtime copies needed data during session creation.
- The bytestream must match the external file layout expected by the model (raw tensor bytes at the specified offsets).

### Returns
None
```

--------------------------------

### RunOptions.log_verbosity_level

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Gets or sets the VLOG level if DEBUG build and run_log_severity_level is 0. Applies to a particular Run() invocation. Default is 0.

```APIDOC
## RunOptions.log_verbosity_level

### Description
Gets or sets the VLOG level if DEBUG build and run_log_severity_level is 0. Applies to a particular Run() invocation. Default is 0.

### Property
log_verbosity_level

### Type
int
```

--------------------------------

### Bind device-resident input with IOBinding

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Creates an OrtValue on CUDA device 0 from a numpy array, then binds it as input using bind_input with device details and buffer pointer. The output is bound and copied back to CPU after running.

```python
# X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
io_binding.bind_input(name='input', device_type=X_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=X_ortvalue.shape(), buffer_ptr=X_ortvalue.data_ptr())
io_binding.bind_output('output')
session.run_with_iobinding(io_binding)
Y = io_binding.copy_outputs_to_cpu()[0]

```

--------------------------------

### get_provider_graph_assignment_info()

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Gets information about the subgraphs assigned to each execution provider and the nodes within. Requires the session configuration 'session.record_ep_graph_assignment_info' to be set to '1'.

```APIDOC
## get_provider_graph_assignment_info()

### Description
Get information about the subgraphs assigned to each execution provider and the nodes within.

Application must enable the recording of graph assignment information by setting the session configuration for the key "session.record_ep_graph_assignment_info" to "1".

### Method
`get_provider_graph_assignment_info()`

### Parameters
None

### Returns
- Sequence[C.OrtEpAssignedSubgraph] - List of subgraph assignment info.
```

--------------------------------

### IOBinding.copy_outputs_to_cpu

Source: https://onnxruntime.ai/docs/api/python/sources/api_summary.rst.txt

Copies the bound outputs to CPU and returns them as a list of numpy arrays.

```APIDOC
## [Python] IOBinding.copy_outputs_to_cpu

### Description
Copies the bound outputs to CPU and returns them as a list of numpy arrays.

### Method
copy_outputs_to_cpu

### Returns
- **list** - List of numpy arrays.

### Example
```python
Y = io_binding.copy_outputs_to_cpu()[0]
```
```

--------------------------------

### Configure execution providers with CUDA and CPU

Source: https://onnxruntime.ai/docs/api/python/sources/api_summary.rst.txt

Specify execution providers in priority order. Kernels are chosen from the first provider that supports them; here CUDA is tried first, then CPU.

```python
session = onnxruntime.InferenceSession(
	model, providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)
```

--------------------------------

### _graph_annotation_id: Get effective gpu_graph_id

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Returns the effective gpu_graph_id for a run, defaulting to 0 when unset. Raises ValueError if the run option is not an integer.

```python
def _graph_annotation_id(run_options) -> int:
    """Return the effective ``gpu_graph_id`` for a run, defaulting to 0 when unset."""
    if run_options is None:
        return 0
    try:
        entry = run_options.get_run_config_entry(_GPU_GRAPH_ID_RUN_CONFIG_KEY)
    except RuntimeError:
        return 0
    if not entry:
        return 0
    try:
        return int(entry)
    except ValueError:
        raise ValueError(f"Run option '{_GPU_GRAPH_ID_RUN_CONFIG_KEY}' must be an integer, got {entry!r}.") from None
```

--------------------------------

### IOBinding.get_outputs

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Returns the output OrtValues from the preceding Run() call.

```APIDOC
## IOBinding.get_outputs

### Description
Returns the output OrtValues from the Run() that preceded the call. The data buffer may not reside on CPU memory.

### Returns
- **list** - List of OrtValue instances.
```

--------------------------------

### IOBinding.get_outputs

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Returns the output OrtValues from the preceding Run() call.

```APIDOC
## get_outputs

### Description
Returns the output OrtValues from the Run() that preceded the call. The data buffer of the obtained OrtValues may not reside on CPU memory.

### Method
get_outputs

### Parameters
None

### Response
- Returns a list of OrtValue objects.
```

--------------------------------

### Backend.run()

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Runs the model on the backend with the provided inputs.

```APIDOC
## Backend.run()

### Description
Runs the model on the backend with the provided inputs.

### Method
N/A (Python method)

### Endpoint
N/A

### Parameters
No parameters documented.

### Request Example
N/A

### Response
#### Success Response
- Returns the model output.

#### Response Example
N/A
```

--------------------------------

### IOBinding.copy_outputs_to_cpu

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Copies output contents to CPU.

```APIDOC
## copy_outputs_to_cpu

### Description
Copies output contents to CPU.

### Method
copy_outputs_to_cpu

### Parameters
None

### Response
None
```

--------------------------------

### get_initializer_location callback for ModelCompiler

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Callback for ModelCompiler's get_initializer_location_func parameter. Return None to store the initializer within the compiled model, or return an OrtExternalInitializerInfo to store it externally. The example stores initializers smaller than 64 bytes internally.

```python
def get_initializer_location(
    initializer_name: str,
    initializer_value: onnxrt.OrtValue,
    external_info: onnxrt.OrtExternalInitializerInfo | None,
) -> onnxrt.OrtExternalInitializerInfo | None:
    byte_size = initializer_value.tensor_size_in_bytes()

    if byte_size < 64:
        return None  # Store small initializer within compiled model.
```

--------------------------------

### IOBinding.copy_outputs_to_cpu

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Copies output contents to CPU.

```APIDOC
## IOBinding.copy_outputs_to_cpu

### Description
Copies output contents to CPU.

### Returns
- **object** - Output contents copied to CPU.
```

--------------------------------

### Run async inference with callback and user data

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Asynchronous inference using run_async. Define a callback that receives results and an error string, and optionally user data. The callback is invoked from a C++ thread. The example shows a MyData class and a callback that saves results.

```python
class MyData:
    def __init__(self):
        # ...
    def save_results(self, results):
        # ...

def callback(results: np.ndarray, user_data: MyData, err: str) -> None:
    if err:
        print(err)
    else:
        # save results to user_data
        sess.run_async([output_name], {input_name: x}, callback)

```

--------------------------------

### onnxruntime.InferenceSession

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Documentation for the onnxruntime.InferenceSession class, including its constructor and all public methods.

```APIDOC
## InferenceSession

### Description
This is the main class used to run a model. It loads a model from a file or byte string and provides methods to run inference, inspect inputs/outputs, and manage execution providers.

### Constructor
`onnxruntime.InferenceSession(path_or_bytes, sess_options=None, providers=None, provider_options=None, **kwargs)`

### Parameters
- **path_or_bytes** (str | bytes | PathLike) - Required - Filename or serialized ONNX or ORT format model in a byte string.
- **sess_options** (SessionOptions | None) - Optional - Session options.
- **providers** (Sequence[str | tuple[str, dict]] | None) - Optional - Sequence of providers in order of decreasing precedence. Values can be provider names or tuples of (provider name, options dict). If not provided, all available providers are used with default precedence.
- **provider_options** (Sequence[dict] | None) - Optional - Sequence of options dicts corresponding to the providers listed in 'providers'.

### Notes
- The model type will be inferred unless explicitly set in SessionOptions. Use `so.add_session_config_entry('session.load_model_format', 'ONNX')` or `'ORT'` to set explicitly.
- A file extension of '.ort' will be inferred as an ORT format model; all other filenames are assumed to be ONNX format models.
- 'providers' can contain either names or names and options. When any options are given in 'providers', 'provider_options' should not be used.
- The list of providers is ordered by precedence. For example, ['CUDAExecutionProvider', 'CPUExecutionProvider'] means execute a node using CUDAExecutionProvider if capable, otherwise CPUExecutionProvider.

### Methods

#### create_ortvalue_from_shape_and_type
`create_ortvalue_from_shape_and_type(shape, element_type, device_type, device_id=0, vendor_id=-1) -> OrtValue`

Create an OrtValue using this session's allocator. The value retains this session and may only be updated or bound through it. A matching allocator and a numeric tensor type are required.

#### disable_fallback
`disable_fallback() -> None`

Disable session.run() fallback mechanism.

#### enable_fallback
`enable_fallback() -> None`

Enable session.Run() fallback mechanism. If session.Run() fails due to an internal Execution Provider failure, reset the Execution Providers enabled for this session. If GPU is enabled, fall back to CUDAExecutionProvider; otherwise fall back to CPUExecutionProvider.

#### end_profiling
`end_profiling()`

End profiling and return results in a file. The results are stored in a filename if the option `onnxruntime.SessionOptions.enable_profiling()` is set.

#### get_input_epdevices
`get_input_epdevices() -> Sequence[OrtEpDevice]`

Return the execution providers for the inputs.

#### get_input_memory_infos
`get_input_memory_infos() -> Sequence[OrtMemoryInfo]`

Return the memory info for the inputs.

#### get_inputs
`get_inputs() -> Sequence[NodeArg]`

Return the inputs metadata as a list of `onnxruntime.NodeArg`.

#### get_modelmeta
`get_modelmeta() -> ModelMetadata`

Return the metadata. See `onnxruntime.ModelMetadata`.

#### get_output_memory_infos
`get_output_memory_infos() -> Sequence[OrtMemoryInfo]`

Return the memory info for the outputs.

#### get_outputs
`get_outputs() -> Sequence[NodeArg]`

Return the outputs metadata as a list of `onnxruntime.NodeArg`.

#### get_overridable_initializers
`get_overridable_initializers() -> Sequence[NodeArg]`

Return the inputs (including initializers) metadata as a list of `onnxruntime.NodeArg`.

#### get_profiling_start_time_ns
`get_profiling_start_time_ns()`

Return the nanoseconds of profiling's start time. Comparable to time.monotonic_ns() after Python 3.3. On some platforms, this timer may not be as precise as nanoseconds (e.g., on Windows and MacOS, precision will be ~100ns).

#### get_provider_graph_assignment_info
`get_provider_graph_assignment_info() -> Sequence[OrtEpAssignedSubgraph]`

Get information about the subgraphs assigned to each execution provider and the nodes within. Application must enable the recording of graph assignment information by setting the session configuration for the key "session.record_ep_graph_assignment_info" to "1".

#### get_provider_options
`get_provider_options()`

Return registered execution providers' configurations.

#### get_providers
`get_providers() -> Sequence[str]`

Return list of registered execution providers.

#### get_session_options
`get_session_options() -> SessionOptions`

Return the session options. See `onnxruntime.SessionOptions`.

#### io_binding
`io_binding() -> IOBinding`

Return an onnxruntime.IOBinding object.

#### release_captured_graph
`release_captured_graph(graph_annotation_id=0) -> None`

Release a captured graph and unpin its IOBinding. `graph_annotation_id` matches the capture's `gpu_graph_id` and defaults to zero. EPs without captured-graph release support treat this as a no-op.

#### run
`run(output_names, input_feed, run_options=None) -> Sequence[ndarray | SparseTensor | list | dict]`

Compute the predictions. WebGPU graph capture requires fixed device OrtValues bound with `run_with_iobinding()`; do not use this convenience API for captured replay.
```

--------------------------------

### SessionIOBinding.copy_outputs_to_cpu

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Copies output contents to CPU.

```APIDOC
## copy_outputs_to_cpu

### Description
Copies output contents to CPU.

### Method
copy_outputs_to_cpu

### Parameters
None

### Response
- Returns a list of output values.
```

--------------------------------

### Create OrtValue from numpy array on CPU

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Shows how to create an OrtValue from a numpy array on CPU and inspect its properties (device_name, shape, data_type, is_tensor). Also demonstrates using the OrtValue as input to a session run.

```python
# X is numpy array on cpu
ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X)
ortvalue.device_name()  # 'cpu'
ortvalue.shape()        # shape of the numpy array X
ortvalue.data_type()    # 'tensor(float)'
ortvalue.is_tensor()    # 'True'
np.array_equal(ortvalue.numpy(), X)  # 'True'

# ortvalue can be provided as part of the input feed to a model
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
results = session.run(["Y"], {"X": ortvalue})

```

--------------------------------

### bind_output with device allocation and get_outputs

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Requests ONNX Runtime to allocate output memory on a device and retrieves the resulting OrtValue using get_outputs().

```APIDOC
## bind_output with device allocation and get_outputs

### Description
Binds an output to a device and lets ONNX Runtime allocate the memory. The allocated output can be accessed via get_outputs() as an OrtValue.

### Method
N/A (Python API)

### Endpoint
N/A (Python API)

### Parameters
#### bind_output
- **name** (str) - Required - The name of the model output.
- **device_type** (str) - Required - The device type (e.g., 'cuda').

### Request Example
```python
# X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
io_binding = session.io_binding()
io_binding.bind_input(
        name='input',
        device_type=X_ortvalue.device_name(),
        device_id=0,
        element_type=np.float32,
        shape=X_ortvalue.shape(),
        buffer_ptr=X_ortvalue.data_ptr()
)
# Request ONNX Runtime to bind and allocate memory on CUDA for 'output'
io_binding.bind_output('output', 'cuda')
session.run_with_iobinding(io_binding)
# The following call returns an OrtValue which has data allocated by ONNX Runtime on CUDA
ort_output = io_binding.get_outputs()[0]
```

### Response
#### Success Response
- **ort_output** (OrtValue) - The output OrtValue allocated by ONNX Runtime on the specified device.
```

--------------------------------

### get_overridable_initializers()

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Returns the inputs (including initializers) metadata as a list of onnxruntime.NodeArg.

```APIDOC
## get_overridable_initializers()

### Description
Returns the inputs (including initializers) metadata as a list of :class:`onnxruntime.NodeArg`.

### Method
`get_overridable_initializers()`

### Parameters
None

### Returns
- Sequence[C.NodeArg] - List of input metadata.
```

--------------------------------

### InferenceSession.__init__

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Constructor for the InferenceSession class. Initializes a session with a model path or bytes, session options, providers, and provider options.

```APIDOC
## InferenceSession.__init__

### Description
Initializes an InferenceSession to run a model. The model can be loaded from a file path or from a byte string. Session options, execution providers, and provider options can be specified.

### Method
Constructor

### Signature
```python
InferenceSession(
    path_or_bytes: str | bytes | os.PathLike,
    sess_options: C.SessionOptions | None = None,
    providers: Sequence[str | tuple[str, dict[Any, Any]]] | None = None,
    provider_options: Sequence[dict[Any, Any]] | None = None,
    **kwargs
) -> None
```

### Parameters
- **path_or_bytes** (str | bytes | os.PathLike) - Required - Filename or serialized ONNX or ORT format model in a byte string.
- **sess_options** (C.SessionOptions | None) - Optional - Session options.
- **providers** (Sequence[str | tuple[str, dict[Any, Any]]] | None) - Optional - Sequence of providers in order of decreasing precedence. Values can be provider names or tuples of (provider name, options dict). If not provided, all available providers are used with default precedence.
- **provider_options** (Sequence[dict[Any, Any]] | None) - Optional - Sequence of options dicts corresponding to the providers listed in 'providers'.
- **kwargs** - Optional - Additional keyword arguments. Supports 'enable_fallback' (int) and 'read_config_from_model' (int).

### Notes
- The model type is inferred unless explicitly set in SessionOptions. Use `so.add_session_config_entry('session.load_model_format', 'ONNX')` or `'ORT'` to set explicitly.
- A file extension of '.ort' is inferred as ORT format; all other filenames are assumed to be ONNX format.
- 'providers' can contain names or names with options. When options are given in 'providers', 'provider_options' should not be used.
- Provider list is ordered by precedence. Example: `['CUDAExecutionProvider', 'CPUExecutionProvider']` means use CUDA if capable, otherwise CPU.

### Example
```python
import onnxruntime

so = onnxruntime.SessionOptions()
so.add_session_config_entry('session.load_model_format', 'ONNX')
sess = onnxruntime.InferenceSession('model.onnx', sess_options=so, providers=['CPUExecutionProvider'])
```
```

--------------------------------

### to_cuda()

Source: https://onnxruntime.ai/docs/api/python/api_summary.html

Returns a copy of this instance on the specified CUDA device.

```APIDOC
## to_cuda()

### Description
Returns a copy of this instance on the specified CUDA device.

### Method
to_cuda

### Parameters
- **ort_device** (OrtDevice) - Required - The device with name 'cuda' and a valid GPU device id.

### Returns
- **SparseTensor** - A copy of this instance on the specified CUDA device.

### Exceptions
Throws if:
- This instance contains strings.
- This instance is already on GPU (cross GPU copy is not supported).
- CUDA is not present in this build.
- The specified device is not valid.
```

--------------------------------

### ModelCompiler.__init__

Source: https://onnxruntime.ai/docs/api/python/modules/onnxruntime/capi/onnxruntime_inference_collection.html

Initializes a ModelCompiler instance with an ONNX model path or bytes, session options, and compilation settings.

```APIDOC
## ModelCompiler.__init__

### Description
Initializes a ModelCompiler instance for compiling an ONNX model. Accepts either a model file path or model bytes, along with session options and compilation flags.

### Method
Constructor

### Signature
```python
ModelCompiler(
    sess_options: SessionOptions,
    input_model_path_or_bytes: str | os.PathLike | bytes,
    embed_compiled_data_into_model: bool = False,
    external_initializers_file_path: str | os.PathLike | None = None,
    external_initializers_size_threshold: int = 0,
    flags: dict[str, str] = {},
    graph_optimization_level: GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL,
    get_initializer_location_func: Callable[[str], str] | None = None
)
```

### Parameters
- **sess_options** (SessionOptions) - Required - Session options for the compilation.
- **input_model_path_or_bytes** (str | os.PathLike | bytes) - Required - Path to the ONNX model or bytes of the model. Must not be empty.
- **embed_compiled_data_into_model** (bool) - Optional - Whether to embed compiled data into the model. Defaults to False.
- **external_initializers_file_path** (str | os.PathLike | None) - Optional - Path to write external initializers. Cannot be used with `get_initializer_location_func`.
- **external_initializers_size_threshold** (int) - Optional - Size threshold for external initializers. Defaults to 0.
- **flags** (dict[str, str]) - Optional - Compilation flags. Defaults to empty dict.
- **graph_optimization_level** (GraphOptimizationLevel) - Optional - Graph optimization level. Defaults to ORT_ENABLE_ALL.
- **get_initializer_location_func** (Callable[[str], str] | None) - Optional - Callable to determine initializer location. Cannot be used with `external_initializers_file_path`.

### Raises
- **ValueError** - If input model path/bytes is empty, or if both `external_initializers_file_path` and `get_initializer_location_func` are provided.
- **TypeError** - If input model type is not str, os.PathLike, or bytes, or if `external_initializers_file_path` is not str or os.PathLike.

### Example
```python
from onnxruntime import SessionOptions, ModelCompiler
sess_opts = SessionOptions()
compiler = ModelCompiler(sess_opts, "model.onnx")
```
```

--------------------------------

### IOBinding.bind_output

Source: https://onnxruntime.ai/docs/api/python/sources/api_summary.rst.txt

Binds an output to a device memory location, either by specifying a buffer pointer or by requesting ONNX Runtime to allocate memory.

```APIDOC
## [Python] IOBinding.bind_output

### Description
Binds an output to a device memory location, either by specifying a buffer pointer or by requesting ONNX Runtime to allocate memory.

### Method
bind_output

### Parameters
- **name** (str) - Required - The output name.
- **device_type** (str) - Optional - The device type, e.g., 'cuda'.
- **device_id** (int) - Optional - The device ID.
- **element_type** (numpy.dtype) - Optional - The data type.
- **shape** (tuple) - Optional - The shape of the output.
- **buffer_ptr** (int) - Optional - The memory pointer to the data.

### Example
```python
io_binding.bind_output('output')
io_binding.bind_output('output', 'cuda')
io_binding.bind_output(name='output', device_type=Y_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=Y_ortvalue.shape(), buffer_ptr=Y_ortvalue.data_ptr())
```
```
Tokens:
Raw