### Run an example

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Runs a specific `ort` example using Cargo.

```shell
$ cargo example-gpt2
```

--------------------------------

### Install Training Dependencies

Source: https://github.com/pykeio/ort/blob/main/examples/training/README.md

Install the required Python packages for creating the initial training graph.

```bash
pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT/pypi/simple/ onnxruntime-training-cpu==1.18.0 onnx~=1.17 torch~=2.3
```

--------------------------------

### web-hands

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Build and serve the web-hands example.

```shell
$ cd examples/web-hands
$ wasm-pack build --target web

# The camera is only available in secure contexts like `localhost`, so you'll need to serve the example from an HTTP server.
# We use sirv: https://npm.im/sirv-cli
$ npx sirv-cli
```

--------------------------------

### Run Phi-3 Vision Example

Source: https://github.com/pykeio/ort/blob/main/examples/phi-3-vision/README.md

Execute the Phi-3 Vision example using Cargo after downloading the required model files.

```bash
cargo example-phi-3-vision
```

--------------------------------

### Add Cargo Alias for Running Examples

Source: https://github.com/pykeio/ort/blob/main/CONTRIBUTING.md

Configuration to add a Cargo alias for running a specific example. This simplifies the command to execute the example.

```toml
[alias]
... 
example-agi-net = ["run", "--manifest-path", "examples/agi-net/Cargo.toml", "--example", "agi-net", "--target-dir", "target"]

```

--------------------------------

### Add New Example Cargo.toml Configuration

Source: https://github.com/pykeio/ort/blob/main/CONTRIBUTING.md

Configuration for a new example's Cargo.toml file. Includes path dependencies for ort and optional backends, and feature flags.

```toml
[package]
publish = false
name = "example-agi-net"
version = "0.0.0"
edition = "2024"

[dependencies]
ort = { path = "../../" }

ort-candle = { path = "../../backends/candle", optional = true }
ort-tract = { path = "../../backends/tract", optional = true }

[features]
load-dynamic = [ "ort/load-dynamic" ]

cuda = [ "ort/cuda" ]
# ...copy the rest of the features from gpt2/Cargo.toml...
azure = [ "ort/azure" ]

backend-candle = [ "ort/alternative-backend", "dep:ort-candle" ]
backend-tract = [ "ort/alternative-backend", "dep:ort-tract" ]

[[example]]
name = "agi-net"
path = "agi-net.rs"

```

--------------------------------

### Run with alternative backend

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Runs an example with an alternative backend (e.g., `tract`) using Cargo feature flags.

```shell
$ cargo example-gpt2 --features backend-tract
```

--------------------------------

### Run with execution provider

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Runs an example with a specific execution provider (e.g., CUDA) using Cargo feature flags.

```shell
$ cargo example-gpt2 --features cuda
```

--------------------------------

### Enter the root of the repository

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Navigates into the cloned `ort` repository directory.

```shell
$ cd ort
```

--------------------------------

### New Example Main Rust File Structure

Source: https://github.com/pykeio/ort/blob/main/CONTRIBUTING.md

Basic structure for a new example's main Rust file. Includes common code inclusion, tracing initialization, and EP registration.

```rust
use ort::{...};

// Include common code for `ort` examples that allows using the various feature flags to enable different EPs and
// backends.
#[path = "../common/mod.rs"]
mod common;

fn main() -> ort::Result<()> {
	// Initialize tracing to receive debug messages from `ort`
	tracing_subscriber::registry()
		.with(tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info,ort=debug".into()))
		.with(tracing_subscriber::fmt::layer())
		.init();

	// Register EPs based on feature flags - this isn't crucial for usage and can be removed.
	common::init()?;

    ... // The AGI part is left as an exercise for the reader.

    Ok(())
}

```

--------------------------------

### Clone the repository

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Clones the `ort` repository from GitHub.

```shell
$ git clone https://github.com/pykeio/ort
```

--------------------------------

### Control log verbosity (PowerShell)

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Runs an example while controlling `ort`'s log verbosity using the `RUST_LOG` environment variable in PowerShell.

```powershell
$env:RUST_LOG = 'ort=warn';
cargo example-gpt2
```

--------------------------------

### Control log verbosity

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Runs an example while controlling `ort`'s log verbosity using the `RUST_LOG` environment variable.

```shell
$ RUST_LOG=ort=warn cargo example-gpt2
```

--------------------------------

### Simple Trainer API Example

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Demonstrates using ORT's simple Trainer API for training, similar to Hugging Face's Trainer.

```rust
trainer.train(
    TrainingArguments::new(dataloader)
        .with_lr(7e-5)
        .with_max_steps(5000)
        .with_ckpt_strategy(CheckpointStrategy::Steps(500))
        .with_callbacks(LoggerCallback::new())
)?

```

--------------------------------

### Install `ort-web`

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/web.mdx

Add `ort-web` as a dependency in your `Cargo.toml` file.

```toml
[dependencies]
ort-web = "0.2.2+1.26"
...
```

--------------------------------

### Web Hands Detection

Source: https://github.com/pykeio/ort/blob/main/examples/web-hands/index.html

This code snippet demonstrates how to set up and run hand detection in a web browser using PykeIO, ONNX, and the MediaDevices API.

```javascript
import init, { HandDetector, setup_ort } from './pkg/example_web_kitra_blazehand.js';
await init();
await setup_ort();
const media = await navigator.mediaDevices.getUserMedia({ video: { facingMode: 'user' } });
const video = document.createElement('video');
video.srcObject = media;
await video.play();
const canvas = /** @type {HTMLCanvasElement} */(document.getElementById('canvas'));
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
document.body.appendChild(canvas);
const ctx = canvas.getContext('2d');
const detector = await HandDetector.create('https://spaces.pyke.io/testwebhands/hand_detector.onnx', 2, 0.5, 0.3);
const update = async () => {
    ctx.globalAlpha = 1.0;
    ctx.drawImage(video, 0, 0);
    const { width, height } = canvas;
    for (const detection of await detector.predict_from_canvas(canvas, ctx)) {
        ctx.globalAlpha = detection.score;
        ctx.strokeStyle = '#ff0000';
        ctx.lineWidth = 2;
        ctx.strokeRect(detection.min.x, detection.min.y, detection.width, detection.height);
    }
    requestAnimationFrame(update);
};
requestAnimationFrame(update);
```

--------------------------------

### Serving a Self-Hosted Build

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/web.mdx

Example of how to use a self-hosted build of ort-web by specifying a custom distribution URL and script name.

```Rust
use ort::session::Session;
use ort_web::Dist;

async fn init_model() -> anyhow::Result<Session> {
	let dist = Dist::new("https://cdn.jsdelivr.net/npm/onnxruntime-web@1.26.0/dist/")
		// we want to load the WebGPU build
		.with_script_name("ort.webgpu.min.js");
	ort::set_api(ort_web::api(dist).await?);
}
```

--------------------------------

### Clone a specific version

Source: https://github.com/pykeio/ort/blob/main/examples/README.md

Clones a specific version of the `ort` repository.

```shell
$ git clone https://github.com/pykeio/ort --branch v2.0.0-rc.12
```

--------------------------------

### Initialize tracing subscriber

Source: https://github.com/pykeio/ort/blob/main/docs/content/troubleshooting/logging.mdx

Call the initialization function in your main entry point to start capturing logs.

```rust
fn main() {
    tracing_subscriber::fmt::init();

    // ... rest of code
}
```

--------------------------------

### Training Progress Output

Source: https://github.com/pykeio/ort/blob/main/examples/training/README.md

Example output showing the cross-entropy loss and generated text during the training process.

```text
100%|██████████████████████████████████████| 5000/5000 [06:29<00:00, 12.83it/s, loss=3.611]
I'm so much better than the game<|endoftext|>I think you can't see it<|endoftext|>I think you can't see it<|endoftext|>I think so it's a new game<|endoftext|>I think I'm sure you can't see what you can't see it<|endoftext|>
```

--------------------------------

### Session creation

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/web.mdx

Example of creating a session using `commit_from_url` and awaiting the result, as required on the Web.

```rust
use ort::{ep, session::Session};
use ort_web::FEATURE_WEBGPU;
use wasm_bindgen::JsError;

async fn init() -> Result<(), JsError> {
    ort::set_api(ort_web::api(FEATURE_WEBGPU).await?);

    let mut session = Session::builder()? 
        .with_execution_providers([
            // only available with FEATURE_WEBGPU
            ep::WebGPU::default().build()
        ])?
        .commit_from_url("./model.onnx")
        .await?; // <- note we must .await on the web
}
```

--------------------------------

### Synchronizing Session Outputs

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/web.mdx

Example of how to synchronize session outputs, either all at once or individually, to access their data in Rust.

```Rust
use ort_web::{TensorExt, SyncDirection};

let mut outputs = session.run_async(ort::inputs![...]).await?;

let mut bounding_boxes = outputs.remove("bounding_boxes").unwrap();
bounding_boxes.sync(SyncDirection::Rust).await?;

// now we can use the data
let data = bounding_boxes.try_extract_tensor::<f32>()?;
```

--------------------------------

### Check EP Availability Before Registration

Source: https://github.com/pykeio/ort/blob/main/docs/content/perf/execution-providers.mdx

Use `is_available()` to check if ONNX Runtime was compiled with support for a specific execution provider. This example checks for CoreML support and exits if it's not available.

```rust
use ort::{
    ep::{self, ExecutionProvider},
    session::Session
};

fn main() -> anyhow::Result<()> {
    let builder = Session::builder()?;

    let coreml = ep::CoreML::default();
    if !coreml.is_available() {
        eprintln!("Please compile ONNX Runtime with CoreML!");
        std::process::exit(1);
    }

    // Note that even though ONNX Runtime was compiled with CoreML, registration could still fail!
    coreml.register(&builder)?;

    let session = builder.commit_from_file("model.onnx")?;

    Ok(())
}
```

--------------------------------

### Install ort-candle Dependency

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/candle.mdx

Add the ort-candle crate to your Cargo.toml to include it as a dependency. Ensure version compatibility.

```toml
[dependencies]
ort-candle = "0.3.0+0.9.2"
...
```

--------------------------------

### Manually Register EP with Custom Error Handling

Source: https://github.com/pykeio/ort/blob/main/docs/content/perf/execution-providers.mdx

Manually register execution providers using `ExecutionProvider::register` for more complex error handling. This example checks if CUDA registration fails and prints an error message before exiting.

```rust
use ort::{
    ep::{self, ExecutionProvider},
    session::Session
};

fn main() -> anyhow::Result<()> {
    let builder = Session::builder()?;

    let cuda = ep::CUDA::default();
    if cuda.register(&builder).is_err() {
        eprintln!("Failed to register CUDA!");
        std::process::exit(1);
    }

    let session = builder.commit_from_file("model.onnx")?;

    Ok(())
}
```

--------------------------------

### Create an IoBinding instance

Source: https://github.com/pykeio/ort/blob/main/docs/content/perf/io-binding.mdx

Initialize an IoBinding object from an existing session.

```rust
let mut binding = session.create_binding()?;
```

--------------------------------

### Initialize the backend

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/web.mdx

Initialize the `ort-web` backend by calling `ort::set_api` with the result of `ort_web::api`.

```rust
use ort_web::FEATURE_WEBGPU;
use wasm_bindgen::JsError;

async fn init() -> Result<(), JsError> {
    // This should always be run before you use any other `ort` API.
    ort::set_api(ort_web::api(FEATURE_WEBGPU).await?);

    ...
}
```

--------------------------------

### Configure Global Execution Providers

Source: https://github.com/pykeio/ort/blob/main/docs/content/perf/execution-providers.mdx

Initialize the ONNX Runtime environment with global execution providers using `ort::init().with_execution_providers(...).commit()`. These EPs will be available for all subsequent sessions created in the program.

```rust
use ort::{ep, session::Session};

fn main() -> anyhow::Result<()> {
    ort::init()
        .with_execution_providers([ep::CUDA::default().build()])
        .commit();

    let session = Session::builder()?.commit_from_file("model.onnx")?;
    // The session will attempt to register the CUDA EP
    // since we configured environment EPs.

    Ok(())
}
```

--------------------------------

### Install ort-tract dependency

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/tract.mdx

Add the ort-tract crate to your Cargo.toml dependencies.

```toml
[dependencies]
ort-tract = "0.3.0+0.22"
...
```

--------------------------------

### Disabling Telemetry

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/web.mdx

Example of how to disable telemetry in ort-web by setting with_telemetry(false) during initialization.

```Rust
use wasm_bindgen::JsError;
async fn init() -> Result<(), JsError> {
    ort::set_api(ort_web::api().await?);

    ort::init()
        .with_telemetry(false)
        .commit();

    // ...
}
```

--------------------------------

### Run session with binding

Source: https://github.com/pykeio/ort/blob/main/docs/content/perf/io-binding.mdx

Execute the session using the configured binding and retrieve the bound outputs.

```rust
let outputs = session.run_binding(&binding)?;
```

```rust
// same `action` we allocated earlier in `bind_output`
let action: Tensor<f32> = outputs.remove("action").unwrap().downcast()?;
```

--------------------------------

### Initialize ort with Dynamic Library Path Programmatically

Source: https://github.com/pykeio/ort/blob/main/docs/content/setup/linking.mdx

Initialize `ort` by providing the path to the ONNX Runtime dynamic library programmatically using `ort::init_from`. This must be called before any other `ort` usage. The function returns an `EnvironmentBuilder` for further configuration.

```rust
fn find_onnxruntime_dylib() -> anyhow::Result<PathBuf> {
    // Find our custom ONNX Runtime dylib path somehow (i.e. resolving it from the root of our program's install folder)
    // The path should point to the `libonnxruntime` binary, which looks like:
    // - on Windows: C:\Program Files\...\onnxruntime.dll
    // - on Linux: /.../.../libonnxruntime.so
    // - on macOS: /.../.../libonnxruntime.dylib
    unimplemented!()
}

fn main() -> anyhow::Result<()> {
    let dylib_path = find_onnxruntime_dylib()?;

    // Initialize ort with the path to the dylib. This **must** be called before any other usage of `ort`!
    // `init_from` returns a `Result<EnvironmentBuilder>` which you can use to further configure the environment
    // before `.commit()`ing; see the Environment docs for more information on what you can configure.
    // `init_from` will return an `Err` if it fails to load the dylib.
    ort::init_from(dylib_path)?.commit();

    // Now we can use `ort`!
    let mut session = Session::builder()? 
        .commit_from_file("model.onnx")?;

    Ok(())
}
```

--------------------------------

### Implement I/O Binding for Diffusion Pipeline

Source: https://github.com/pykeio/ort/blob/main/docs/content/perf/io-binding.mdx

Demonstrates using I/O binding to optimize a text-to-image pipeline by keeping tensors on the GPU and reusing memory allocations.

```rs
let mut text_encoder = Session::builder()?
	.with_execution_providers([ep::CUDA::default().build()])?
	.commit_from_file("text_encoder.onnx")?;
let mut unet = Session::builder()?
	.with_execution_providers([ep::CUDA::default().build()])?
	.commit_from_file("unet.onnx")?;

let text_condition = {
    let mut binding = text_encoder.create_binding()?;
    binding.bind_input("tokens", &Tensor::<i64>::from_array((
        vec![1, 22],
        vec![49, 272, 503, 286, 1396, 353, 9653, 284, 1234, 287, 616, 2438, 11, 7926, 13, 3423, 338, 3362, 25, 12520, 238, 242]
    ))?)?;
    binding.bind_output_to_device("output0", &text_encoder.allocator().memory_info())?;
    text_encoder.run_binding(&binding)?.remove("output0").unwrap()
};

let input_allocator = Allocator::new(
	&unet,
	MemoryInfo::new(AllocationDevice::CUDA_PINNED, 0, AllocatorType::Device, MemoryType::CPUInput)?
)?;
let mut latents = Tensor::<f32>::new(&input_allocator, [1, 4, 64, 64])?;

let mut io_binding = unet.create_binding()?;
io_binding.bind_input("condition", &text_condition)?;

let output_allocator = Allocator::new(
	&unet,
	MemoryInfo::new(AllocationDevice::CUDA_PINNED, 0, AllocatorType::Device, MemoryType::CPUOutput)?
)?;
io_binding.bind_output("noise_pred", Tensor::<f32>::new(&output_allocator, [1, 4, 64, 64])?)?;

for _ in 0..20 {
	io_binding.bind_input("latents", &latents)?;
	let noise_pred = unet.run_binding(&io_binding)?.remove("noise_pred").unwrap();

	let mut latents = latents.extract_array_mut();
	latents += &noise_pred.try_extract_array::<f32>()?;
}
```

--------------------------------

### Load an ONNX Model

Source: https://github.com/pykeio/ort/blob/main/docs/content/index.mdx

Initialize a session with specific optimization levels and thread configurations.

```rust
use ort::session::{builder::GraphOptimizationLevel, Session};

let mut model = Session::builder()?
    .with_optimization_level(GraphOptimizationLevel::Level3)?
    .with_intra_threads(4)?
    .commit_from_file("yolov8m.onnx")?;
```

--------------------------------

### Pinning ort's minor version

Source: https://github.com/pykeio/ort/blob/main/docs/content/migrating/version-mapping.mdx

Example of how to pin ort's minor version in Cargo.toml using a tilde requirement.

```toml
[dependencies]
ort = { version = "~2.0", ... }
```

--------------------------------

### Enable the `alternative-backend` feature

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/web.mdx

Enable the `alternative-backend` feature for the `ort` crate to instruct it to use the alternative backend.

```toml
[dependencies.ort]
version = "=2.0.0-rc.12"
default-features = false # Disables the `download-binaries` feature since we don't need it
features = [
    "std",
    "ndarray",
    "api-26",
    "alternative-backend"
]
```

--------------------------------

### Format Code with Nightly Rust Toolchain

Source: https://github.com/pykeio/ort/blob/main/CONTRIBUTING.md

Command to format Rust code using the nightly toolchain. Requires installing the nightly toolchain first.

```shell
$ cargo +nightly fmt

```

--------------------------------

### Initialize ort-candle Backend

Source: https://github.com/pykeio/ort/blob/main/docs/content/backends/candle.mdx

Call `ort::set_api` with `ort_candle::api()` as early as possible in your application's main function to set the ort-candle backend. This must be done before any other `ort` operations.

```rust
fn main() {
    // This should run as early in your application as possible - before you ever use `ort`!
    ort::set_api(ort_candle::api());
}
```

--------------------------------

### Downcasting DynValue to TensorRef<'_, f32> via view().downcast()

Source: https://github.com/pykeio/ort/blob/main/docs/content/fundamentals/value.mdx

An alternative method to obtain a reference to a strongly-typed TensorRef<'_, f32> from a DynValue by first getting a view and then downcasting. This is equivalent to using downcast_ref.

```rust
// is equivalent to
let tensor_view: ort::value::TensorRef<'_, f32> = dyn_value.view().downcast()?;
```

--------------------------------

### Cargo.toml Configuration for Specific API Version

Source: https://github.com/pykeio/ort/blob/main/docs/content/setup/multiversion.mdx

This example shows how to configure `Cargo.toml` to use a specific API version of ONNX Runtime, disabling default features and enabling only necessary ones.

```toml
[dependencies.ort]
version = "=2.0.0-rc.12"
default-features = false
features = [
    "std",
    "ndarray",
    "download-binaries",
    
    # We use the `Adapter` API, which is only available since ONNX Runtime 1.20,
    # so set our minimum API version to 20.
    "api-20"
]
```

--------------------------------

### Configure rpath for Linux and macOS

Source: https://github.com/pykeio/ort/blob/main/docs/content/setup/linking.mdx

Set the linker arguments in .cargo/config.toml to point to the executable's origin directory.

```toml
[target.x86_64-unknown-linux-gnu]
rustflags = [ "-Clink-args=-Wl,-rpath,\$ORIGIN" ]

# do this for any other Linux targets as well
```

```toml
[target.x86_64-apple-darwin]
rustflags = [ "-Clink-args=-Wl,-rpath,@loader_path" ]

# do this for any other macOS targets as well
```

--------------------------------

### Enable Compile-time Dynamic Linking

Source: https://github.com/pykeio/ort/blob/main/docs/content/setup/linking.mdx

To use compile-time dynamic linking, configure the environment as for static linking and set `ORT_PREFER_DYNAMIC_LINK` to `1`.

```shell
export ORT_PREFER_DYNAMIC_LINK=1
export ORT_LIB_PATH=~/onnxruntime/build/Linux/Release
cargo build
```