OpenVINO

https://docs.openvino.ai/2026/
Admin
OpenVINO is an open-source toolkit for optimizing, accelerating, and deploying deep learning models...
Tokens:2,314,485
Snippets:17,019
Trust Score:9.2
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Pre-production OpenVINO GenAI (Linux)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Installs the pre-production versions of OpenVINO, tokenizers, and GenAI packages from a nightly index for Linux.

```bash
pip install --pre openvino openvino-tokenizers openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
```

--------------------------------

### C Project Structure Example

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Illustrates a typical project directory structure for a C application using OpenVINO and CMake.

```text
project/
   ├── CMakeLists.txt  - CMake file to build
   ├── ...             - Additional folders like includes/
   └── src/            - source folder
       └── main.c
build/                  - build directory
   ...


```

--------------------------------

### C++ Project Structure Example

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Illustrates a typical project directory structure for a C++ application using OpenVINO and CMake.

```text
project/
   ├── CMakeLists.txt  - CMake file to build
   ├── ...             - Additional folders like includes/
   └── src/            - source folder
       └── main.cpp
build/                  - build directory
   ...


```

--------------------------------

### Install OpenVINO GenAI Dependencies (Windows)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Installs the necessary OpenVINO and related packages for GenAI inference on NPU in a virtual environment on Windows.

```bash
python -m venv npu-env
npu-env\Scripts\activate
pip install nncf==2.18.0 onnx==1.18.0 optimum-intel==1.25.2 transformers==4.51.3
pip install openvino==2026.2 openvino-tokenizers==2026.2 openvino-genai==2026.2
```

--------------------------------

### Install OpenVINO GenAI Dependencies (Linux)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Installs the necessary OpenVINO and related packages for GenAI inference on NPU in a virtual environment on Linux.

```bash
python3 -m venv npu-env
source npu-env/bin/activate
pip install nncf==2.18.0 onnx==1.18.0 optimum-intel==1.25.2 transformers==4.51.3
pip install openvino==2026.2 openvino-tokenizers==2026.2 openvino-genai==2026.2
```

--------------------------------

### OpenVINO GPU Plugin Queue and Context Sharing Example

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Demonstrates initializing OpenVINO, sharing an OpenCL command queue and context with the GPU plugin, creating remote tensors, and synchronizing user kernels with inference execution.

```cpp
// ...

    // initialize the core and read the model
    ov::Core core;
    auto model = core.read_model("model.xml");

    // get opencl queue object
    cl::CommandQueue queue = get_ocl_queue();
    cl::Context cl_context = get_ocl_context();

    // share the queue with GPU plugin and compile model
    auto remote_context = ov::intel_gpu::ocl::ClContext(core, queue.get());
    auto exec_net_shared = core.compile_model(model, remote_context);

    auto input = model->get_parameters().at(0);
    auto input_size = ov::shape_size(input->get_shape());
    auto output = model->get_results().at(0);
    auto output_size = ov::shape_size(output->get_shape());
    cl_int err;

    // create the OpenCL buffers within the context
    cl::Buffer shared_in_buffer(cl_context, CL_MEM_READ_WRITE, input_size, NULL, &err);
    cl::Buffer shared_out_buffer(cl_context, CL_MEM_READ_WRITE, output_size, NULL, &err);
    // wrap in and out buffers into RemoteTensor and set them to infer request
    auto shared_in_blob = remote_context.create_tensor(input->get_element_type(), input->get_shape(), shared_in_buffer);
    auto shared_out_blob = remote_context.create_tensor(output->get_element_type(), output->get_shape(), shared_out_buffer);
    auto infer_request = exec_net_shared.create_infer_request();
    infer_request.set_tensor(input, shared_in_blob);
    infer_request.set_tensor(output, shared_out_blob);

    // ...
    // execute user kernel
    cl::Program program;
    cl::Kernel kernel_preproc(program, "user_kernel_preproc");
    kernel_preproc.setArg(0, shared_in_buffer);
    queue.enqueueNDRangeKernel(kernel_preproc, 
                               cl::NDRange(0), 
                               cl::NDRange(input_size), 
                               cl::NDRange(1), 
                               nullptr, 
                               nullptr);
    // Blocking clFinish() call is not required, but this barrier is added to the queue to guarantee that user kernel is finished
    // before any inference primitive is started
    queue.enqueueBarrierWithWaitList(nullptr, nullptr);
    // ...

    // pass results to the inference
    // since the remote context is created with queue sharing, start_async() guarantees that scheduling is finished
    infer_request.start_async();

    // execute some postprocessing kernel.
    // infer_request.wait() is not called, synchronization between inference and post-processing is done via
    // enqueueBarrierWithWaitList call.
    cl::Kernel kernel_postproc(program, "user_kernel_postproc");
    kernel_postproc.setArg(0, shared_out_buffer);
    queue.enqueueBarrierWithWaitList(nullptr, nullptr);
    queue.enqueueNDRangeKernel(kernel_postproc, 
                               cl::NDRange(0), 
                               cl::NDRange(output_size), 
                               cl::NDRange(1), 
                               nullptr, 
                               nullptr);

    // Wait for pipeline completion
    queue.finish();

```

--------------------------------

### JavaScript/TypeScript Chat Scenario with KVCache

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Node.js example for interactive chat using LLMPipeline. Demonstrates starting and finishing chat sessions and handling user input.

```javascript
import { LLMPipeline } from "openvino-genai-node";
import readline from 'readline';

const pipe = await LLMPipeline(model_path, 'CPU');

const config = {
    max_new_tokens: 100,
    num_beam_groups: 3,
    num_beams: 15,
    diversity_penalty: 1.5
};

await pipe.startChat();

const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
});

console.log('question:');
rl.on('line', async (prompt) => {
    console.log('answer:');
    const answer = await pipe.generate(prompt, config);
    console.log(answer);
    console.log('\n----------\nquestion:');
});

rl.on('close', async () => {
    // highlight-next-line
    await pipe.finishChat();
    process.exit(0);
});

```

--------------------------------

### VLM Configuration with Device Properties (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

This C++ example shows how to pass device-specific configurations, such as prompt and response lengths, to the VLM pipeline for NPU inference.

```C++
#include "load_image.hpp"
#include <openvino/genai/visual_language/pipeline.hpp>
#include <iostream>

bool print_subword(std::string&& subword) {
   return !(std::cout << subword << std::flush);
}

int main(int argc, char* argv[]) {
   std::string model_path = "Phi-4-multimodal-instruct";
   std::string image_path = "cat.png";

   std::vector<ov::Tensor> rgbs = utils::load_images(image_path);
   ov::AnyMap pipeline_config = {
      {"DEVICE_PROPERTIES", ov::AnyMap{
         {"NPU", ov::AnyMap{
            {"MAX_PROMPT_LEN", 2048},
            {"MIN_RESPONSE_LEN", 512}
         }}
      }}
   };

   ov::genai::VLMPipeline pipe(model_path, "NPU", pipeline_config);
   ov::genai::GenerationConfig config;
   config.max_new_tokens=100;
   std::cout << pipe.generate("Describe the image",
      ov::genai::images(rgbs),
      ov::genai::generation_config(config),
      ov::genai::streamer(print_subword));
}


```

--------------------------------

### Run Text Generation with OpenVINO GenAI on NPU (Python)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Performs text generation using the LLMPipeline for NPU devices. Ensure the latest NPU driver is installed.

```Python
import openvino_genai as ov_genai
model_path = "TinyLlama"
pipe = ov_genai.LLMPipeline(model_path, "NPU")
print(pipe.generate("The Sun is yellow because", max_new_tokens=100))
```

--------------------------------

### Run Text Generation with OpenVINO GenAI on NPU (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Performs text generation using the LLMPipeline for NPU devices in C++. Ensure the latest NPU driver is installed.

```C++
#include "openvino/genai/llm_pipeline.hpp"
#include <iostream>

int main(int argc, char* argv[]) {
   std::string model_path = "TinyLlama";
   ov::genai::LLMPipeline pipe(models_path, "NPU");
   ov::genai::GenerationConfig config;
   config.max_new_tokens=100;
   std::cout << pipe.generate("The Sun is yellow because", config);
}
```

--------------------------------

### Blob Encryption Import Example (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Import an NPU LLM blob with decryption enabled. Requires defining the `decrypt` callback and providing `BLOB_PATH` and `WEIGHTS_PATH` if the blob is weightless.

```C++
ov::EncryptionCallbacks encryption_callbacks;
encryption_callbacks.decrypt = [](const std::string& s) { return s; };
ov::AnyMap pipeline_config = { { "BLOB_PATH", ".npucache\compiled_model.blob" }, { "WEIGHTS_PATH", "path\to\original\model.bin" }, { "CACHE_ENCRYPTION_CALLBACKS", encryption_callbacks } };
ov::genai::LLMPipeline pipe(model_path, "NPU", pipeline_config);
```

--------------------------------

### VLM Inference on NPU (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

This C++ example demonstrates VLM inference on NPU. It includes image loading and pipeline generation.

```C++
#include "load_image.hpp"
#include <openvino/genai/visual_language/pipeline.hpp>
#include <iostream>

bool print_subword(std::string&& subword) {
   return !(std::cout << subword << std::flush);
}

int main(int argc, char* argv[]) {
   std::string model_path = "Google-Gemma-3-4B-it";
   std::string image_path = "cat.png";

   std::vector<ov::Tensor> rgbs = utils::load_images(image_path);

   ov::genai::VLMPipeline pipe(model_path, "NPU");
   ov::genai::GenerationConfig config;
   config.max_new_tokens=100;
   std::cout << pipe.generate("Describe the image",
      ov::genai::images(rgbs),
      ov::genai::generation_config(config),
      ov::genai::streamer(print_subword));
}


```

--------------------------------

### C++: Inference with GGUF Model

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Perform text generation using a GGUF model in C++. This example demonstrates setting up the LLMPipeline and generation configuration.

```C++
#include <openvino/openvino.hpp>
#include "openvino/genai/llm_pipeline.hpp"

int main(int argc, char* argv[]) try {
    ov::genai::GenerationConfig config;
    config.max_new_tokens = 100;

    std::string model_path = "SmolLM2-135M.F16.gguf";
    std::string prompt = "The Sun is yellow because";

    ov::genai::LLMPipeline pipe(model_path, "CPU");

    auto result = pipe.generate("The Sun is yellow because", config);
    std::cout << "result = " << result << std::endl;
} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
}


```

--------------------------------

### C++ CMake Configuration with PyPI OpenVINO

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Configures a C++ project using CMake when OpenVINO is installed via PyPI. It dynamically finds the CMake path from Python utilities.

```cmake
cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 17)

if(NOT CMAKE_CROSSCOMPILING)
    find_package(Python3 QUIET COMPONENTS Interpreter)
    if(Python3_Interpreter_FOUND)
        execute_process(
            COMMAND ${Python3_EXECUTABLE} -c "from openvino.utils import get_cmake_path; print(get_cmake_path(), end='')"
            OUTPUT_VARIABLE OpenVINO_DIR_PY
            ERROR_QUIET)
    endif()
endif()

find_package(OpenVINO REQUIRED PATHS "${OpenVINO_DIR_PY}")

add_executable(${TARGET_NAME_PY} src/main.cpp)

target_link_libraries(${TARGET_NAME_PY} PRIVATE openvino::runtime)


```

--------------------------------

### JavaScript/TypeScript: Inference with GGUF Model

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Execute text generation with a GGUF model using the OpenVINO GenAI Node.js binding. This asynchronous example shows pipeline initialization and generation.

```JavaScript/TypeScript
import { LLMPipeline, GenerationConfig } from 'openvino-genai-node';

async function main() {
  const config: GenerationConfig = {
    maxNewTokens: 100
  };

  const modelPath = 'SmolLM2-135M.F16.gguf';
  const prompt = 'The Sun is yellow because';

  const pipe = await LLMPipeline(modelPath, 'CPU');

  const result = await pipe.generate(prompt, config);
  console.log('result =', result.toString());
}

main();


```

--------------------------------

### Blob Encryption Export Example (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Export an NPU LLM blob with encryption enabled. For weightless blobs, the entire blob is encrypted. For blobs with weights, everything except the model weights is encrypted. Requires defining `encrypt` callback.

```C++
ov::EncryptionCallbacks encryption_callbacks;
encryption_callbacks.encrypt = [](const std::string& s) { return s; };
ov::AnyMap pipeline_config = { { "EXPORT_BLOB", "YES" }, { "BLOB_PATH", ".npucache\compiled_model.blob" }, { "CACHE_ENCRYPTION_CALLBACKS", encryption_callbacks } };
ov::genai::LLMPipeline pipe(model_path, "NPU", pipeline_config);
```

--------------------------------

### Initialize LLMPipeline and Generate Text (Python)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Initialize the LLMPipeline with a model path and device, then generate text. Ensure the model path is correctly set.

```python
import openvino_genai as ov_genai
pipe = ov_genai.LLMPipeline(model_path, "CPU")
print(pipe.generate("The Sun is yellow because", max_new_tokens=100))
```

--------------------------------

### Initialize LLMPipeline and Generate Text (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Initialize the LLMPipeline with a model path and device, then generate text. The model path is passed as a command-line argument.

```cpp
#include "openvino/genai/llm_pipeline.hpp"
#include <iostream>

int main(int argc, char* argv[]) {
   std::string model_path = argv[1];
   ov::genai::LLMPipeline pipe(model_path, "CPU");
   std::cout << pipe.generate("The Sun is yellow because", ov::genai::max_new_tokens(100));
}
```

--------------------------------

### Get Output Tensor by Index

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Retrieves an output tensor by its index using the C API.

```C
ov_tensor_t* output_tensor = NULL;
// Get output tensor by tensor index
ov_infer_request_get_output_tensor_by_index(infer_request, 0, &output_tensor);

```

--------------------------------

### Get Output Tensor Data

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Retrieves the output tensor for a model with a single output and accesses its data buffer.

```Python
# Get output tensor for model with one output
output = infer_request.get_output_tensor()
output_buffer = output.data
# output_buffer[] - accessing output tensor data

```

--------------------------------

### Initialize LLMPipeline and Generate Text (JavaScript/TypeScript)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Initialize the LLMPipeline asynchronously with a model path and device, then generate text. The model path is passed as a command-line argument.

```javascript
import { LLMPipeline, GenerationConfig } from 'openvino-genai-node';

async function main() {
  const modelPath = process.argv[2];
  const pipe = await LLMPipeline.LLMPipeline(modelPath, 'CPU');

  const config: GenerationConfig = { maxNewTokens: 100 };
  const result = await pipe.generate('The Sun is yellow because', config);

  console.log(result.texts[0]);
}

main();
```

--------------------------------

### OpenCL Kernel Execution on Shared Buffer with GPU Plugin

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Demonstrates initializing OpenVINO, compiling a model for the GPU, obtaining an OpenCL context, creating a shared OpenCL buffer, wrapping it into a RemoteBlob, executing a user-defined OpenCL kernel on the buffer, and then passing the buffer to the inference request.

```cpp
// ...

    // initialize the core and load the network
    ov::Core core;
    auto model = core.read_model("model.xml");
    auto compiled_model = core.compile_model(model, "GPU");
    auto infer_request = compiled_model.create_infer_request();


    // obtain the RemoteContext from the compiled model object and cast it to ClContext
    auto gpu_context = compiled_model.get_context().as<ov::intel_gpu::ocl::ClContext>();
    // obtain the OpenCL context handle from the RemoteContext,
    // get device info and create a queue
    cl::Context cl_context = gpu_context;
    cl::Device device = cl::Device(cl_context.getInfo<CL_CONTEXT_DEVICES>()[0].get(), true);
    cl_command_queue_properties props = CL_QUEUE_OUT_OF_ORDER_EXEC_MODE_ENABLE;
    cl::CommandQueue queue = cl::CommandQueue(cl_context, device, props);

    // create the OpenCL buffer within the obtained context
    auto input = model->get_parameters().at(0);
    auto input_size = ov::shape_size(input->get_shape());
    cl_int err;
    cl::Buffer shared_buffer(cl_context, CL_MEM_READ_WRITE, input_size, NULL, &err);
    // wrap the buffer into RemoteBlob
    auto shared_blob = gpu_context.create_tensor(input->get_element_type(), input->get_shape(), shared_buffer);

    // ...
    // execute user kernel
    cl::Program program;
    cl::Kernel kernel(program, "user_kernel");
    kernel.setArg(0, shared_buffer);
    queue.enqueueNDRangeKernel(kernel,
                               cl::NDRange(0),
                               cl::NDRange(input_size),
                               cl::NDRange(1),
                               nullptr,
                               nullptr);
    queue.finish();
    // ...
    // pass results to the inference
    infer_request.set_tensor(input, shared_blob);
    infer_request.infer();


```

--------------------------------

### Build Project with CMake

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Commands to navigate to the build directory and use CMake to build the configured project.

```bash
cd build/
cmake ../project
cmake --build .


```

--------------------------------

### Get Output Tensor Data by Name

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Retrieves an output tensor by its name and accesses its data buffer as const float pointers in C++.

```C++
// Get output tensor by tensor name
auto output = infer_request.get_tensor("tensor_name");
const float *output_buffer = output.data<const float>();
// output_buffer[] - accessing output tensor data

```

--------------------------------

### Configure Default Caching with CACHE_DIR (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Use `CACHE_DIR` to enable OpenVINO's default caching mechanism. The `CACHE_MODE` hint controls how weights are stored. `OPTIMIZE_SPEED` includes weights for faster loading, while `OPTIMIZE_SIZE` creates a weightless blob.

```C++
ov::AnyMap pipeline_config = { { "CACHE_DIR",  ".npucache" } };
ov::genai::LLMPipeline pipe(model_path, "NPU", pipeline_config);
```

--------------------------------

### Configure Default Caching with CACHE_DIR (Python)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Use `CACHE_DIR` to enable OpenVINO's default caching mechanism. The `CACHE_MODE` hint controls how weights are stored. `OPTIMIZE_SPEED` includes weights for faster loading, while `OPTIMIZE_SIZE` creates a weightless blob.

```Python
pipeline_config = { "CACHE_DIR": ".npucache" }
pipe = ov_genai.LLMPipeline(model_path, "NPU", pipeline_config)
```

--------------------------------

### Create NV12 Remote Blobs from ID3D11Texture2D

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Shows how to create NV12 format remote tensors from an `ID3D11Texture2D` using `D3DContext`. This involves preprocessing setup and wrapping decoder output into `RemoteBlobs`.

```C++
using namespace ov::preprocess;
auto p = PrePostProcessor(model);
p.input().tensor().set_element_type(ov::element::u8)
                  .set_color_format(ov::preprocess::ColorFormat::NV12_TWO_PLANES, {"y", "uv"})
                  .set_memory_type(ov::intel_gpu::memory_type::surface);
p.input().preprocess().convert_color(ov::preprocess::ColorFormat::BGR);
p.input().model().set_layout("NCHW");
model = p.build();

CComPtr<ID3D11Device> device_ptr = get_d3d_device_ptr();
// create the shared context object
auto shared_d3d_context = ov::intel_gpu::ocl::D3DContext(core, device_ptr);
// compile model within a shared context
auto compiled_model = core.compile_model(model, shared_d3d_context);

auto param_input_y = model->get_parameters().at(0);
auto param_input_uv = model->get_parameters().at(1);

D3D11_TEXTURE2D_DESC texture_description = get_texture_desc();
CComPtr<ID3D11Texture2D> dx11_texture = get_texture();
//     ...
//wrap decoder output into RemoteBlobs and set it as inference input
auto nv12_blob = shared_d3d_context.create_tensor_nv12(texture_description.Heights, texture_description.Width, dx11_texture);

auto infer_request = compiled_model.create_infer_request();
infer_request.set_tensor(param_input_y->get_friendly_name(), nv12_blob.first);
infer_request.set_tensor(param_input_uv->get_friendly_name(), nv12_blob.second);
infer_request.start_async();
infer_request.wait();
```

--------------------------------

### Direct NV12 VAAPI Surface Consumption on Linux with GPU Plugin

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Demonstrates configuring OpenVINO preprocessing for NV12 format with memory type set to surface, sharing a VAAPI display context with the GPU plugin, and consuming decoded VAAPI surfaces directly for inference.

```cpp
// ...

    using namespace ov::preprocess;
    auto p = PrePostProcessor(model);
    p.input().tensor().set_element_type(ov::element::u8)
                      .set_color_format(ov::preprocess::ColorFormat::NV12_TWO_PLANES, {"y", "uv"})
                      .set_memory_type(ov::intel_gpu::memory_type::surface);
    p.input().preprocess().convert_color(ov::preprocess::ColorFormat::BGR);
    p.input().model().set_layout("NCHW");
    model = p.build();

    VADisplay disp = get_va_display();
    // create the shared context object
    auto shared_va_context = ov::intel_gpu::ocl::VAContext(core, disp);
    // compile model within a shared context
    auto compiled_model = core.compile_model(model, shared_va_context);

    auto input0 = model->get_parameters().at(0);
    auto input1 = model->get_parameters().at(1);

    auto shape = input0->get_shape();
    auto width = shape[1];
    auto height = shape[2];

    // execute decoding and obtain decoded surface handle
    VASurfaceID va_surface = decode_va_surface();
    //     ...
    //wrap decoder output into RemoteBlobs and set it as inference input
    auto nv12_blob = shared_va_context.create_tensor_nv12(height, width, va_surface);

    auto infer_request = compiled_model.create_infer_request();
    infer_request.set_tensor(input0->get_friendly_name(), nv12_blob.first);
    infer_request.set_tensor(input1->get_friendly_name(), nv12_blob.second);
    infer_request.start_async();
    infer_request.wait();


```

--------------------------------

### Create USM Host Tensor (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Requests the GPU plugin to allocate USM host memory and wraps it into an ov::intel_gpu::ocl::USMTensor.

```cpp
    ov::intel_gpu::ocl::USMTensor remote_tensor = gpu_context.create_usm_host_tensor(in_element_type, in_shape);
    // Extract raw usm pointer from remote tensor
    void* usm_ptr = remote_tensor.get();

```

--------------------------------

### Text-to-Image Generation with Python

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Use `openvino_genai.Text2ImagePipeline` for text-to-image models. Specify the model directory and prompt. Adjust inference device and parameters as needed.

```Python
import argparse

import openvino_genai
from PIL import Image


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('model_dir')
    parser.add_argument('prompt')
    args = parser.parse_args()

    device = 'CPU'  # GPU can be used as well
    pipe = openvino_genai.Text2ImagePipeline(args.model_dir, device)

    image_tensor = pipe.generate(
        args.prompt,
        width=512,
        height=512,
        num_inference_steps=20,
        num_images_per_prompt=1)

    image = Image.fromarray(image_tensor.data[0])
    image.save("image.bmp")

```

--------------------------------

### Python: Compare OpenVINO GenAI with Hugging Face Results

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Compare text generation results from OpenVINO GenAI with a Hugging Face model. This snippet loads both models, generates text, and asserts equality.

```Python
from transformers import AutoTokenizer, AutoModelForCausalLM
import openvino_genai as ov_genai

tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

max_new_tokens = 32
prompt = 'table is made of'

encoded_prompt = tokenizer.encode(prompt, return_tensors='pt', add_special_tokens=False)
hf_encoded_output = model.generate(encoded_prompt, max_new_tokens=max_new_tokens, do_sample=False)
hf_output = tokenizer.decode(hf_encoded_output[0, encoded_prompt.shape[1]:])
print(f'hf_output: {hf_output}')

pipe = ov_genai.LLMPipeline('TinyLlama-1.1B-Chat-v1.0')
ov_output = pipe.generate(prompt, max_new_tokens=max_new_tokens)
print(f'ov_output: {ov_output}')

assert hf_output == ov_output


```

--------------------------------

### Set Multiple Batches for NV12 Grey Input

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Demonstrates setting multiple batches for a single Y-plane input, effectively processing multiple grayscale frames. Uses the OpenVINO C++ API.

```cpp
    cl::Image2D img_y_plane_0, img_y_plane_l;
    auto input_y = model_with_preproc->input(0);
    auto remote_y_tensor_0 = remote_context.create_tensor(input_y.get_element_type(), input.get_shape(), img_y_plane_0);
    auto remote_y_tensor_1 = remote_context.create_tensor(input_y.get_element_type(), input.get_shape(), img_y_plane_l);
    std::vector<ov::Tensor> y_tensors = {remote_y_tensor_0, remote_y_tensor_1};
    infer_request.set_tensors(input_y.get_any_name(), y_tensors);
    infer_request.infer();
```

--------------------------------

### Export Pre-compressed Hugging Face Models

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Exports pre-compressed models from Hugging Face, such as GPTQ or OpenVINO-optimized models, directly for OpenVINO inference.

```bash
optimum-cli export openvino -m TheBloke/Llama-2-7B-Chat-GPTQ
```

```bash
optimum-cli export openvino -m OpenVINO/Mistral-7B-Instruct-v0.2-int4-cw-ov
```

--------------------------------

### C CMake Configuration

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference.html

Configures a C project using CMake to find and link against the OpenVINO C runtime library.

```cmake
cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 17)

find_package(OpenVINO REQUIRED)

add_executable(${TARGET_NAME_C} src/main.c)

target_link_libraries(${TARGET_NAME_C} PRIVATE openvino::runtime::c)


```

--------------------------------

### Create USM Device Tensor (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Requests the GPU plugin to allocate USM device memory and wraps it into an ov::RemoteTensor.

```cpp
    auto remote_tensor = gpu_context.create_usm_device_tensor(in_element_type, in_shape);
    // Extract raw usm pointer from remote tensor
    void* usm_ptr = remote_tensor.get();

```

--------------------------------

### Text-to-Image Generation with LoRA Adapters (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Applies multiple LoRA adapters simultaneously for text-to-image generation. Demonstrates generating an image with and without adapters.

```cpp
#include "audio_utils.hpp"
#include "openvino/genai/whisper_pipeline.hpp"

int main(int argc, char* argv[]) try {
    if (3 > argc) {
        throw std::runtime_error(std::string{"Usage: "} + argv[0] + " <MODEL_DIR> \"<WAV_FILE_PATH>\"");
    }

    std::filesystem::path models_path = argv[1];
    std::string wav_file_path = argv[2];
    std::string device = "CPU";  // GPU or NPU can be used as well.

    ov::genai::WhisperPipeline pipeline(models_path, device);

    ov::genai::WhisperGenerationConfig config(models_path / "generation_config.json");
    config.max_new_tokens = 100;
    config.language = "<|en|>";
    config.task = "transcribe";
    config.return_timestamps = true;

    // The pipeline expects normalized audio with a sampling rate of 16kHz.
    ov::genai::RawSpeechInput raw_speech = utils::audio::read_wav(wav_file_path);
    auto result = pipeline.generate(raw_speech, config);

    std::cout << result << "\n";

    for (auto& chunk : *result.chunks) {
        std::cout << "timestamps: [" << chunk.start_ts << ", " << chunk.end_ts << "] text: " << chunk.text << "\n";
    }

} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {
    }
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {
    }
    return EXIT_FAILURE;
}

```

--------------------------------

### Python: Inference with GGUF Model

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Instantiate an LLMPipeline with a GGUF model file and perform text generation. Ensure the GGUF model is compatible with OpenVINO GenAI.

```Python
import openvino_genai

pipe = openvino_genai.LLMPipeline("SmolLM2-135M.F16.gguf", "CPU")

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100

pipe.generate("The Sun is yellow because", config)

```

--------------------------------

### VLM Configuration with Device Properties (Python)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Configure VLM inference on NPU with specific device properties like MAX_PROMPT_LEN and MIN_RESPONSE_LEN using this Python snippet.

```Python
import numpy as np
from PIL import Image
from openvino import Tensor
import openvino_genai as ov_genai

model_path = "Phi-4-multimodal-instruct"
image_path = "cat.png"
image = Tensor(np.array(Image.open(image_path).convert("RGB")))
pipeline_config = {
   "DEVICE_PROPERTIES": {
      "NPU": {
         "MAX_PROMPT_LEN": 2048,
         "MIN_RESPONSE_LEN": 512
      },
   }
}

pipe = ov_genai.VLMPipeline(model_path, "NPU", config=pipeline_config)
print(pipe.generate("Describe the image",  images=image, max_new_tokens=100))

```

--------------------------------

### Python Speculative Decoding with OpenVINO GenAI

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Demonstrates how to set up and use speculative decoding in Python. Requires importing openvino_genai, queue, and threading. The streamer function handles output, and the infer function initializes the draft and main models, configures the scheduler and generation parameters, and initiates the generation process.

```python
import openvino_genai
import queue
import threading

def streamer(subword):
        print(subword, end='', flush=True)
        return False

def infer(model_dir: str, draft_model_dir: str, prompt: str):
    main_device = 'CPU'  # GPU can be used as well.
    draft_device = 'CPU'

    scheduler_config = openvino_genai.SchedulerConfig()
    scheduler_config.cache_size = 2

    draft_model = openvino_genai.draft_model(draft_model_dir, draft_device)

    pipe = openvino_genai.LLMPipeline(model_dir, main_device, scheduler_config=scheduler_config, draft_model=draft_model)

    config = openvino_genai.GenerationConfig()
    config.max_new_tokens = 100
    config.num_assistant_tokens = 5

    pipe.generate("The Sun is yellow because", config, streamer)

```

--------------------------------

### Python: Image Loading and Preprocessing

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Loads images from a directory or a single file, converts them to RGB format, and prepares them as OpenVINO Tensors for the VLM pipeline. Handles both single images and directories of images.

```Python
from PIL import Image
from openvino import Tensor
from pathlib import Path
import numpy as np

def read_image(path: str) -> Tensor:
    pic = Image.open(path).convert("RGB")
    image_data = np.array(pic.getdata()).reshape(1, pic.size[1], pic.size[0], 3).astype(np.uint8)
    return Tensor(image_data)

def read_images(path: str) -> list[Tensor]:
    entry = Path(path)
    if entry.is_dir():
        return [read_image(str(file)) for file in sorted(entry.iterdir())]
    return [read_image(path)]
```

--------------------------------

### Text-to-Image Generation with C++

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Use `ov::genai::Text2ImagePipeline` in C++ for text-to-image models. Specify model directory and prompt. Images are saved with a pattern name.

```C++
#include "openvino/genai/image_generation/text2image_pipeline.hpp"

#include "imwrite.hpp"

int32_t main(int32_t argc, char* argv[]) try {
    OPENVINO_ASSERT(argc == 3, "Usage: ", argv[0], " <MODEL_DIR> '<PROMPT>'");

    const std::string models_path = argv[1], prompt = argv[2];
    const std::string device = "CPU";  // GPU can be used as well

    ov::genai::Text2ImagePipeline pipe(models_path, device);
    ov::Tensor image = pipe.generate(
        prompt,
        ov::genai::width(512),
        ov::genai::height(512),
        ov::genai::num_inference_steps(20),
        ov::genai::num_images_per_prompt(1));

    // writes `num_images_per_prompt` images by pattern name
    imwrite("image_%d.bmp", image, true);

    return EXIT_SUCCESS;
} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
}


```

--------------------------------

### Python: GenAI Inference Pipeline

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Initializes and uses the OpenVINO GenAI VLM pipeline for text generation based on prompts and images. Supports streaming output and interactive chat.

```Python
import openvino_genai

def streamer(subword: str) -> bool:
    print(subword, end='', flush=True)

def infer(model_dir: str, image_dir: str):
    rgbs = read_images(image_dir)
    device = 'CPU'  # GPU can be used as well.
    enable_compile_cache = dict()
    if "GPU" == device:
        enable_compile_cache["CACHE_DIR"] = "vlm_cache"
    pipe = openvino_genai.VLMPipeline(model_dir, device, **enable_compile_cache)

    config = openvino_genai.GenerationConfig()
    config.max_new_tokens = 100

    pipe.start_chat()
    prompt = input('question:\n')
    pipe.generate(prompt, images=rgbs, generation_config=config, streamer=streamer)

    while True:
        try:
            prompt = input("\n----------\n" \
                "question:\n")
        except EOFError:
            break
        pipe.generate(prompt, generation_config=config, streamer=streamer)
    pipe.finish_chat()
```

--------------------------------

### Share OpenCL Context with GPU Plugin for Inference

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Shows how to compile an OpenVINO model using a user-supplied OpenCL context, enabling shared context execution between the GPU plugin and custom OpenCL processing.

```cpp
    cl::Context ctx = get_ocl_context();

    ov::Core core;
    auto model = core.read_model("model.xml");

    // share the context with GPU plugin and compile ExecutableNetwork
    auto remote_context = ov::intel_gpu::ocl::ClContext(core, ctx.get());
    auto exec_net_shared = core.compile_model(model, remote_context);
    auto inf_req_shared = exec_net_shared.create_infer_request();


    // ...
    // do OpenCL processing stuff
    // ...

    // run the inference
    inf_req_shared.infer();


```

--------------------------------

### Weightless Blob Import with BLOB_PATH and WEIGHTS_PATH (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Import a weightless blob by specifying the `BLOB_PATH` and the `WEIGHTS_PATH` to the original model weights.

```C++
ov::AnyMap pipeline_config = { { "BLOB_PATH", ".npucache\compiled_model.blob" }, { "WEIGHTS_PATH", "path\to\original\model.bin" } };
ov::genai::LLMPipeline pipe(model_path, "NPU", pipeline_config);
```

--------------------------------

### Text-to-Image Generation with LoRA Adapters in C++

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Apply LoRA adapters to text-to-image generation in C++ using `ov::genai::Text2ImagePipeline`. Command-line arguments are used to specify model path, prompt, and LoRA adapters with their alpha values.

```C++
#include "openvino/genai/image_generation/text2image_pipeline.hpp"

#include "imwrite.hpp"

int32_t main(int32_t argc, char* argv[]) try {
    OPENVINO_ASSERT(argc >= 3 && (argc - 3) % 2 == 0, "Usage: ", argv[0], " <MODEL_DIR> '<PROMPT>' [<LORA_SAFETENSORS> <ALPHA> ...]]");

    const std::string models_path = argv[1], prompt = argv[2];
    const std::string device = "CPU";  // GPU, NPU can be used as well


```

--------------------------------

### Text-to-Image Generation with LoRA Adapters in Python

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Apply LoRA adapters to text-to-image generation using `openvino_genai.Text2ImagePipeline`. Multiple adapters can be applied simultaneously. Adapters can be passed during pipeline initialization or generation.

```Python
import openvino as ov
import openvino_genai

def image_write(path: str, image_tensor: ov.Tensor):
    from PIL import Image
    image = Image.fromarray(image_tensor.data[0])
    image.save(path)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('models_path')
    parser.add_argument('prompt')
    args, adapters = parser.parse_known_args()

    prompt = args.prompt

    device = "CPU"  # GPU, NPU can be used as well
    adapter_config = openvino_genai.AdapterConfig()

    # Multiple LoRA adapters applied simultaneously are supported, parse them all and corresponding alphas from cmd parameters:
    for i in range(int(len(adapters) / 2)):
        adapter = openvino_genai.Adapter(adapters[2 * i])
        alpha = float(adapters[2 * i + 1])
        adapter_config.add(adapter, alpha)

    # LoRA adapters passed to the constructor will be activated by default in next generates
    pipe = openvino_genai.Text2ImagePipeline(args.models_path, device, adapters=adapter_config)

    print("Generating image with LoRA adapters applied, resulting image will be in lora.bmp")
    image = pipe.generate(prompt,
                          width=512,
                          height=896,
                          num_inference_steps=20,
                          rng_seed=42)

    image_write("lora.bmp", image)
    print("Generating image without LoRA adapters applied, resulting image will be in baseline.bmp")
    image = pipe.generate(prompt,
                          # passing adapters in generate overrides adapters set in the constructor; openvino_genai.AdapterConfig() means no adapters
                          adapters=openvino_genai.AdapterConfig(),
                          width=512,
                          height=896,
                          num_inference_steps=20,
                          rng_seed=42)
    image_write("baseline.bmp", image)


```

--------------------------------

### C++: Image Loading and Preprocessing

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai.html

Loads images from a directory or a single file using a utility function, preparing them as OpenVINO Tensors for the VLM pipeline. Assumes `load_image.hpp` provides the `utils::load_images` function.

```C++
#include "load_image.hpp"
#include <openvino/genai/visual_language/pipeline.hpp>
#include <filesystem>

bool print_subword(std::string&& subword) {
    return !(std::cout << subword << std::flush);
}

int main(int argc, char* argv[]) try {
    if (3 != argc) {
        throw std::runtime_error(std::string{"Usage "} + argv[0] + " <MODEL_DIR> <IMAGE_FILE OR DIR_WITH_IMAGES>");
    }

    std::vector<ov::Tensor> rgbs = utils::load_images(argv[2]);

    std::string device = "CPU";  // GPU can be used as well.
    ov::AnyMap enable_compile_cache;
    if ("GPU" == device) {
        enable_compile_cache.insert({ov::cache_dir("vlm_cache")});
    }
    ov::genai::VLMPipeline pipe(argv[1], device, enable_compile_cache);

    ov::genai::GenerationConfig generation_config;
    generation_config.max_new_tokens = 100;

    std::string prompt;

    pipe.start_chat();
    std::cout << "question:\n";

    std::getline(std::cin, prompt);
    pipe.generate(prompt,
                  ov::genai::images(rgbs),
                  ov::genai::generation_config(generation_config),
                  ov::genai::streamer(print_subword));
    std::cout << "\n----------\n"
        "question:\n";
    while (std::getline(std::cin, prompt)) {
        pipe.generate(prompt,
                      ov::genai::generation_config(generation_config),
                      ov::genai::streamer(print_subword));
        std::cout << "\n----------\n"
            "question:\n";
    }
    pipe.finish_chat();
} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
}
```

--------------------------------

### Configure Generation Hint for NPU LLM Pipeline (Python)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Sets the GENERATE_HINT to 'BEST_PERF' for optimizing the text generation stage in the NPU LLM pipeline. Default is 'FAST_COMPILE'.

```Python
pipeline_config = { "GENERATE_HINT": "BEST_PERF" }
pipe = ov_genai.LLMPipeline(model_path, "NPU", pipeline_config)
```

--------------------------------

### Set Multiple Batches for Two-Plane NV12 Input

Source: https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/remote-tensor-api-gpu-plugin.html

Illustrates setting multiple batches of two-plane NV12 inputs using the OpenVINO C++ API. Requires separate tensors for Y and UV planes for each batch.

```cpp
    auto input0 = model_with_preproc->get_parameters().at(0);
    auto input1 = model_with_preproc->get_parameters().at(1);
    std::vector<ov::Tensor> y_tensors = {y_tensor_0, y_tensor_1};
    std::vector<ov::Tensor> uv_tensors = {uv_tensor_0, uv_tensor_1};
    infer_request.set_tensors(input0->get_friendly_name(), y_tensors);
    infer_request.set_tensors(input1->get_friendly_name(), uv_tensors);
    infer_request.infer();
```

--------------------------------

### Full-Weight Blob Import (C++)

Source: https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html

Import a full-weight blob by specifying the `BLOB_PATH`. The weights are included within the blob itself.

```C++
ov::AnyMap pipeline_config = { { "BLOB_PATH",  ".npucache\compiled_model.blob" } };
ov::genai::LLMPipeline pipe(model_path, "NPU", pipeline_config);
```
Tokens:
Raw