OpenVINO GenAI

https://openvinotoolkit.github.io/openvino.genai/
Admin
OpenVINO GenAI provides optimized pipelines for running generative AI models including text...
Tokens:100,854
Snippets:930
Trust Score:9.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies for Model Conversion

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/text_generation

Install the necessary dependencies, including upgrading optimum-intel, to convert models for OpenVINO.

```bash
pip install --upgrade-strategy eager -r ../../export-requirements.txt  
```

--------------------------------

### Install Node.js dependencies

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/js/speech_generation

Install project dependencies for the Node.js sample application.

```bash
npm install
```

--------------------------------

### Quick Installation with Pip

Source: https://openvinotoolkit.github.io/openvino.genai

Use this command to quickly install OpenVINO™ GenAI from PyPI. Ensure you have pip installed.

```bash
python -m pip install openvino-genai
```

--------------------------------

### Install Timm and Einops for InternVL2

Source: https://openvinotoolkit.github.io/openvino.genai/docs/supported-models

Install the necessary libraries for converting InternVL2 models.

```bash
pip install timm einops
```

--------------------------------

### Example Command for Text-to-Image Benchmarking

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/image_generation

An example command to run the text-to-image pipeline benchmark. Specify the pipeline type, model path, number of iterations, and target device.

```bash
./benchmark_image_gen -t text2image -m dreamlike_anime_1_0_ov/FP16 -n 10 -d CPU
```

--------------------------------

### Install Dependencies for Running Samples

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/python/text_generation

Installs the necessary Python packages required to run the OpenVINO GenAI samples. This includes dependencies for deployment.

```bash
pip install --upgrade-strategy eager -r ../../deployment-requirements.txt  
```

--------------------------------

### Install huggingface_hub Package

Source: https://openvinotoolkit.github.io/openvino.genai/docs/guides/model-preparation/download-openvino-models

Install the `huggingface_hub` package to facilitate model downloads from Hugging Face. This is a prerequisite for the download command.

```bash
pip install huggingface_hub
```

--------------------------------

### Install Dependencies and Export Model

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/image_generation

Installs necessary dependencies and exports a Stable Diffusion model to OpenVINO format using optimum-cli.

```bash
pip install --upgrade-strategy eager -r ../../requirements.txt
```

```bash
optimum-cli export openvino --model dreamlike-art/dreamlike-anime-1.0 --task stable-diffusion --weight-format fp16 dreamlike_anime_1_0_ov/FP16
```

--------------------------------

### Install Dependencies and Export Model

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/visual_language_chat

Install necessary Python packages and export the specified model to OpenVINO format. Ensure optimum-intel is up-to-date.

```bash
pip install --upgrade-strategy eager -r ../../requirements.txt

optimum-cli export openvino --model openbmb/MiniCPM-V-2_6 --trust-remote-code MiniCPM-V-2_6
```

--------------------------------

### Image-to-Image Pipeline Setup

Source: https://openvinotoolkit.github.io/openvino.genai/assets/files/benchmark_image_gen-846f3ddc89d66a25a00f038e35477d67.cpp

Initializes an Image2Image pipeline, loading an input image and preparing for generation. Supports optional reshaping based on input image dimensions.

```cpp
void image2image(cxxopts::ParseResult& result) {
    std::string prompt = result["prompt"].as<std::string>();
    const std::string models_path = result["model"].as<std::string>();
    std::string image_path = result["image"].as<std::string>();
    auto devices = device_string_to_triplet(result["device"].as<std::string>());
    size_t num_warmup = result["num_warmup"].as<size_t>();
    size_t num_iter = result["num_iter"].as<size_t>();
    const std::string output_dir = result["output_dir"].as<std::string>();
    float strength = result["strength"].as<float>();

    ov::Tensor image_input = utils::load_image(image_path);

    ov::genai::Image2ImagePipeline pipe(models_path);
    if (result["reshape"].as<bool>()) {
        auto height = image_input.get_shape()[1];
        auto width = image_input.get_shape()[2];

```

--------------------------------

### Example Command for Encrypted Stable Diffusion

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/image_generation

An example of how to run the encrypted stable diffusion sample with specific model path, prompt, and default device.

```bash
./encrypted_stable_diffusion ./dreamlike_anime_1_0_ov/FP16 'cyberpunk cityscape like Tokyo New York with tall buildings at dusk golden hour cinematic lighting'
```

--------------------------------

### Install JavaScript Dependencies

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/js/image_generation

Installs Node.js dependencies for the JavaScript sample.

```bash
npm install  
```

--------------------------------

### Install Dependencies and Export Model

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/js/whisper_speech_recognition

Install necessary Python packages and export the Whisper model to OpenVINO format. Ensure optimum-intel is up-to-date.

```bash
pip install --upgrade-strategy eager -r <GENAI_ROOT_DIR>/samples/requirements.txt  

optimum-cli export openvino --trust-remote-code --model openai/whisper-base whisper-base  
```

--------------------------------

### Install ModelScope and optimum-intel Dependencies

Source: https://openvinotoolkit.github.io/openvino.genai/docs/guides/model-preparation/convert-to-openvino

Install the modelscope and optimum-intel packages required for downloading and converting models from ModelScope.

```bash
pip install modelscope --requirement ./samples/export-requirements.txt
```

--------------------------------

### Image-to-Image Generation Pipeline Setup

Source: https://openvinotoolkit.github.io/openvino.genai/assets/files/image2image_concurrency-9d625e543241e9f0ca023dbd664a75cf.cpp

Initializes the Image2ImagePipeline, loads an input image, and prepares prompts for generation. Handles command-line arguments for model path, prompts, and the input image.

```cpp
// Copyright (C) 2023-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0

#include <vector>
#include <string>

#include "openvino/genai/image_generation/image2image_pipeline.hpp"

#include "imwrite.hpp"
#include "load_image.hpp"
#include "progress_bar.hpp"

int32_t main(int32_t argc, char* argv[]) try {
    OPENVINO_ASSERT(argc >= 4, "Usage: ", argv[0], " <MODEL_DIR> '<PROMPT>' '<PROMPT>' ... <IMAGE>");

    const std::string models_path = argv[1];
    const std::string device = "CPU";  // GPU and NPU can be used as well
    const std::string image_path = argv[argc - 1];
    ov::Tensor image = utils::load_image(image_path);

    std::vector<std::thread> threads;
    std::vector<std::string> prompts;
    std::vector<ov::genai::Image2ImagePipeline> pipelines;

    for (int32_t i = 2; i < argc - 1; ++i)
        prompts.push_back(argv[i]);

    ov::AnyMap properties;
    if (device == "NPU") {
        // Define static shape and guidance scale for NPU
        const int num_images_per_prompt = 1;
        const int height = 512;
        const int width = 512;
        const float guidance_scale = 7.5f;

        pipelines.emplace_back(models_path);
        pipelines.back().reshape(num_images_per_prompt, height, width, guidance_scale);
        pipelines.back().compile(device);  // All models are compiled for NPU
        //pipelines.back().compile("NPU", "NPU", "GPU");  // Compile for NPU and GPU, if needed

        // Don't specify N, H, W, and guidance_scale in the properties map because they were made static
        properties = ov::AnyMap{ov::genai::strength(0.8f),
                                ov::genai::num_inference_steps(4)};
    } else {
        pipelines.emplace_back(models_path, device);

        properties = ov::AnyMap{ov::genai::strength(0.8f),  // controls how initial image is noised after being converted to latent space. `1` means initial image is fully noised
                                ov::genai::num_inference_steps(4)};
    }

    // Clone pipeline for concurrent usage
    for (size_t i = 1; i < prompts.size(); ++i)
        pipelines.emplace_back(pipelines.begin()->clone());

    for (size_t i = 0; i < prompts.size(); ++i) {
        std::string prompt = prompts[i];
        auto& pipe = pipelines.at(i);

        std::cout << "Starting to generate with prompt: '" << prompt << "'..." << std::endl;

        threads.emplace_back([i, &pipe, prompt, image, &properties] () {

            ov::Tensor generated_image = pipe.generate(prompt, image, properties);

            // writes `num_images_per_prompt` images by pattern name
            imwrite("image_" + std::to_string(i) + "_%d.bmp", generated_image, true);
        });
    }

    for (auto& thread : threads) {
        thread.join();
    }

    return EXIT_SUCCESS;
} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
}

```

--------------------------------

### Install optimum-intel Dependencies

Source: https://openvinotoolkit.github.io/openvino.genai/docs/guides/model-preparation/convert-to-openvino

Install the necessary dependencies for downloading, converting, and optimizing models using optimum-cli. Some models may require additional packages.

```bash
pip install --requirement ./samples/export-requirements.txt
```

--------------------------------

### Install dependencies for model export

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/js/speech_generation

Install necessary packages for exporting models, ensuring optimum-intel is up-to-date.

```bash
pip install --upgrade-strategy eager -r <GENAI_ROOT_DIR>/samples/export-requirements.txt
```

--------------------------------

### Install Dependencies and Export Model

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/whisper_speech_recognition

Install necessary Python packages and export the Whisper model to OpenVINO IR format using optimum-cli.

```bash
pip install --upgrade-strategy eager -r ../../export-requirements.txt  

optimum-cli export openvino --trust-remote-code --model openai/whisper-tiny whisper-tiny  
```

--------------------------------

### C++ Chat Example with Grouped Beam Search

Source: https://openvinotoolkit.github.io/openvino.genai/docs/guides/chat-scenario

This C++ example demonstrates initiating a chat session and configuring generation parameters for grouped beam search. It reads prompts from standard input and prints answers to standard output, managing the chat context with `start_chat()` and `finish_chat()`.

```cpp
#include "openvino/genai/llm_pipeline.hpp"

#include <iostream>

int main(int argc, char* argv[]) {
    std::string prompt;

    std::string model_path = argv[1];
    ov::genai::LLMPipeline pipe(model_path, "CPU");

    ov::genai::GenerationConfig config;
    config.max_new_tokens = 100;
    config.num_beam_groups = 3;
    config.num_beams = 15;
    config.diversity_penalty = 1.0f;

    pipe.start_chat();
    std::cout << "question:\n";
    while (std::getline(std::cin, prompt)) {
        std::cout << "answer:\n";
        auto answer = pipe.generate(prompt, config);
        std::cout << answer << std::endl;
        std::cout << "\n----------\n"
            "question:\n";
    }
    pipe.finish_chat();
}
```

--------------------------------

### Concurrent Text-to-Image Generation Example

Source: https://openvinotoolkit.github.io/openvino.genai/assets/files/text2image_concurrency-7e5bd0524d56eb198354bc984af3c768.cpp

This C++ example demonstrates concurrent image generation using the Text2Image pipeline. It prepares pipelines, compiles models for a specified device, and launches multiple threads to generate images from different prompts simultaneously. Ensure the model directory and prompts are provided as command-line arguments.

```cpp
#include <iostream>

#include "openvino/genai/image_generation/text2image_pipeline.hpp"


#include "imwrite.hpp"
#include "progress_bar.hpp"

int32_t main(int32_t argc, char* argv[]) try {
    OPENVINO_ASSERT(argc >= 3, "Usage: ", argv[0], " <MODEL_DIR> '<PROMPT>' '<PROMPT>' ...");

    const std::string models_path = argv[1];
    const std::string device = "CPU";  // GPU and NPU can be used as well

    std::vector<std::thread> threads;
    std::vector<std::string> prompts;
    std::vector<ov::genai::Text2ImagePipeline> pipelines;

    for (int i = 2; i < argc; ++i)
        prompts.push_back(argv[i]);

    // Prepare initial pipeline and compiled models into device
    ov::AnyMap properties;
    if (device == "NPU") {
        // Define static shape and guidance scale for NPU
        const int num_images_per_prompt = 1;
        const int height = 512;
        const int width = 512;
        const float guidance_scale = 7.5f;

        pipelines.emplace_back(models_path);
        pipelines.back().reshape(num_images_per_prompt, height, width, guidance_scale);
        pipelines.back().compile(device);  // All models are compiled for NPU
        // pipelines.back().compile("NPU", "NPU", "GPU");  // Compile for NPU and GPU, if needed

        // Don't specify N, H, W, and guidance_scale in the properties map because they were made static
        properties = ov::AnyMap{ov::genai::num_inference_steps(2)};
    } else {
        pipelines.emplace_back(models_path, device);

        properties = ov::AnyMap{ov::genai::width(512),
                                ov::genai::height(512),
                                ov::genai::num_inference_steps(2),
                                ov::genai::num_images_per_prompt(1)};
    }

    // Clone pipeline for concurrent usage
    for (size_t i = 1; i < prompts.size(); ++i)
        pipelines.emplace_back(pipelines.begin()->clone());

    for (size_t i = 0; i < prompts.size(); ++i) {
        std::string prompt = prompts[i];
        auto& pipe = pipelines.at(i);

        std::cout << "Starting to generate with prompt: '" << prompt << "'..." << std::endl;

        threads.emplace_back([i, &pipe, prompt, &properties] () {
            ov::Tensor image = pipe.generate(prompt, properties);

            imwrite("image_" + std::to_string(i) + "_%d.bmp", image, true);
        });
    }

    for (auto& thread : threads) {
        thread.join();
    }

    return EXIT_SUCCESS;
} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
}

```

--------------------------------

### Install OpenVINO GenAI Node.js Bindings via npm

Source: https://openvinotoolkit.github.io/openvino.genai/docs/getting-started/installation

Use this command to install the OpenVINO GenAI Node.js bindings from npm. Refer to the Node.js Bindings documentation for more information.

```bash
npm install openvino-genai-node
```

--------------------------------

### LLMPipeline Quick Start with Streaming

Source: https://openvinotoolkit.github.io/openvino.genai/docs/bindings/node-js

Demonstrates how to initialize and use the LLMPipeline for text generation with streaming output. Ensure your model is converted to OpenVINO IR format and imported correctly.

```javascript
import { LLMPipeline } from "openvino-genai-node";  

  
async function main() {  
  const modelPath = "/path/to/ov/model";  
  const device = "CPU";  
  const pipe = await LLMPipeline(modelPath, device);  

  
  const input = "What is OpenVINO?";  
  const config = { max_new_tokens: 100 };  

  
  for await (const chunk of pipe.stream(input, config)) {  
    process.stdout.write(chunk);  
  }  
}  


main();  

```

--------------------------------

### Install dependencies and export model

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/speech_generation

Install necessary packages and export the SpeechT5 TTS model with a vocoder using optimum-cli. Ensure optimum-intel is up-to-date.

```bash
pip install --upgrade-strategy eager -r ../../export-requirements.txt  

optimum-cli export openvino --model microsoft/speecht5_tts --model-kwargs "{\"vocoder\": \"microsoft/speecht5_hifigan\"}" speecht5_tts  
```

--------------------------------

### Text-to-Image Pipeline Setup and Generation

Source: https://openvinotoolkit.github.io/openvino.genai/assets/files/benchmark_image_gen-846f3ddc89d66a25a00f038e35477d67.cpp

Initializes and configures a Text2Image pipeline, including optional reshaping, compilation for specified devices, and generation of images with performance metric collection.

```cpp
void text2image(cxxopts::ParseResult& result) {
    std::string prompt = result["prompt"].as<std::string>();
    const std::string models_path = result["model"].as<std::string>();
    auto devices = device_string_to_triplet(result["device"].as<std::string>());
    size_t num_warmup = result["num_warmup"].as<size_t>();
    size_t num_iter = result["num_iter"].as<size_t>();
    const std::string output_dir = result["output_dir"].as<std::string>();

    ov::genai::Text2ImagePipeline pipe(models_path);
    if (result["reshape"].as<bool>()) {
        pipe.reshape(result["num_images_per_prompt"].as<size_t>(),
                     result["height"].as<size_t>(),
                     result["width"].as<size_t>(),
                     pipe.get_generation_config().guidance_scale);
    }
    pipe.compile(devices[0], devices[1], devices[2]);

    ov::genai::ImageGenerationConfig config = pipe.get_generation_config();
    config.width = result["width"].as<size_t>();
    config.height = result["height"].as<size_t>();
    config.num_inference_steps = result["num_inference_steps"].as<size_t>();
    config.num_images_per_prompt = result["num_images_per_prompt"].as<size_t>();
    pipe.set_generation_config(config);

    std::cout << std::fixed << std::setprecision(2);
    std::vector<ov::genai::ImageGenerationPerfMetrics> warmup_metrics;
    for (size_t i = 0; i < num_warmup; i++) {
        pipe.generate(prompt);
        ov::genai::ImageGenerationPerfMetrics metrics = pipe.get_performance_metrics();
        warmup_metrics.emplace_back(metrics);
        print_one_generate(metrics, "warmup", i);
    }

    std::vector<ov::genai::ImageGenerationPerfMetrics> iter_metrics;
    for (size_t i = 0; i < num_iter; i++) {
        ov::Tensor image = pipe.generate(prompt);
        ov::genai::ImageGenerationPerfMetrics metrics = pipe.get_performance_metrics();
        iter_metrics.emplace_back(metrics);
        std::string image_name = output_dir + "/image_" + std::to_string(i) + ".bmp";
        imwrite(image_name, image, true);
        print_one_generate(metrics, "iter", i);
    }

    print_statistic(warmup_metrics, iter_metrics);
}
```

--------------------------------

### Install OpenVINO GenAI via PyPI

Source: https://openvinotoolkit.github.io/openvino.genai/docs/getting-started/installation

Use this command to install the OpenVINO GenAI package from PyPI. Some models may require additional dependencies.

```bash
pip install openvino-genai
```

--------------------------------

### Install Deployment Requirements

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/text_generation

Installs packages required to run OpenVINO GenAI samples. This is necessary for using samples like chat, greedy causal LM, and beam search causal LM.

```bash
pip install --upgrade-strategy eager -r ../../deployment-requirements.txt
```

--------------------------------

### Example Usage of Whisper ASR C Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/whisper_speech_recognition

Demonstrates a typical command to run the Whisper ASR C sample with a specified model and audio file.

```bash
./whisper_speech_recognition_c whisper-tiny how_are_you_doing_today.wav  
```

--------------------------------

### Benchmark Output Metrics

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/visual_language_chat

Example output from the benchmark tool, detailing various performance metrics like load time, generation time, and throughput.

```text
Load time: 1982.00 ms  
Generate time: 13820.99 ± 64.62 ms  
Tokenization time: 1.26 ± 0.09 ms  
Detokenization time: 0.33 ± 0.05 ms  
Embeddings preparation time: 5733.85 ± 26.34 ms  
TTFT: 11246.98 ± 80.55 ms  
TPOT: 135.45 ± 4.73 ms/token  
Throughput: 7.38 ± 0.26 tokens/s  

```

--------------------------------

### Run Benchmark Command

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/visual_language_chat

Example command to run the benchmark tool for visual language models. Specify the model, an image, and the number of runs.

```bash
benchmark_vlm -m miniCPM-V_2_6 -i 319483352-d5fbbd1a-d484-415c-88cb-9986625b7b11.jpg -n 3  

```

--------------------------------

### Record Audio using recorder.py

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/python/whisper_speech_recognition

Runs the recorder.py script to capture 5 seconds of audio from the microphone. Ensure PyAudio is installed following its instructions.

```bash
python recorder.py
```

--------------------------------

### Main Function for Export/Import Examples

Source: https://openvinotoolkit.github.io/openvino.genai/assets/files/stable_diffusion_export_import-6296d69da0bda9efb046e34d39d0f8a0.cpp

Entry point for the program, demonstrating the usage of pipeline export/import, dedicated model export/import, and export/import with reshape functionalities. It takes model directory and a prompt as command-line arguments.

```cpp
int32_t main(int32_t argc, char* argv[]) try {
    OPENVINO_ASSERT(argc == 3, "Usage: ", argv[0], " <MODEL_DIR> '<PROMPT>'");

    const std::string models_path = argv[1], prompt = argv[2];

    std::filesystem::path root_dir = models_path;

    pipeline_export_import(root_dir);
    dedicated_models_export_import(root_dir);
    export_import_with_reshape(root_dir, prompt);

    return EXIT_SUCCESS;
} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {
    }
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {
    }
    return EXIT_FAILURE;
}

```

--------------------------------

### Build Samples for Linux and macOS

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/whisper_speech_recognition

Set up the environment and build the C samples on Linux and macOS systems.

```bash
source <INSTALL_DIR>/setupvars.sh  

./<INSTALL_DIR>/samples/c/build_samples.sh  
```

--------------------------------

### Timestamp Prediction for Transcription

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/whisper_speech_recognition

Enable timestamp prediction during transcription to get start and end times for each transcribed text segment. This is useful for aligning text with audio.

```cpp
ov::genai::RawSpeechInput raw_speech = read_wav("how_are_you_doing_today.wav");

auto result = pipeline.generate(raw_speech, ov::genai::return_timestamps(true));
  
std::cout << std::setprecision(2);
for (auto& chunk : *result.chunks) {
    std::cout << "timestamps: [" << chunk.start_ts << ", " << chunk.end_ts << "] text: " << chunk.text << "\n";
}
// timestamps: [0, 2] text:  How are you doing today?  
```

--------------------------------

### Build Samples for Windows Command Prompt

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/whisper_speech_recognition

Set up the environment and build the C samples using the Windows Command Prompt.

```batch
<INSTALL_DIR>\setupvars.bat  

<INSTALL_DIR>\samples\c\build_samples_msvc.bat  
```

--------------------------------

### Build OpenVINO GenAI Samples for Windows

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/text_generation

Set up the environment and build the C samples for Windows using either Command Prompt or PowerShell scripts.

```batch
<INSTALL_DIR>\setupvars.bat  

<INSTALL_DIR>\samples\c\build_samples_msvc.bat  
```

```powershell
.<INSTALL_DIR>\setupvars.ps1  

.<INSTALL_DIR>\samples\c\build_samples.ps1  
```

--------------------------------

### Install Optimum Intel

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/rag

Installs or upgrades optimum-intel to the latest version. Ensure you have the necessary requirements by installing from the provided file.

```bash
pip install --upgrade-strategy eager -r ../../export-requirements.txt
```

--------------------------------

### Build Samples for Windows PowerShell

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/whisper_speech_recognition

Set up the environment and build the C samples using Windows PowerShell.

```powershell
.<INSTALL_DIR>\setupvars.ps1  

.<INSTALL_DIR>\samples\c\build_samples.ps1  
```

--------------------------------

### Run Chat Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/js/text_generation

Launches the interactive chat sample. Provide the path to the model directory as an argument.

```bash
node chat_sample.js model_dir
```

--------------------------------

### Run Chat Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/text_generation

Launches the interactive chat sample. Replace <MODEL_DIR> with the path to your model.

```bash
./chat_sample <MODEL_DIR>
```

--------------------------------

### Run Image-to-Text Chat Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/visual_language_chat

Execute the image-to-text chat sample with a specified model and image. Consider using a dGPU for better performance with larger models.

```bash
visual_language_chat miniCPM-V-2_6 319483352-d5fbbd1a-d484-415c-88cb-9986625b7b11.jpg
```

--------------------------------

### Text-to-Video Generation Pipeline Example

Source: https://openvinotoolkit.github.io/openvino.genai/assets/files/text2video-146e84902d17c9eaca4213bed288a1c7.cpp

This C++ code demonstrates how to initialize and use the OpenVINO Text2VideoPipeline to generate a video from a text prompt. It includes setting generation parameters like height, width, number of frames, and inference steps. Ensure the model directory and prompt are provided as command-line arguments.

```cpp
// Copyright (C) 2025-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0

#include <memory>
#include <string>
#include <random>
#include <filesystem>

#include "progress_bar.hpp"
#include "imwrite_video.hpp"

#include <openvino/genai/video_generation/text2video_pipeline.hpp>

int main(int32_t argc, char* argv[]) try {
    OPENVINO_ASSERT(argc == 3, "Usage: ", argv[0], " <MODEL_DIR> '<PROMPT>'");

    std::filesystem::path models_dir = argv[1];
    std::string prompt = argv[2];

    const std::string device = "CPU";  // GPU can be used as well
    float frame_rate = 25.0f;

    ov::genai::Text2VideoPipeline pipe(models_dir, device);
    auto output = pipe.generate(
        prompt,
        ov::genai::negative_prompt("worst quality, inconsistent motion, blurry, jittery, distorted"),
        ov::genai::height(480),
        ov::genai::width(704),
        ov::genai::num_frames(161),
        ov::genai::num_inference_steps(25),
        ov::genai::num_videos_per_prompt(1),
        ov::genai::callback(progress_bar),
        ov::genai::frame_rate(frame_rate),
        ov::genai::guidance_scale(3)
    );

    save_video("genai_video.avi", output.video, frame_rate);

    return EXIT_SUCCESS;
} catch (const std::exception& error) {
    try {
        std::cerr << error.what() << '\n';
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
} catch (...) {
    try {
        std::cerr << "Non-exception object thrown\n";
    } catch (const std::ios_base::failure&) {}
    return EXIT_FAILURE;
}

```

--------------------------------

### Install modelscope Package

Source: https://openvinotoolkit.github.io/openvino.genai/docs/guides/model-preparation/download-openvino-models

Install the `modelscope` package to enable downloading models from ModelScope. This package is required for the `modelscope download` command.

```bash
pip install modelscope
```

--------------------------------

### Initialize Pipeline with Multiple LoRA Adapters (C++)

Source: https://openvinotoolkit.github.io/openvino.genai/docs/guides/lora-adapters

Shows how to initialize an LLMPipeline with multiple LoRA adapters in C++, assigning weights and demonstrating how to switch adapter configurations for subsequent generation calls.

```cpp
#include "openvino/genai/llm_pipeline.hpp"  

int main() {  
    ov::genai::AdapterConfig adapter_config;  

    // Add multiple adapters with different weights  
    ov::genai::Adapter adapter1("path/to/lora1.safetensors");  
    ov::genai::Adapter adapter2("path/to/lora2.safetensors");  

    adapter_config.add(adapter1, 0.5f);  
    adapter_config.add(adapter2, 0.5f);  

    ov::genai::LLMPipeline pipe(  
        model_path,  
        "CPU",  
        ov::genai::adapters(adapter_config)  
    );  

    // Generate with current adapters  
    auto output1 = pipe.generate("Generate story about", ov::genai::max_new_tokens(100));  

    // Switch to different adapter configuration  
    ov::genai::AdapterConfig new_config;  
    new_config.add(adapter1, 1.0f);  
    auto output2 = pipe.generate(  
        "Generate story about",  
        ov::genai::adapters(new_config),  
        ov::genai::max_new_tokens(100)  
    );  
}
```

--------------------------------

### Run LLM ReAct Agent Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/js/text_generation

Launches the ReAct Agent sample for interactive reasoning and action. Provide the path to the model directory.

```bash
node react_sample.js model_dir
```

--------------------------------

### Install OpenCV

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/python/video_generation

Install a specific version of the OpenCV library. This is required to resolve potential issues when running samples that utilize OpenCV functionalities.

```bash
pip install opencv-python==4.12.0.88
```

--------------------------------

### Run Video-to-Text Chat Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/visual_language_chat

Execute the video-to-text chat sample using a model that supports video input and a sample video file. This showcases VLM capabilities with dynamic visual data.

```bash
video_to_text_chat ./LLaVA-NeXT-Video-7B-hf/ sample_demo_1.mp4
```

--------------------------------

### Example of Step-by-Step Equation Solving Output

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/text_generation

An example of structured output generated by the LLM for solving an equation step-by-step, adhering to a predefined JSON schema.

```json
{
  "steps": [
    {"explanation": "Rearranging the equation to isolate x.", "output": "8x + 7 = -23"},
    {"explanation": "Subtracting 7 from both sides.", "output": "8x + 7 - 7 = -23 - 7"},
    {"explanation": "Simplifying the left side.", "output": "8x = -30"},
    {"explanation": "Dividing both sides by 8.", "output": "8x / 8 = -30 / 8"},
    {"explanation": "Simplifying the right side.", "output": "x = -30 / 8"},
    {"explanation": "Finding the value of x.", "output": "x = -15/4"}
  ],
  "final_answer": "x = -15/4"
}
```

--------------------------------

### Run Chat Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/text_generation

Execute the chat sample to engage in multi-turn conversations with an interactive chat interface.

```bash
./chat_sample_c model_dir  
```

--------------------------------

### Run Basic Text to Video Generation

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/video_generation

Executes the text-to-video sample with a specified model directory and a text prompt. The output is a video file generated based on the prompt.

```bash
./text2video model_dir prompt  
```

```bash
./text2video ltx_video_ov/FP32 "A woman with long brown hair and light skin smiles at another woman with long blonde hair"  
```

--------------------------------

### Run with LoRA Adapter

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/cpp/image_generation

Demonstrates running the lora_text2image sample with a single LoRA adapter. Ensure the adapter file is downloaded and placed in the correct directory. The alpha parameter is interpreted as the final scaling weight.

```bash
./lora_text2image dreamlike_anime_1_0_ov/FP16 'curly-haired unicorn in the forest, anime, line' soulcard.safetensors 0.7
```

--------------------------------

### Run Text Rerank Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/python/rag

Execute the text_rerank.py script to rerank a list of documents based on a query. Ensure deployment-requirements.txt is installed.

```bash
python text_rerank.py <MODEL_DIR> "<QUERY>" "<TEXT 1>" ["<TEXT 2>"] ...
```

--------------------------------

### Run Text Embedding Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/python/rag

Execute the text_embeddings.py script to generate vector embeddings for input documents. Ensure deployment-requirements.txt is installed.

```bash
python text_embeddings.py <MODEL_DIR> "Document 1" "Document 2"
```

--------------------------------

### Run LLMs Benchmarking Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/c/text_generation

Benchmark LLMs in OpenVINO GenAI using the C language sample. Various options are available for customization.

```bash
./benchmark_genai_c  [-m MODEL] [-p PROMPT] [-nw NUM_WARMUP] [-n NUM_ITER] [-mt MAX_NEW_TOKENS] [-d DEVICE]  
```

--------------------------------

### Run TextRerankPipeline on CPU (Python)

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/text-rerank

Initialize the TextRerankPipeline for CPU execution. Specify the model path, device, and the number of top results to return.

```python
import openvino_genai  

pipeline = openvino_genai.TextRerankPipeline(model_path, "CPU", top_n=3)  

rerank_result = pipeline.rerank(query, documents)  

print("Reranked documents:")  
for index, score in rerank_result:  
  print(f"Document {index} (score: {score:.4f}): {documents[index]}")  
```

--------------------------------

### Text2ImagePipeline with GPU (C++)

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/image-generation

Generate an image from a text prompt using the Text2ImagePipeline on a GPU in C++. Ensure the C++ compatible package is installed.

```cpp
#include "openvino/genai/image_generation/text2image_pipeline.hpp"  

#include "imwrite.hpp"  


int main(int argc, char* argv[]) {  

  const std::string models_path = argv[1], prompt = argv[2];  

  

  ov::genai::Text2ImagePipeline pipe(models_path, "GPU");  

  ov::Tensor image = pipe.generate(prompt);  

  

  imwrite("image.bmp", image, true);  

}  

```

--------------------------------

### Text2ImagePipeline with CPU (C++)

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/image-generation

Generate an image from a text prompt using the Text2ImagePipeline on a CPU in C++. Ensure the C++ compatible package is installed.

```cpp
#include "openvino/genai/image_generation/text2image_pipeline.hpp"  

#include "imwrite.hpp"  


int main(int argc, char* argv[]) {  

  const std::string models_path = argv[1], prompt = argv[2];  

  

  ov::genai::Text2ImagePipeline pipe(models_path, "CPU");  

  ov::Tensor image = pipe.generate(prompt);  

  

  imwrite("image.bmp", image, true);  

}  

```

--------------------------------

### Word-Level Timestamps Prediction Setup in Python

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/speech-recognition

Configures the Whisper pipeline for word-level timestamp prediction. This requires setting 'word_timestamps=True' during pipeline initialization.

```python
# Word timestamps require decomposition of cross-attention decoder SDPA layers,  
# so word_timestamps must be passed to the pipeline constructor (not just in generation config)  
pipe = openvino_genai.WhisperPipeline(model_path, "CPU", word_timestamps=True)

# Enable word-level timestamp prediction
result = pipe.generate(raw_speech, word_timestamps=True)
```

--------------------------------

### Run Text to Image Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/python/image_generation

Executes the text-to-image generation sample script with a specified model directory and prompt.

```bash
python text2image.py ./dreamlike_anime_1_0_ov/FP16 "cyberpunk cityscape like Tokyo New York with tall buildings at dusk golden hour cinematic lighting"
```

--------------------------------

### Predict Timestamps for Speech Segments in C++

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/speech-recognition

Enables segment-level timestamp prediction in C++. The output includes start and end timestamps for each text chunk.

```cpp
ov::genai::WhisperPipeline pipe(model_path, "CPU");

    // Enable timestamp prediction
    result = pipe.generate(raw_speech, ov::genai::return_timestamps(true));

    // Print timestamps and text segments
    for (auto& chunk : *result.chunks) {
        std::cout << "timestamps: [" << chunk.start_ts << ", " << chunk.end_ts
                  << "] text: " << chunk.text << "\n";
    }
```

--------------------------------

### Run TextRerankPipeline on CPU (C++)

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/text-rerank

Initialize the TextRerankPipeline for CPU execution in C++. Pass the model path, device, and top_n configuration.

```cpp
#include "openvino/genai/rag/text_rerank_pipeline.hpp"  

int main(int argc, char* argv[]) {  
  std::vector<std::string> documents(argv + 3, argv + argc);  
  std::string models_path = argv[1], query = argv[2];  
  
  ov::genai::TextRerankPipeline pipeline(models_path, "CPU", ov::genai::top_n(3));  
  
  auto rerank_result = pipeline.rerank(query, documents);  
  
  std::cout << "Reranked documents:\n";  
  for (const auto& [index, score] : rerank_result) {  
      std::cout << "Document " << index << " (score: " << score << "): " << documents[index] << '\n';  
  }  
}  
```

--------------------------------

### Run Whisper Speech Recognition Sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/python/whisper_speech_recognition

Executes the Whisper speech recognition sample with a specified model directory and audio file. Requires deployment-requirements.txt to be installed.

```bash
python whisper_speech_recognition.py whisper-base how_are_you_doing_today.wav
```

--------------------------------

### Run Text-to-Speech sample

Source: https://openvinotoolkit.github.io/openvino.genai/docs/samples/js/speech_generation

Execute the Node.js sample to convert text to speech. Basic usage without optional arguments.

```bash
node speech_generation/text2speech.js speecht5_tts "Hello OpenVINO GenAI"
```

--------------------------------

### Generate Video from Text Prompt (C++, CPU)

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/video-generation

Generate a video from a text prompt using the C++ Text2VideoPipeline on the CPU. This example requires including OpenVINO GenAI headers and a helper function for saving videos.

```cpp
#include "openvino/genai/video_generation/text2video_pipeline.hpp"  

#include "imwrite_video.hpp"  

  
int main(int argc, char* argv[]) {  
  const std::string models_path = argv[1], prompt = argv[2];  
  
  ov::genai::Text2VideoPipeline pipe(models_path, "CPU");  
  ov::Tensor video = pipe.generate(prompt).video;  
  
  imwrite_video("genai_video.avi", video);  
}  
```

--------------------------------

### Run Whisper Inference with JavaScript on GPU

Source: https://openvinotoolkit.github.io/openvino.genai/docs/use-cases/speech-recognition

Perform speech recognition using the WhisperPipeline in JavaScript on a GPU. This example assumes a 'wav_utils.js' file for audio reading.

```javascript
import { WhisperPipeline } from 'openvino-genai-node';  

import { readAudio } from './wav_utils.js';  

const rawSpeech = readAudio('sample.wav');  

const pipeline = await WhisperPipeline(modelPath, "GPU");  
const generationConfig = { max_new_tokens: 100 };  
const result = await pipeline.generate(rawSpeech, { generationConfig });  
console.log(result.texts[0]);  
```

--------------------------------

### Initialize Pipeline with Multiple LoRA Adapters (Python)

Source: https://openvinotoolkit.github.io/openvino.genai/docs/guides/lora-adapters

Demonstrates initializing an LLMPipeline with multiple LoRA adapters, each assigned a specific weight (alpha). Adapters can be dynamically switched between generation calls by providing a new AdapterConfig.

```python
import openvino_genai as ov_genai  

# Initialize pipeline with adapters  
adapter_config = ov_genai.AdapterConfig()  

# Add multiple adapters with different weights  
adapter1 = ov_genai.Adapter("path/to/lora1.safetensors")  
adapter2 = ov_genai.Adapter("path/to/lora2.safetensors")  

adapter_config.add(adapter1, alpha=0.5)  
adapter_config.add(adapter2, alpha=0.5)  

pipe = ov_genai.LLMPipeline(  
    model_path,  
    "CPU",  
    adapters=adapter_config  
)  

# Generate with current adapters  
output1 = pipe.generate("Generate story about", max_new_tokens=100)  

# Switch to different adapter configuration  
new_config = ov_genai.AdapterConfig()  
new_config.add(adapter1, alpha=1.0)  
output2 = pipe.generate(  
    "Generate story about",  
    max_new_tokens=100,  
    adapters=new_config  
)
```
Tokens:
Raw