VPI - Vision Programming Interface

https://docs.nvidia.com/vpi/
Admin
NVIDIA Vision Programming Interface (VPI) is a software library that implements computer vision and...
Tokens:574,450
Snippets:6,260
Trust Score:10
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install VPI

Source: https://docs.nvidia.com/vpi/install_in_docker.html

Installs the main VPI runtime, development files, and sample applications.

```bash
apt-get install libnvvpi4 vpi4-dev vpi4-samples pva-allow-2
```

--------------------------------

### C/C++ Usage Example

Source: https://docs.nvidia.com/vpi/algo_rescale.html

Step-by-step guide for using the vpiSubmitRescale function in C/C++ including initialization, processing, and cleanup phases.

```APIDOC
## C/C++ Usage Example

### Description
Demonstrates the complete workflow for using the `vpiSubmitRescale` function in C/C++, covering image and stream creation, algorithm submission, and resource cleanup.

### Initialization Phase
1. **Include Header:**
   ```c
   #include <vpi/algo/Rescale.h>
   ```
2. **Define Input Image:**
   ```c
   VPIImage input = /*...*/;
   ```
3. **Create Output Image:**
   ```c
   int32_t w, h;
   vpiImageGetSize(input, &w, &h);
   VPIImageFormat type;
   vpiImageGetFormat(input, &type);
   VPIImage output;
   vpiImageCreate(w * 2 / 3.0f, h * 3 / 2.0f, type, 0, &output);
   ```
4. **Create Stream:**
   ```c
   VPIStream stream;
   vpiStreamCreate(0, &stream);
   ```

### Processing Phase
1. **Submit Algorithm:**
   ```c
   vpiSubmitRescale(stream, VPI_BACKEND_CUDA, input, output, VPI_INTERP_LINEAR, VPI_BORDER_ZERO, 0);
   ```
2. **Wait for Completion (Optional):**
   ```c
   vpiStreamSync(stream);
   ```

### Cleanup Phase
1. **Destroy Resources:**
   ```c
   vpiStreamDestroy(stream);
   vpiImageDestroy(input);
   vpiImageDestroy(output);
   ```
```

--------------------------------

### Install VPI Host Packages

Source: https://docs.nvidia.com/vpi/installation.html

Install the VPI package and its essential dependencies, including development files and sample applications, on your host system.

```bash
sudo apt install libnvvpi4 vpi4-dev vpi4-samples
```

--------------------------------

### Install VPI Dependencies

Source: https://docs.nvidia.com/vpi/tutorial_c.html

Installs necessary C++ compiler, CMake, and OpenCV packages on Ubuntu.

```bash
sudo apt-get install g++ cmake libopencv-dev
```

--------------------------------

### C/C++ Usage Example

Source: https://docs.nvidia.com/vpi/algo_minmaxloc.html

Illustrates the C/C++ usage of the MinMaxLoc algorithm, covering initialization, payload creation, output array setup, stream creation, and algorithm submission.

```c
#include <vpi/algo/MinMaxLoc.h>
#include <vpi/Image.h>
#include <vpi/Array.h>
#include <vpi/Stream.h>
#include <vpi/Status.h>

// ... assume VPIImage input is already created ...

// Initialization phase
// Include the header that defines the needed functions.

// Define the input image object.
// VPIImage input = /*...*/;

// Create the payload that will contain all temporary buffers needed for processing.
// Payload creation requires input's dimensions and format.
VPIImageFormat format;
vpiImageGetFormat(input, &format);
int width, height;
vpiImageGetSize(input, &width, &height);
VPIPayload payload;
vpiCreateMinMaxLoc(VPI_BACKEND_CPU, width, height, format, &payload);

// Create the output arrays for coordinates with minimum and maximum values.
// In this example, their capacity it set to 10,000, which limits the number of locations to be found.
VPIArray minCoords;
vpiArrayCreate(10000, VPI_ARRAY_TYPE_KEYPOINT_F32, 0, &minCoords);
VPIArray maxCoords;
vpiArrayCreate(10000, VPI_ARRAY_TYPE_KEYPOINT_F32, 0, &maxCoords);

// Create the stream where the algorithm will be submitted for execution.
VPIStream stream;
vpiStreamCreate(0, &stream);

// Processing phase
// Submit the algorithm to the stream, passing the input image and output arrays.
// In this example, it will be executed by the CPU backend.
vpiSubmitMinMaxLoc(stream, VPI_BACKEND_CPU, payload, input, minCoords, maxCoords);

// Optionally, retrieve the minimum and maximum values.
// Synchronize the stream to make sure the operation is completed
vpiStreamSync(stream);

// Lock the input image and the minimum/maximum output arrays.
VPIImageData inputImageData;
vpiImageLockData(input, VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &inputImageData);
VPIArrayData minCoordsData, maxCoordsData;
vpiArrayLockData(minCoords, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &minCoordsData);
vpiArrayLockData(maxCoords, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &maxCoordsData);

// ... process data ...

// Unlock data and destroy resources when done
// vpiImageUnlockData(input);
// vpiArrayUnlockData(minCoords);
// vpiArrayUnlockData(maxCoords);
// vpiPayloadDestroy(payload);
// vpiArrayDestroy(minCoords);
// vpiArrayDestroy(maxCoords);
// vpiStreamDestroy(stream);

```

--------------------------------

### Install apt Repository Utilities

Source: https://docs.nvidia.com/vpi/installation.html

Install software-properties-common, which provides scripts for managing repositories. This is necessary for adding new apt repositories.

```bash
sudo apt install software-properties-common
```

--------------------------------

### Install APT Packages

Source: https://docs.nvidia.com/vpi/install_in_docker.html

Installs essential packages required for adding new APT repositories and managing software.

```bash
apt-get update
apt-get install ca-certificates curl gnupg software-properties-common
```

--------------------------------

### Install VPI Python 3.10 Bindings

Source: https://docs.nvidia.com/vpi/installation.html

Install the VPI Python 3.10 bindings. Note that the numpy module must still be installed separately.

```bash
sudo apt install python3.10-vpi4
```

--------------------------------

### Install VPI Dependencies

Source: https://docs.nvidia.com/vpi/install_in_docker.html

Installs the core VPI dependencies, including NPP, cuFFT, CUDA runtime, and EGL.

```bash
apt-get update
apt-get install libnpp-13-2 libcufft-13-2 cuda-cudart-13-2 libegl1-mesa
```

--------------------------------

### C/C++ Usage Example

Source: https://docs.nvidia.com/vpi/algo_histogram.html

Example demonstrating the initialization, processing, and cleanup phases for computing an image histogram using the C API.

```APIDOC
```c
#include <vpi/algo/Histogram.h>

// Initialization phase
VPIImage input = /* ... */;
VPIArray output;
vpiArrayCreate(numBins, VPI_ARRAY_TYPE_U32, 0, &output);
VPIPayload payload;
vpiCreateHistogramEven(VPI_BACKEND_CUDA, VPI_IMAGE_FORMAT_U8, start, end, numBins, &payload);
VPIStream stream;
vpiStreamCreate(0, &stream);

// Processing phase
vpiSubmitHistogram(stream, VPI_BACKEND_CUDA, payload, input, output, 0);
vpiStreamSync(stream); // Optional: wait for completion

// Cleanup phase
vpiStreamDestroy(stream);
vpiImageDestroy(input);
vpiArrayDestroy(output);
vpiPayloadDestroy(payload);
```
```

--------------------------------

### Rescale Application Command Line Example

Source: https://docs.nvidia.com/vpi/sample_rescale.html

Demonstrates how to run the VPI Rescale application using different backends and input images. Ensure the input image respects the algorithm's constraints.

```bash
./vpi_sample_04_rescale cuda ../assets/kodim08.png
```

```bash
python3 main.py cuda ../assets/kodim08.png
```

--------------------------------

### Install VPI Python 3.12 Bindings

Source: https://docs.nvidia.com/vpi/installation.html

Install the VPI Python 3.12 bindings. Note that the numpy module must still be installed separately.

```bash
sudo apt install python3.12-vpi4
```

--------------------------------

### Main Function Setup

Source: https://docs.nvidia.com/vpi/sample_klt_tracker.html

Initializes OpenCV matrices for template and reference images. These are defined early to ensure they persist throughout the tracking process.

```cpp
int main(int argc, char *argv[])
{
 cv::Mat cvTemplate, cvReference;

```

--------------------------------

### Running the Image View Sample (C++)

Source: https://docs.nvidia.com/vpi/sample_image_view.html

Execute the C++ sample application by providing the input image file path as a command-line argument.

```bash
./vpi_sample_15_image_view ../assets/kodim08.png
```

--------------------------------

### VPI Resource Initialization and Cleanup

Source: https://docs.nvidia.com/vpi/sample_perspwarp.html

This snippet shows the setup and teardown of VPI resources including streams and images. It's crucial for managing memory and ensuring proper execution of VPI operations. Ensure the backend is correctly selected based on command-line arguments.

```cpp
// OpenCV image that will be wrapped by a VPIImage.
// Define it here so that it's destroyed *after* wrapper is destroyed
cv::Mat cvFrame;

// VPI objects that will be used
VPIStream stream = NULL;
VPIImage imgInput = NULL, imgOutput = NULL;
VPIImage frameBGR = NULL;

int retval = 0;

try
{
// =============================
// Parse command line parameters

if (argc != 3)
{
throw std::runtime_error(std::string("Usage: ") + argv[0] + " <cpu|vic|cuda> <input_video>");
}

std::string strBackend = argv[1];
std::string strInputVideo = argv[2];

// Now parse the backend
VPIBackend backend;

if (strBackend == "cpu")
{
backend = VPI_BACKEND_CPU;
}
else if (strBackend == "cuda")
{
backend = VPI_BACKEND_CUDA;
}
else if (strBackend == "vic")
{
backend = VPI_BACKEND_VIC;
}
else
{
throw std::runtime_error("Backend '" + strBackend + "' not recognized, it must be either cpu, cuda or vic.");
}

// ===============================
// Prepare input and output videos

// Load the input video
cv::VideoCapture invid;
if (!invid.open(strInputVideo))
{
throw std::runtime_error("Can't open '" + strInputVideo + "'");
}

// Open the output video for writing using input's characteristics
int w = invid.get(cv::CAP_PROP_FRAME_WIDTH);
int h = invid.get(cv::CAP_PROP_FRAME_HEIGHT);
int fourcc = cv::VideoWriter::fourcc('M', 'P', 'E', 'G');
double fps = invid.get(cv::CAP_PROP_FPS);

cv::VideoWriter outVideo("perspwarp_" + strBackend + ".mp4", fourcc, fps, cv::Size(w, h));
if (!outVideo.isOpened())
{
throw std::runtime_error("Can't create output video");
}

// =================================
// Allocate all VPI resources needed

// Create the stream for the given backend. We'll be using CUDA for image format conversion.
CHECK_STATUS(vpiStreamCreate(backend | VPI_BACKEND_CUDA, &stream));

CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgInput));
CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgOutput));

VPIPerspectiveTransform xform;
memset(&xform, 0, sizeof(xform));

// =========================
// Destroy all VPI resources

vpiStreamDestroy(stream);
vpiImageDestroy(imgInput);
vpiImageDestroy(imgOutput);
vpiImageDestroy(frameBGR);

return retval;
}
catch (std::exception &e)
{
std::cerr << e.what() << std::endl;
retval = 1;
}

```

--------------------------------

### Main Function Setup and Calibration

Source: https://docs.nvidia.com/vpi/sample_fisheye.html

Initializes VPI objects, loads the first image to determine dimensions, and sets up pre-defined fisheye distortion coefficients and camera intrinsic matrix. Outputs the calibration values being used.

```cpp
int main(int argc, char *argv[])
{
    // OpenCV image that will be wrapped by a VPIImage.
    // Define it here so that it's destroyed *after* wrapper is destroyed
    cv::Mat cvImage;

    // VPI objects that will be used
    VPIStream stream = NULL;
    VPIPayload remap = NULL;
    VPIImage tmpIn = NULL, tmpOut = NULL;
    VPIImage vimg = nullptr;

    int retval = 0;

    try
    {
        // First parse command line paramers
        Params params = ParseParameters(argc, argv);
        if (params.images.empty()) // user just wanted the help message?
        {
            return 0;
        }

        // Store image size from first image
        cv::Size imgSize = {};
        cv::Mat firstImg = cv::imread(params.images[0]);
        if (firstImg.empty())
        {
            throw std::runtime_error("Can't read " + std::string(params.images[0]));
        }
        imgSize = firstImg.size();

        // Pre-defined calibration values
        using Mat3 = cv::Matx<double, 3, 3>;
        Mat3 camMatrix = Mat3::eye();
        std::vector<double> coeffs(4);

        // Set fisheye coefficients
        coeffs[0] = -0.073234;
        coeffs[1] = 0.121251;
        coeffs[2] = -0.120230;
        coeffs[3] = 0.038480;

        // Set camera matrix
        camMatrix(0, 0) = 440.956750;
        camMatrix(0, 2) = 659.822931;
        camMatrix(1, 1) = 439.322113;
        camMatrix(1, 2) = 451.685233;
        camMatrix(2, 2) = 1.0;

        // Output calibration values being used
        printf("Using pre-defined calibration values:\n");
        printf("Fisheye coefficients: %lf %lf %lf %lf\n", coeffs[0], coeffs[1], coeffs[2], coeffs[3]);
        printf("Camera matrix:\n");
        printf("[%lf %lf %lf; %lf %lf %lf; %lf %lf %lf]\n", camMatrix(0, 0), camMatrix(0, 1), camMatrix(0, 2),
               camMatrix(1, 0), camMatrix(1, 1), camMatrix(1, 2), camMatrix(2, 0), camMatrix(2, 1), camMatrix(2, 2));

        // Now use VPI to undistort the input images:
        // Allocate a dense map.
        VPIWarpMap map = {};
        map.grid.numHorizRegions = 1;
        map.grid.numVertRegions = 1;
        map.grid.regionWidth[0] = imgSize.width;
        map.grid.regionHeight[0] = imgSize.height;
        map.grid.horizInterval[0] = 1;
        map.grid.vertInterval[0] = 1;
        CHECK_STATUS(vpiWarpMapAllocData(&map));

        // Initialize the fisheye lens model with the coefficients given by calibration procedure.
        VPIFisheyeLensDistortionModel distModel = {};
        distModel.mapping = VPI_FISHEYE_EQUIDISTANT;
        distModel.k1 = coeffs[0];
        distModel.k2 = coeffs[1];
        distModel.k3 = coeffs[2];
        distModel.k4 = coeffs[3];

        // Fill up the camera intrinsic parameters given by camera calibration procedure.
        VPICameraIntrinsic K;
        for (int i = 0; i < 2; ++i)
        {
            for (int j = 0; j < 3; ++j)
            {
                K[i][j] = camMatrix(i, j);
            }
        }

        // Camera extrinsics is be identity.
        VPICameraExtrinsic X = {};
        X[0][0] = X[1][1] = X[2][2] = 1;

        // Generate a warp map to undistort an image taken from fisheye lens with
        // given parameters calculated above.
        vpiWarpMapGenerateFromFisheyeLensDistortionModel(K, X, K, &distModel, &map);

        // Create the Remap payload for undistortion given the map generated above.
        CHECK_STATUS(vpiCreateRemap(VPI_BACKEND_CUDA, &map, &remap));

        // Now that the remap payload is created, we can destroy the warp map.
        vpiWarpMapFreeData(&map);

        // Create a stream where operations will take place. We're using CUDA
        // processing.
        CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA, &stream));
        

```

--------------------------------

### Running the Image View Sample (Python)

Source: https://docs.nvidia.com/vpi/sample_image_view.html

Execute the Python sample application by providing the input image file path as a command-line argument.

```bash
python3 main.py ../assets/kodim08.png
```

--------------------------------

### Python Usage Example

Source: https://docs.nvidia.com/vpi/algo_template_matching.html

Example of how to use the template matching function in Python with the CUDA backend.

```APIDOC
## Python Usage

1. Import VPI module
```python
import vpi
```

2. Match the template image in the input image by using CUDA backend. It's using the normalized cross-correlation.
```python
with vpi.Backend.CUDA:
    output = vpi.templateMatching(input, templ)
```
```

--------------------------------

### Build the Application

Source: https://docs.nvidia.com/vpi/tutorial_c.html

Use CMake to configure the build and Make to compile the application. The executable 'vpi_blur' will be created in the current directory.

```bash
cmake .
make
```

--------------------------------

### Install Cross-Compilation Toolchain

Source: https://docs.nvidia.com/vpi/sample_cross_aarch64.html

Installs the AArch64 GNU cross-compilation toolchain if not already present. This is a prerequisite for cross-compiling.

```bash
apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
```

--------------------------------

### DCF Tracker Sample Command Line

Source: https://docs.nvidia.com/vpi/sample_dcf_tracker.html

Example of how to run the DCF Tracker sample application using the CUDA backend with specified input video and bounding box files.

```bash
./vpi_sample_19_dcf_tracker cuda ../assets/pedestrians.mp4 ../assets/pedestrians_bboxes.txt
```

--------------------------------

### C/C++ Dynamic Remap Initialization and Setup

Source: https://docs.nvidia.com/vpi/algo_dynamic_remap.html

Provides the C/C++ code for initializing and setting up the Dynamic Remap algorithm. This includes including the necessary header, creating source and destination images, and setting up a stream.

```c++
#include <vpi/algo/DynamicRemap.h>

VPIImage src = /*...*/;
VPIImageFormat srcFormat;
vpiImageGetFormat(src, &srcFormat);
int32_t srcSize[2];
vpiImageGetSize(src, &srcSize[0], &srcSize[1]);
int32_t dstSize[2] = {(int32_t)(srcSize[0] * .8), (int32_t)(srcSize[1] * .8)};
VPIImage dst;
vpiImageCreate(dstSize[0], dstSize[1], srcFormat, 0, &dst);

VPIStream stream;
vpiStreamCreate(0, &stream);
```

--------------------------------

### Run the Application

Source: https://docs.nvidia.com/vpi/tutorial_c.html

Execute the compiled 'vpi_blur' application by providing an image file name as an argument. Replace '<image file name>' with the actual path to your image.

```bash
./vpi_blur <image file name>
```

--------------------------------

### Python Usage Example

Source: https://docs.nvidia.com/vpi/algo_histogram.html

Example of how to calculate the histogram of an input VPI image using the CUDA backend in Python.

```APIDOC
```python
import vpi

# Assuming 'input' is a VPIImage object and start, end, numBins are defined
with vpi.Backend.CUDA:
    output = input.histogram(numBins, range=(start, end))
```
```

--------------------------------

### Python Usage Example

Source: https://docs.nvidia.com/vpi/algo_rescale.html

Example of how to use the rescale function in Python with the CUDA backend, linear interpolation, and zero border extension.

```APIDOC
## Python Rescale Example

### Description
Rescales a VPI image input so that the output has 66% of the original width and the height is increased by 50%, using the CUDA backend and linear interpolation.

### Usage
```python
import vpi

with vpi.Backend.CUDA:
    output = input.rescale((input.width*2//3, input.height*3//2), interp=vpi.Interp.LINEAR, border=vpi.Border.ZERO)
```

### Parameters
- `input`: The input VPI image.
- `output`: The output VPI image.
- `interp`: The interpolation method (e.g., `vpi.Interp.LINEAR`).
- `border`: The border extension mode (e.g., `vpi.Border.ZERO`).
```

--------------------------------

### C/C++ Usage Example for Optical Flow PyrLK

Source: https://docs.nvidia.com/vpi/algo_optflow_lk.html

Example demonstrating the usage of the Pyramidal LK Optical Flow algorithm in C/C++.

```APIDOC
### C/C++

1. **Initialization phase**
   * Include the header that defines the needed functions and structures.
     ```c
     #include <vpi/algo/OpticalFlowPyrLK.h>
     ```
     `OpticalFlowPyrLK.h` declares functions that implement the Pyramidal LK Optical Flow algorithm.

   * Create the stream where the algorithm will be submitted for execution.
     ```c
     VPIStream stream;
     vpiStreamCreate(0, &stream);
     ```
     `VPIStream` is a handle to a stream.
     `vpiStreamCreate` creates a stream instance.

   * Define the required images, pyramids and arrays.
     ```c
     VPIImage prevImage = /* previous frame */;
     VPIPyramid pyrPrevFrame = /* pyramid out of previous frame */;
     VPIPyramid pyrCurFrame = /* pyramid for current frame */;
     VPIArray arrPrevPts = /* array with previous frame's keypoints, type VPI_ARRAY_TYPE_KEYPOINT_F32 */;
     VPIArray arrCurPts = /* array with current frame's keypoints, type VPI_ARRAY_TYPE_KEYPOINT_F32 */;
     VPIArray arrStatus = /* array with keypoint tracking status, type VPI_ARRAY_TYPE_U8 */;
     VPIArray scores = /* array with keypoint scores, type VPI_ARRAY_TYPE_U8 */;
     ```
     `VPIArray` is a handle to an array.
     `VPIImage` is a handle to an image.
     `VPIPyramid` is a handle to an image pyramid.

   * Create the payload that will contain all temporary buffers needed for processing. Its parameters are taken from the input pyramid and image used.
     ```c
     int levels;
     vpiPyramidGetNumLevels(pyrPrevFrame, &levels);
     float scale;
     vpiPyramidGetScale(pyrPrevFrame, &scale);
     VPIImageFormat format;
     vpiImageGetFormat(prevImage, &format);
     int width, height;
     vpiImageGetSize(prevImage, &width, &height);
     VPIPayload optflow;
     vpiCreateOpticalFlowPyrLK(VPI_BACKEND_CUDA, width, height, format, levels, scale, &optflow);
     ```
     `VPIImageFormat` defines pre-defined image formats.
     `vpiImageGetFormat` gets the image format.
     `vpiImageGetSize` gets the image dimensions in pixels.
     `vpiCreateOpticalFlowPyrLK` creates payload for `vpiSubmitOpticalFlowPyrLK`.
     `VPIPayload` is a handle to an algorithm payload.
     `vpiPyramidGetNumLevels` gets the image pyramid level count.
     `vpiPyramidGetScale` returns the scale factor of the pyramid levels.
     `VPI_BACKEND_CUDA` specifies the CUDA backend.
```

--------------------------------

### Main Function for Optical Flow Sample

Source: https://docs.nvidia.com/vpi/sample_optflow_lk.html

Sets up VPI objects, parses command-line arguments for backend, input video, pyramid levels, and output file. It initializes the video capture and retrieves the first frame.

```cpp
int main(int argc, char *argv[])
{
// OpenCV image that will be wrapped by a VPIImage.
// Define it here so that it's destroyed *after* wrapper is destroyed
cv::Mat cvFrame;

// VPI objects that will be used
VPIStream stream = NULL;
VPIImage imgTempFrame = NULL;
VPIImage imgFrame = NULL;
VPIPyramid pyrPrevFrame = NULL, pyrCurFrame = NULL;
VPIArray prevFeatures = NULL, curFeatures = NULL, status = NULL;
VPIPayload optflow = NULL;
VPIArray scores = NULL;
VPIPayload harris = NULL;

int retval = 0;

try
{
// ============================
// Parse command line arguments

if (argc != 5)
{
throw std::runtime_error(std::string("Usage: ") + argv[0] +
" <cpu|cuda|pva> <input_video> <pyramid_levels> <output>");
}

std::string strBackend = argv[1];
std::string strInputVideo = argv[2];
int32_t pyrLevel = std::stoi(argv[3]);
std::string strOutputFiles = argv[4];

// Now parse the backend
VPIBackend backend;

if (strBackend == "cpu")
{
backend = VPI_BACKEND_CPU;
}
else if (strBackend == "cuda")
{
backend = VPI_BACKEND_CUDA;
}
else if (strBackend == "pva")
{
backend = VPI_BACKEND_PVA;
}
else
{
throw std::runtime_error("Backend '" + strBackend +
"' not recognized, it must be either cpu, cuda or pva.");
}

{
int ext = strOutputFiles.rfind('.');
strOutputFiles = strOutputFiles.substr(0, ext) + "_" + strBackend + strOutputFiles.substr(ext);
}

// ====================
// Load the input video
cv::VideoCapture invid;
if (!invid.open(strInputVideo))
{
throw std::runtime_error("Can't open '" + strInputVideo + "'");
}

// Fetch the first frame and wrap it into a VPIImage.
// The points to be tracked will be gathered from this frame later on.
if (!invid.read(cvFrame))
{
throw std::runtime_error("Can't retrieve first frame from '" + strInputVideo + "'");
}


```

--------------------------------

### Sample Code: Exception Handling and Exit

Source: https://docs.nvidia.com/vpi/sample_submit_cuda_host.html

Illustrates exception handling for VPI operations and proper program exit using std::_Exit for success.

```c
catch (std::exception &e)
{
std::cerr << e.what() << std::endl;
retval = 1;
}

vpiContextDestroy(context);

if (retval == 0)
{
// The Jetson CUDA/EGL driver stack can abort in process-global finalizers
// after this CUDA-runtime sample has already released its resources.
std::cout.flush();
std::cerr.flush();
std::_Exit(EXIT_SUCCESS);
}

return retval;
```

--------------------------------

### Python Usage Example for Optical Flow PyrLK

Source: https://docs.nvidia.com/vpi/algo_optflow_lk.html

Example demonstrating the usage of the Pyramidal LK Optical Flow algorithm in Python.

```APIDOC
## Usage

Language: C/C++ Python

### Python

1. **Import VPI module**
   ```python
   import vpi
   ```

2. **Initialization phase**
   * Create the Pyramidal Optical Flow LK object, feeding it the initial frame and the VPI array with the keypoints to track. The CUDA backend will be used to execute the algorithm.
     ```python
     with vpi.Backend.CUDA:
         optflow = vpi.OpticalFlowPyrLK(frame, curFeatures, 4)
     ```

3. **Processing phase**
   * Fetch a new frame from input video sequence into a VPI image.
     ```python
     while inVideo.read(input)[0]:
         # Feed this VPI image into the OptFlow object. It'll return the estimated keypoint positions in the passed frame, along with a vector that informs the keypoint state, i.e., whether it's being tracked or not.
         curFeatures, status = optflow(input)
     ```
```

--------------------------------

### Run Dense Optical Flow Sample (Python)

Source: https://docs.nvidia.com/vpi/sample_optflow_dense.html

Example command to run the dense optical flow sample using Python.

```bash
python3 main.py ofa ../assets/pedestrians.mp4 high 2
```

--------------------------------

### C/C++ Initialization Phase

Source: https://docs.nvidia.com/vpi/algo_template_matching.html

Steps to initialize the VPI Template Matching algorithm in C/C++.

```APIDOC
## C/C++ Initialization Phase

1. Include the header that defines the template matching function.
```c
#include <vpi/algo/TemplateMatching.h>
```

2. Define the input image object.
```c
VPIImage input = /*...*/;
```

3. Define the template image object.
```c
VPIImage templ = /*...*/;
```

4. Create an output image with the required size and format.
```c
int32_t srcW, srcH;
vpiImageGetSize(input, &srcW, &srcH);
int32_t templW, templH;
vpiImageGetSize(templ, &templW, &templH);
VPIImage output;
vpiImageCreate(srcW - templW + 1, srcH - templH + 1, VPI_IMAGE_FORMAT_F32, 0, &output);
```

5. Create the stream where the algorithm will be submitted for execution.
```c
VPIStream stream;
vpiStreamCreate(0, &stream);
```

6. Create the payload that will contain all temporary buffers needed for processing. It'll be created on the CUDA backend.
```c
VPIPayload payload;
vpiCreateTemplateMatching(VPI_BACKEND_CUDA, srcW, srcH, &payload);
```
```

--------------------------------

### Check NVIDIA Driver Installation

Source: https://docs.nvidia.com/vpi/install_in_docker.html

Verify that a functional NVIDIA CUDA driver is installed on the device. This is a prerequisite for using NVIDIA container toolkit and VPI.

```bash
nvidia-smi
```

--------------------------------

### Main function for VPI Interop Sample

Source: https://docs.nvidia.com/vpi/sample_lock_extract_interop.html

Parses command-line arguments to select an export mode (CUDA pointer, NvBufSurface, or NvSciBufObj) and demonstrates creating a VPI-owned image, then exporting it in the selected format. This sample highlights zero-copy extraction.

```C++
int main(int argc, char *argv[])
{
    ExportMode exportMode = ParseExportMode(argc, argv);

    std::cout
        << "This sample shows how to extract either a CUDA pointer, an NvBufSurface, or an NvSciBufObj\n"
        << "from a VPI-owned image. These extraction paths are zero-copy.\n"
        << "The image is created with all platform-supported backend flags so VPI can choose compatible memory.\n\n";

    VPIContext context = nullptr;
    CHECK_STATUS(vpiContextCreate(0, &context));
    CHECK_STATUS(vpiContextSetCurrent(context));

    uint64_t imageBackends = 0;
    CHECK_STATUS(vpiContextGetFlags(context, &imageBackends));
    imageBackends &= VPI_BACKEND_ALL;

    VPIImage image = nullptr;

    try
    {
        CHECK_STATUS(vpiImageCreate(320, 240, VPI_IMAGE_FORMAT_NV12_ER, imageBackends, &image));

```

--------------------------------

### Initialize KLT Feature Tracker

Source: https://docs.nvidia.com/vpi/sample_klt_tracker.html

Loads the first frame of the input video, converts it to grayscale, and prepares a VPI image for the selected backend, which serves as the initial template for the KLT tracker.

```python
validFrame, cvFrame = inVideo.read()
if not validFrame:
print("Error reading first input frame", file=sys.stderr)
exit(1)

cvGray, imgTemplate = convertFrameImage(cvFrame, backend)
```

--------------------------------

### Install pva-allow

Source: https://docs.nvidia.com/vpi/install_in_docker.html

Installs the 'pva-allow' package, which may be required for certain VPI functionalities related to PVA (presumably Programmable Vision Accelerator) on the target system.

```bash
sudo apt-get install pva-allow-2
```

--------------------------------

### Install VPI Cross-Compilation Packages

Source: https://docs.nvidia.com/vpi/installation.html

Optionally, install development packages for cross-compilation targeting the aarch64-l4t architecture. This is useful for developing applications on an x86_64 system for Jetson devices.

```bash
sudo apt install vpi4-cross-aarch64-l4t
```

--------------------------------

### Main Function for Optical Flow Sample

Source: https://docs.nvidia.com/vpi/sample_optflow_dense.html

Sets up VPI and OpenCV objects, parses command-line arguments for input video, quality, grid size, and number of levels, and initializes the video capture.

```cpp
int main(int argc, char *argv[])
{
 // OpenCV image that will be wrapped by a VPIImage.
 // Define it here so that it's destroyed *after* wrapper is destroyed
 cv::Mat cvPrevFrame, cvCurFrame;

 // VPI objects that will be used
 VPIStream stream = NULL;
 VPIImage imgPrevFrameLoad = NULL;
 VPIImage imgCurFrameLoad = NULL;
 VPIImage imgPrevFramePL = NULL;
 VPIImage imgPrevFrameTmp = NULL;
 VPIImage imgCurFramePL = NULL;
 VPIImage imgCurFrameTmp = NULL;
 VPIImage imgMotionVecBL = NULL;
 VPIImage imgMotionVecPL = NULL;

 VPIPyramid prevPyrTmp = NULL;
 VPIPyramid prevPyrBL = NULL;
 VPIPyramid curPyrTmp = NULL;
 VPIPyramid curPyrBL = NULL;

 VPIPayload payload = NULL;

 int retval = 0;

 try
 {
 if (argc != 6)
 {
 throw std::runtime_error(std::string("Usage: ") + argv[0] +
 " <ofa> <input_video> <low|medium|high> <gridsize> <numlevels>");
 }

 // Parse input parameters
 std::string strBackend = argv[1];
 std::string strInputVideo = argv[2];
 std::string strQuality = argv[3];
 std::string strGridSize = argv[4];
 std::string strNumLevels = argv[5];

 VPIOpticalFlowQuality quality;
 if (strQuality == "low")
 {
 quality = VPI_OPTICAL_FLOW_QUALITY_LOW;
 }
 else if (strQuality == "medium")
 {
 quality = VPI_OPTICAL_FLOW_QUALITY_MEDIUM;
 }
 else if (strQuality == "high")
 {
 quality = VPI_OPTICAL_FLOW_QUALITY_HIGH;
 }
 else
 {
 throw std::runtime_error("Unknown quality provided");
 }

 VPIBackend backend;
 if (strBackend == "ofa")
 {
 backend = VPI_BACKEND_OFA;
 }
 else
 {
 throw std::runtime_error("Backend '" + strBackend + "' not recognized, it must be ofa.");
 }

 char *endptr;
 int gridSize = strtol(strGridSize.c_str(), &endptr, 10);
 if (*endptr != '\0')
 {
 throw std::runtime_error("Syntax error parsing gridsize " + strGridSize);
 }

 int numLevels = strtol(strNumLevels.c_str(), &endptr, 10);
 if (*endptr != '\0')
 {
 throw std::runtime_error("Syntax error parsing numlevels " + strNumLevels);
 }

 // Load the input video
 cv::VideoCapture invid;
 if (!invid.open(strInputVideo))
 {
 
```

--------------------------------

### Optional: Install VPI Python Bindings and NumPy

Source: https://docs.nvidia.com/vpi/install_in_docker.html

Installs Python 3.12 bindings for VPI and the NumPy library, enabling VPI usage within Python scripts.

```bash
apt-get install python3.12-vpi4 python3-numpy
```

--------------------------------

### Python: Initialize KLTFeatureTracker

Source: https://docs.nvidia.com/vpi/algo_klt_tracker.html

Create a VPI KLTFeatureTracker object with an image template, input bounding boxes, and specify the backend (e.g., CUDA).

```python
klt = vpi.KLTFeatureTracker(imgTemplate, inBoxes, backend=vpi.Backend.CUDA)
```

--------------------------------

### Install NVIDIA Container Toolkit

Source: https://docs.nvidia.com/vpi/install_in_docker.html

Installs the NVIDIA container toolkit from the stable NVIDIA repository. This toolkit is essential for enabling GPU access within Docker containers.

```bash
sudo apt-get update
sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install nvidia-container-toolkit
```

--------------------------------

### Run Dense Optical Flow Sample (C++)

Source: https://docs.nvidia.com/vpi/sample_optflow_dense.html

Example command to run the dense optical flow sample using C++ on Jetson Orin or Thor.

```bash
./vpi_sample_13_optflow_dense ofa ../assets/pedestrians.mp4 high 1 3
```

--------------------------------

### Python Image Format Conversion Example

Source: https://docs.nvidia.com/vpi/algo_imageconv.html

Example demonstrating image format conversion from an input image to S16 format with scaling and offset using the VPI Python API.

```APIDOC
## Python Image Conversion

### Description
Converts an input VPI image to S16 format (signed 16-bit), mapping the pixel range from [0,255] to [−32768,32767].

### Usage
```python
import vpi

with vpi.Backend.CPU:
    output = input.convert(vpi.Format.S16, scale=257, offset=-32768)
```

### Parameters
- `input`: The input VPI image object.
- `vpi.Format.S16`: Specifies the target format as signed 16-bit integer.
- `scale`: Scaling factor (257 in this example).
- `offset`: Offset value (-32768 in this example).
```

--------------------------------

### out_estimations

Source: https://docs.nvidia.com/vpi/python/build/vpi.KLTFeatureTracker.html

Get current output estimations.

```APIDOC
## out_estimations

### Description
Get current output estimations.

### Method
out_estimations(self)
```

--------------------------------

### VPI Benchmark Main Function

Source: https://docs.nvidia.com/vpi/sample_benchmark.html

This is the main entry point for the VPI benchmark. It parses command-line arguments to select the backend (CPU, PVA, or CUDA), initializes VPI objects, performs benchmarked operations, collects timing data, and cleans up resources. Use this code to measure the performance of VPI algorithms on your system.

```C++
#include <vpi/Event.h>
#include <vpi/Image.h>
#include <vpi/Status.h>
#include <vpi/Stream.h>
#include <vpi/algo/GaussianFilter.h>

#include <algorithm>
#include <cstdio>
#include <iostream>
#include <sstream>
#include <vector>

#define CHECK_STATUS(STMT) \
do \
{\
VPIStatus status = (STMT); \
if (status != VPI_SUCCESS) \
{\
char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
std::ostringstream ss; \
ss << "" #STMT "\n"; \
ss << vpiStatusGetName(status) << ": " << buffer; \
throw std::runtime_error(ss.str()); \
}\
} while (0)

int main(int argc, char *argv[])
{
    VPIImage image = NULL;
    VPIImage blurred = NULL;
    VPIStream stream = NULL;

    VPIEvent evStart = NULL;
    VPIEvent evStop = NULL;

    int retval = 0;

    try
    {
        // 1. Processing of command line parameters -----------

        if (argc != 2)
        {
            throw std::runtime_error(std::string("Usage: ") + argv[0] + " <cpu|pva|cuda>");
        }

        std::string strBackend = argv[1];

        // Parse the backend
        VPIBackend backend;

        if (strBackend == "cpu")
        {
            backend = VPI_BACKEND_CPU;
        }
        else if (strBackend == "cuda")
        {
            backend = VPI_BACKEND_CUDA;
        }
        else if (strBackend == "pva")
        {
            backend = VPI_BACKEND_PVA;
        }
        else
        {
            throw std::runtime_error("Backend '" + strBackend + "' not recognized, it must be either cpu, cuda or pva.");
        }

        // 2. Initialization stage ----------------------

        // Create the stream for only the target backend.
        uint64_t streamFlags = (uint64_t)backend | VPI_REQUIRE_BACKENDS;
        CHECK_STATUS(vpiStreamCreate(streamFlags, &stream));

        int width = 1920, height = 1080;
        VPIImageFormat imgFormat = VPI_IMAGE_FORMAT_U16;

        std::cout << "Input size: " << width << " x " << height << '\n'
                  << "Image format: " << vpiImageFormatGetName(imgFormat) << '\n'
                  << "Algorithm: 5x5 Gaussian Filter on " << strBackend << std::endl;

        // Memory flags set to guarantee top performance.
        // Only the benchmarked backend is enabled, and memories
        // are guaranteed to be used by only one stream.
        uint64_t memFlags = (uint64_t)backend | VPI_EXCLUSIVE_STREAM_ACCESS;

        // Create image with zero content
        CHECK_STATUS(vpiImageCreate(width, height, imgFormat, memFlags, &image));

        // Create a temporary image convolved with a low-pass filter.
        CHECK_STATUS(vpiImageCreate(width, height, imgFormat, memFlags, &blurred));

        // Create the events we'll need to get timing info
        CHECK_STATUS(vpiEventCreate(backend, &evStart));
        CHECK_STATUS(vpiEventCreate(backend, &evStop));

        // 3. Gather timings --------------------

        const int BATCH_COUNT = 20;
        const int AVERAGING_COUNT = 50;

        // Collect measurements for each execution batch
        std::vector<float> timingsMS;
        for (int batch = 0; batch < BATCH_COUNT; ++batch)
        {
            // Record stream queue when we start processing
            CHECK_STATUS(vpiEventRecord(evStart, stream));

            // Get the average running time within this batch.
            for (int i = 0; i < AVERAGING_COUNT; ++i)
            {
                // Call the algorithm to be measured.
                CHECK_STATUS(vpiSubmitGaussianFilter(stream, backend, image, blurred, 5, 5, 1, 1, VPI_BORDER_ZERO));
            }

            // Record stream queue just after blurring
            CHECK_STATUS(vpiEventRecord(evStop, stream));

            // Wait until the batch processing is done
            CHECK_STATUS(vpiEventSync(evStop));

            float elapsedMS;
            CHECK_STATUS(vpiEventElapsedTimeMillis(evStart, evStop, &elapsedMS));
            timingsMS.push_back(elapsedMS / AVERAGING_COUNT);
        }

        // 4. Performance analysis ----------------------

        // Get the median of the measurements so that outliers aren't considered.
        nth_element(timingsMS.begin(), timingsMS.begin() + timingsMS.size() / 2, timingsMS.end());
        float medianMS = timingsMS[timingsMS.size() / 2];

        printf("Approximated elapsed time per call on %s: %f ms\n", strBackend.c_str(), medianMS);
    }
    catch (std::exception &e)
    {
        std::cerr << e.what() << std::endl;
        retval = 1;
    }

    // 4. Clean up -----------------------------------

    // Destroy stream first, it'll make sure all processing
    // submitted to it is finished.
    vpiStreamDestroy(stream);

    // Now we can destroy other VPI objects, since they aren't being
    // used anymore.
    vpiImageDestroy(image);
    vpiImageDestroy(blurred);
    vpiEventDestroy(evStart);
    vpiEventDestroy(evStop);

    return retval;
}

```
Tokens:
Raw