NVIDIA Optical Flow SDK

https://docs.nvidia.com/video-technologies/optical-flow-sdk/
Admin
NVIDIA Optical Flow SDK provides hardware-accelerated optical flow and stereo disparity estimation...
Tokens:30,273
Snippets:196
Trust Score:10
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Build and install NvFRUCSample (Linux)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Build the executable and install it into the ubuntu directory.

```bash
make && make install
```

--------------------------------

### Build and install samples on Linux (cmake, make, sudo make install)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Build and install the Optical Flow SDK basic samples on Linux in release mode. Run these commands in the 'build' directory inside NvOFBasicSamples; the binaries are copied to 'build/bin/x64'.

```bash
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=. ..
```

```bash
make
```

```bash
sudo make install
```

--------------------------------

### Build and install NvFRUCSample (Windows)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Build the project in release configuration and install it. This copies the executable and FreeImage.dll to the appropriate win32 or win64 directory.

```bash
cmake --build . --target install --config release
```

--------------------------------

### Run NvOFFRUCSample with YUV Input

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Example command line for processing a YUV420 video file. The input path must not contain whitespaces.

```bash
NvOFFRUCSample.exe 
    --input=C:\fruc\inputfile.yuv 
    --width=1920 
    --height=1080
    --output=C:\fruc\outputdir
```

--------------------------------

### NvOFFRUCSample Help Menu Parameters

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Run the application without parameters to view the help menu. This list details the mandatory and optional command-line arguments required for execution.

```text
Mandatory Parameters
    --input             Input name , Path must be Absolute
    --width             Input frame width
    --height            Input frame height
    --output            Output Directory to store results,
                        Path could be Absolute or Relative
Optional Parameters
    --surfaceformat     Surface format : Set 0 for NV12, 1 for ABGR
    --startframe        Start processing from this frame index
    --endframe          Process till this frame index
    --allocationtype    Specify 0 to create CUDA and 1 to create DX allocations
    --cudasurfacetype   Specify 0 to create cuDevicePtr and
                        1 for cuArray, please note this option takes effect 
                        only if client wants to use CUDA code path
```

--------------------------------

### Run NvOFFRUCSample with PNG Sequence Input

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Example command line for processing a sequence of PNG frames. The input path uses a wildcard pattern to identify the sequence.

```bash
NvOFFRUCSample.exe
    --input=C:\fruc\wonders\rainbow_*.png
    --width=1920
    --height=1080
    --output=C:\fruc\outputdir
```

--------------------------------

### Create build directory for NvFRUCSample (Windows)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Navigate to the NvFRUCSample directory and create a build directory. This is the first step in the Windows build process.

```bash
cd C:/Users/TestPC/Downloads/OpticalFlowSDK/NvFRUCSample && mkdir build && cd build
```

--------------------------------

### Navigate to NvOFFRUCSample binary directory (Windows 64-bit)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Change to the directory containing the 64-bit NvOFFRUCSample executable and its dependencies.

```bash
C:/Users/TestPC/Downloads/OpticalFlowSDK/NvOFFRUC/NvOFFRUCSample/bin/win64
```

--------------------------------

### Interpolate intermediate frame with NvFRUCProcess

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/nvfruc-programming-guide/index.html

Call NvFRUCProcess in a loop with consecutive input frames to get interpolated frames. Fill stInParams and stOutParams before each call. The first call returns the input frame itself; subsequent calls return interpolated frames at the requested timestamps. The function uses the current and cached previous frame.

```cpp
NvFRUC_PROCESS_IN_PARAMS stInParams = { 0 };
NvFRUC_PROCESS_OUT_PARAMS stOutParams = { 0 };

status = NvFRUCProcess(
               hFRUC,
               &stInParams,
               &stOutParams);
```

--------------------------------

### Create build directory for NvFRUCSample (Linux)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Navigate to the NvFRUCSample directory and create a build directory. This is the first step in the Linux build process.

```bash
cd /home/Downloads/OpticalFlowSDK/NvFRUCSample && mkdir build && cd build
```

--------------------------------

### Navigate to NvOFFRUCSample binary directory (Windows 32-bit)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Change to the directory containing the 32-bit NvOFFRUCSample executable and its dependencies.

```bash
C:/Users/TestPC/Downloads/OpticalFlowSDK/NvOFFRUC/NvOFFRUCSample/bin/win32
```

--------------------------------

### Build NvOFTSample and NvOFTracker with CMake

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Commands to configure and build the NvOFTSample and NvOFTracker projects. Replace opencvDir and videocodecsdkroot with actual paths.

```bash
cd /home/Downloads/OpticalFlowSDK/NvOFTracker && mkdir build && cd build
```

```bash
cmake -DOpenCV_DIR=opencvDir -DVIDEOCODEC_SDK_ROOT=videocodecsdkroot ..
```

```bash
cmake -DOpenCV_DIR=opencvDir -DVIDEOCODEC_SDK_ROOT=videocodecsdkroot -DTRT_ROOT=trtRoot ..
```

```bash
make install
```

--------------------------------

### Navigate to NvOFFRUCSample binary directory (Linux)

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Change to the directory containing the Linux NvOFFRUCSample executable and its dependencies.

```bash
/home/Downloads/OpticalFlowSDK/NvOFFRUC/NvOFFRUCSample/bin/ubuntu
```

--------------------------------

### Load libNvFRUC.so on Linux using dlopen

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/nvfruc-programming-guide/index.html

Loads the FRUC library on Linux using dlopen with RTLD_LAZY.

```cpp
hDLL = dlopen("libNvFRUC.so", RTLD_LAZY);
```

--------------------------------

### NvOFTSample help menu output

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Output of running NvOFTSample to display its help menu, showing mandatory and optional parameters.

```text
123456789

Mandatory Parameters
    -i           Input video file
    -e           TensorRT Engine file for Detector
    Optional Parameters:
    -o           Output video file
    -fT          Filename to dump tracked objects
    -dC          Dump tracked objects to Console
    -sI          Detection skip interval. Must be 0 or greater
    -g           GPU Id on which the tracker needs to run. Default is 0
```

--------------------------------

### Vulkan Interface Initialization

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/nvofa-programming-guide/index.html

This snippet documents the initialization steps for the Vulkan interface of the NVIDIA Optical Flow SDK, including the required extensions, features, and the parameters for NV_OF_INIT_PARAMS.

```APIDOC
## [Vulkan] Initialize Vulkan Interface for Optical Flow

### Description
Initializes the Vulkan interface for the NVIDIA Optical Flow SDK. This involves loading the optical flow module, creating a Vulkan instance, selecting a physical device, creating a logical device, and populating the API function pointer list.

### Steps
1. Load the optical flow module `nvofapi.dll`/`nvofapi64.dll`.
2. Initialize Vulkan by creating a `VkInstance` with the `VK_KHR_get_physical_device_properties2` extension enabled, and ensure that `VkApplicationInfo::apiVersion` is set to `VK_API_VERSION_1_3` or higher.
3. Select a physical device which supports `VK_KHR_timeline_semaphore` and `VK_NV_optical_flow` extensions.
4. Query the queue family which supports `VK_QUEUE_OPTICAL_FLOW_BIT_NV`.
5. Create a logical device which enables:
   - `VK_KHR_timeline_semaphore`, `VK_NV_optical_flow` extensions.
   - `VkPhysicalDeviceTimelineSemaphoreFeatures`, `VkPhysicalDeviceSynchronization2Features` and `VkPhysicalDeviceOpticalFlowFeaturesNV` features.
   - A device queue for optical flow family along with other device queues for graphics, compute and/or other tasks.
6. Retrieve the address of exported function `NvOFAPICreateInstanceVk` from the loaded optical flow module.
7. Populate API function pointer list:
   - Call `NvOFAPICreateInstanceVk` to populate Vulkan API function pointer list.
   - The API accepts `NV_OF_API_VERSION` and a pointer `NV_OF_VK_API_FUNCTION_LIST*` to a memory block of function pointer list `NV_OF_VK_API_FUNCTION_LIST` that receives the API addresses.
   - The function pointers enable the access to the optical flow functionality.
8. Create optical flow instance:
   - Call `nvCreateOpticalFlowVk` and pass the `VkInstance`, `VkPhysicalDevice`, `VkDevice` created in the previous step.
   - The API returns an `NvOFHandle` which should be preserved and used throughout the session.
9. Call `NvOFGetCaps` to query the capabilities supported by the NVIDIA display driver and the GPU.
10. Call `NvOFInit` after filling up `NV_OF_INIT_PARAMS`.

### Parameters for NV_OF_INIT_PARAMS
- **Image height** (uint32_t) - Required - The height of the image/video sequence for which the vectors will be evaluated.
- **Image width** (uint32_t) - Required - The width of the image/video sequence for which the vectors will be evaluated.
- **Flag to enable external hints** (bool) - Optional - NVOF API provides clients the flexibility for feeding external flow vectors which will be used as hints by NVOFA while performing the motion search.
- **Output grid size** (NV_OF_OUTPUT_GRID_SIZE) - Required - Specify `NV_OF_INIT_PARAMS::outGridSize`. The flow vectors will be generated for the specified grid size.
- **Hint grid size** (NV_OF_HINT_GRID_SIZE) - Optional - In case the client is using external hints, the hint grid size should be specified.
- **Preset** (NV_OF_PRESET_TYPE) - Required - The NVOF API exposes three presets to trade quality vs performance.
- **Operating mode** (NV_OF_MODE) - Required - The NVOFA can be used for generating optical flow vectors (which comprise X and Y components and is referred to as optical flow mode) or stereo vectors (which comprise only the X component and is referred to as stereo mode).
- **Enable output cost** (bool) - Optional - Set this flag to generate confidence associated with the flow vectors. Higher cost value implies the flow vector to be less accurate and vice-versa. It is recommended to use the 8-bit cost (by allocating a buffer of type `NV_OF_BUFFER_FORMAT_UINT8`) instead of 32-bit cost as that saves CUDA bandwidth.
- **Disparity Range** (uint32_t) - Optional - This is valid from Ampere and later GPUs and applicable in `NV_OF_MODE_STEREODISPARITY` mode. It specifies maximum stereo disparity range value.
- **Enable ROI** (bool) - Optional - Set the flag to estimate the flow for a rectangular region within the image. Available on Ampere and later GPUs.
- **Prediction direction** (NV_OF_PREDICTION_DIRECTION) - Required - Direction of the prediction. Supports either forward or forward-backward directions.
- **Enable Global Flow** (bool) - Optional - Global flow estimation is enabled when this flag is set.
- **Input buffer format** (NV_OF_BUFFER_FORMAT) - Required - `NV_OF_BUFFER_FORMAT` for the input buffer.
```

--------------------------------

### Run CMake to configure NvOFTracker build

Source: https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html

Configure the build with CMake, specifying paths to OpenCV, TensorRT, and Video Codec SDK. Replace opencvDir, trtRoot, and videocodecsdkroot with actual paths.

```bash
cmake -DOpenCV_DIR=opencvDir -DTRT_ROOT=trtRoot -DVIDEOCODEC_SDK_ROOT=videocodecsdkroot ..
```
Tokens:
Raw