NVIDIA TensorRT

https://docs.nvidia.com/deeplearning/tensorrt/latest/
Admin
NVIDIA TensorRT is an SDK for optimizing and accelerating deep learning inference on NVIDIA GPUs...
Tokens:1,649,640
Snippets:14,219
Trust Score:10
Update:13 hours ago
Show doc for...
Tokens:
Raw
### Clone TensorRT repository and navigate to quickstart directory

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/quick-start-runtime-tutorial.html

Downloads the source code for the quick start tutorial from the TensorRT Open Source Software repository and changes to the quickstart directory.

```bash
git clone https://github.com/NVIDIA/TensorRT.git
cd TensorRT/quickstart
```

--------------------------------

### Install development tools for sample compilation

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Install the 'Development Tools' group to resolve sample compilation failures during RPM installation.

```bash
sudo dnf groupinstall "Development Tools"
```

--------------------------------

### Install lean runtime only

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the lean runtime and its VC plugin.

```bash
sudo dnf install libnvinfer-lean11
sudo dnf install libnvinfer-vc-plugin11
```

--------------------------------

### Install build essentials for sample compilation

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Install build-essential and cmake to resolve sample compilation failures. This is required when samples fail to compile during TensorRT installation.

```bash
sudo apt-get install build-essential cmake
```

--------------------------------

### Install dispatch runtime packages

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the dispatch runtime and its VC plugin.

```bash
sudo apt-get install libnvinfer-dispatch11
sudo apt-get install libnvinfer-vc-plugin11
```

--------------------------------

### Build and run command for TensorRT end-to-end example

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/c-api-docs.html

This command compiles and runs the end-to-end example. It requires replacing the include and library paths with your TensorRT install location. On first invocation, the program builds and serializes model.engine, then runs one inference on a random tensor sized for ResNet-50.

```bash
g++ -std=c++17 trt_end_to_end.cpp \
    -I/usr/include/x86_64-linux-gnu -I/usr/local/cuda/include \
    -L/usr/lib/x86_64-linux-gnu -L/usr/local/cuda/lib64 \
    -lnvinfer -lnvonnxparser -lcudart \
    -o trt_end_to_end
./trt_end_to_end resnet50-v1-12.onnx
```

--------------------------------

### Install dispatch runtime only

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the dispatch runtime and its VC plugin.

```bash
sudo dnf install libnvinfer-dispatch11
sudo dnf install libnvinfer-vc-plugin11
```

--------------------------------

### Install lean runtime only (network repo)

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the lean runtime using the network repository.

```bash
sudo dnf install libnvinfer-lean11
```

--------------------------------

### List Installed libnvinfer Packages with dpkg-query

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Use dpkg-query with a wildcard to list all installed libnvinfer* files, which is useful for verifying Lean/Dispatch runtime installations.

```bash
dpkg-query -W "*nvinfer*"
```

--------------------------------

### Add a plugin to a TensorRT network using add_plugin_v3() in Python

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/plugins-cpp.html

Python example showing how to get a plugin creator, populate plugin fields, create the plugin object, add it to the network, and delete it to avoid memory leaks. The plugin object is created on the heap and must be deleted after adding it to the network.

```python
creator = trt.get_plugin_registry().get_creator(plugin_name, plugin_version, plugin_namespace)
plugin_fc = creator.get_field_names()
# parse_and_fill_fields is application code that validates layer_fields
# against plugin_fc and returns a trt.PluginFieldCollection.
plugin_data = parse_and_fill_fields(plugin_fc, layer_fields)
plugin_obj = creator.create_plugin(layer_name, plugin_data, trt.TensorRTPhase.BUILD)
shape_inputs = []
layer = network.add_plugin_v3(inputs, shape_inputs, plugin_obj)
# build the rest of the network and serialize the engine
del plugin_obj
```

--------------------------------

### Install dispatch runtime only (network repo)

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the dispatch runtime using the network repository.

```bash
sudo dnf install libnvinfer-dispatch11
```

--------------------------------

### Install lean runtime package (network repo)

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the lean runtime package using the network repository.

```bash
sudo apt-get install libnvinfer-lean11
```

--------------------------------

### Install Windows Builder Resource Library

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the Windows Builder Resource Library for cross-platform support.

```bash
sudo dnf install libnvinfer-win-builder-resource11
```

--------------------------------

### Install dispatch TensorRT runtime

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-pip.html

Installs the minimal dispatch runtime.

```bash
python3 -m pip install --upgrade tensorrt-dispatch
```

--------------------------------

### Install Windows builder resource library

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the Windows builder resource library for cross-platform support.

```bash
sudo apt-get install libnvinfer-win-builder-resource11
```

--------------------------------

### Install dispatch runtime package (network repo)

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the dispatch runtime package using the network repository.

```bash
sudo apt-get install libnvinfer-dispatch11
```

--------------------------------

### Install C++ Lean Runtime Development Package

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the C++ development package for the Lean runtime.

```bash
sudo apt-get install libnvinfer-lean-dev
```

--------------------------------

### Install C++ applications runtime only (network repo)

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the C++ applications runtime library using the network repository.

```bash
sudo dnf install tensorrt-libs
```

--------------------------------

### getPrePaddingNd

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/c-api/classnvinfer1_1_1_i_padding_layer.html

Gets the padding that is applied at the start of the tensor.

```APIDOC
## getPrePaddingNd

### Description
Gets the padding that is applied at the start of the tensor.

### Method
Dims getPrePaddingNd() const noexcept

### Returns
Dims - The padding dimensions applied at the start of the tensor.
```

--------------------------------

### Launch TensorRT Runtime Tutorial Notebook

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/quick-start-runtime-tutorial.html

Command to start the tutorial-runtime.ipynb notebook in the OSS repo for an interactive walkthrough. Use this to run the runtime tutorial steps in editable cells.

```bash
jupyter notebook --port=8888 --no-browser --ip=0.0.0.0 --allow-root
```

--------------------------------

### getPrePaddingNd

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/c-api/classnvinfer1_1_1_i_padding_layer.html

Get the padding that is applied at the start of the tensor. Only 2 dimensional padding is currently supported.

```APIDOC
## getPrePaddingNd()

### Description
Get the padding that is applied at the start of the tensor. Only 2 dimensional padding is currently supported.

### Method
const

### Signature
Dims nvinfer1::IPaddingLayer::getPrePaddingNd() const

### Parameters
None

### Returns
- **Dims** - The padding dimensions applied at the start of the tensor.

### See Also
- setPrePaddingNd

```

--------------------------------

### Install sample requirements

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/sample-python-building.html

Run this command to install the dependencies listed in the sample's requirements.txt file. This is the first step before running any Python sample.

```bash
python3 -m pip install -r requirements.txt
```

--------------------------------

### Install TensorRT with specific CUDA version

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-pip.html

By default, TensorRT Python packages install the CUDA 13.x variants. Append -cu12 or -cu13 to select a specific CUDA major version. The example shows installing the full runtime with CUDA 12.x, CUDA 13.x (explicit), and lean and dispatch runtimes with CUDA 12.x.

```bash
# For CUDA 12.x
python3 -m pip install --upgrade tensorrt-cu12

# For CUDA 13.x (default, explicit)
python3 -m pip install --upgrade tensorrt-cu13

# Lean and Dispatch runtimes with CUDA 12.x
python3 -m pip install --upgrade tensorrt-lean-cu12 tensorrt-dispatch-cu12
```

--------------------------------

### Run a Single C++ Sample on Windows

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/sample-cpp-building.html

Navigate to the build output directory and execute the sample binary using Windows command prompt syntax.

```batch
> cd <cloned_tensorrt_dir>\build\out
> .\<sample_bin>
```

--------------------------------

### Run a Single C++ Sample on Linux

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/sample-cpp-building.html

Navigate to the build output directory and execute the sample binary. Replace <sample_bin> with the actual sample name.

```bash
$ cd <cloned_tensorrt_dir>/build/out
$ ./<sample_bin>
```

--------------------------------

### PadPlugin::supportsFormatCombination override

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/plugins-cpp.html

Checks if the plugin accepts a given format and type combination. This example only supports linear formats and FP32 types. The interface indexes inputs and outputs uniformly as connections, starting at 0 for the first input.

```cpp
bool PadPlugin::supportsFormatCombination(
        int32_t pos, DynamicPluginTensorDesc const* inOut, int32_t nbInputs, int32_t nbOutputs) noexcept
{
    if (inOut == nullptr || nbInputs != 1 || nbOutputs != 1
        || pos < 0 || pos >= nbInputs + nbOutputs)
    {
        return false;
    }
    return inOut[pos].desc.format == PluginFormat::kLINEAR && inOut[pos].desc.type == DataType::kFLOAT;
}
```

--------------------------------

### Migrating Plugins from V2 to V3 - Full Example

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x-python-api-patterns.html

Complete Python example of a V3 plugin class and creator, showing the required inheritance, methods, and usage. Key changes include get_capability_interface(), get_output_shapes(), get_output_data_types(), get_fields_to_serialize(), and network.add_plugin_v3().

```python
import tensorrt as trt

class MyPluginV3(trt.IPluginV3, trt.IPluginV3OneCore,
                trt.IPluginV3OneBuild, trt.IPluginV3OneRuntime):
    def __init__(self):
        trt.IPluginV3.__init__(self)
        trt.IPluginV3OneCore.__init__(self)
        trt.IPluginV3OneBuild.__init__(self)
        trt.IPluginV3OneRuntime.__init__(self)
        self.num_outputs = 1
        self.plugin_namespace = ""
        self.plugin_name = "MyPlugin"
        self.plugin_version = "1"

    def get_capability_interface(self, type):
        return self

    def get_output_data_types(self, input_types):
        return [input_types[0]]  # Same type as input

    def get_output_shapes(self, inputs, shape_inputs, exprBuilder):
        # Return a list of DimsExprs, one per output
        output = trt.DimsExprs(len(inputs[0]))
        for i in range(len(inputs[0])):
            output[i] = inputs[0][i]
        return [output]

    def supports_format_combination(self, pos, in_out, num_inputs):
        return in_out[pos].format == trt.TensorFormat.LINEAR and \
              in_out[pos].type == trt.DataType.FLOAT

    def configure_plugin(self, inp, out):
        pass

    def on_shape_change(self, inp, out):
        return 0

    def enqueue(self, input_desc, output_desc, inputs, outputs, workspace, stream):
        # Kernel execution
        pass

    def get_fields_to_serialize(self):
        return trt.PluginFieldCollection([])

    def get_valid_tactics(self):
        return []

    def set_tactic(self, tactic):
        return 0

    def clone(self):
        cloned_plugin = MyPluginV3()
        cloned_plugin.__dict__.update(self.__dict__)
        return cloned_plugin

    def attach_to_context(self, context):
        return self.clone()

# Creator
class MyPluginCreatorV3(trt.IPluginCreatorV3One):
    def __init__(self):
        trt.IPluginCreatorV3One.__init__(self)
        self.name = "MyPlugin"
        self.plugin_namespace = ""
        self.plugin_version = "1"
        self.field_names = trt.PluginFieldCollection([])

    def create_plugin(self, name, fc, phase):
        return MyPluginV3()

# Usage
trt.get_plugin_registry().register_creator(MyPluginCreatorV3())
plugin = MyPluginV3()
layer = network.add_plugin_v3([input_tensor], [], plugin)
```

--------------------------------

### Update pip and install wheel

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-pip.html

Prerequisite step before installing TensorRT. Updates pip and installs the wheel package.

```bash
python3 -m pip install --upgrade pip
python3 -m pip install wheel
```

--------------------------------

### Run C++ segmentation_tutorial binary

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/quick-start-runtime-tutorial.html

Runs the compiled segmentation_tutorial binary with the engine, input, output, and dimensions. On success it writes the output image and exits 0. If it exits non-zero, verify the engine path, that input.ppm matches the height×width arguments, and that the GPU driver matches the Support Matrix.

```bash
make
./bin/segmentation_tutorial fcn-resnet101.engine input.ppm output.ppm 1026x1282
```

--------------------------------

### Install lean runtime packages

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the lean runtime and its VC plugin. Use for a minimal runtime without development headers.

```bash
sudo apt-get install libnvinfer-lean11
sudo apt-get install libnvinfer-vc-plugin11
```

--------------------------------

### TensorRT Operator Boilerplate Framework

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/operators/index.html

Complete boilerplate for running TensorRT operator examples. Includes imports, OutputAllocator, Runner, example template, and run_example. Insert operator-specific code between 'Example Begin' and 'Example End' comments.

```python
import numpy as np
import math # example_plugin_v2.py
import ctypes

import tensorrt as trt
from cuda.bindings import driver as cuda, runtime as cudart
from common import cuda_call, CudaStreamContext, DeviceMem, PinnedHostMem, memcpy_host_to_device_async, memcpy_device_to_host_async




class OutputAllocator(trt.IOutputAllocator):
    def __init__(self, curr_size):
        trt.IOutputAllocator.__init__(self)
        self.curr_size = curr_size
        self.allocated_mem = None
        if curr_size > 0:
            self.allocated_mem = DeviceMem(curr_size)
        self.tensor_shape = None

    def reallocate_output(self, tensor_name, memory, size, alignment):
        assert size > 0
        if size > self.curr_size:
            self.allocated_mem = DeviceMem(size)
            self.curr_size = size
        return int(self.allocated_mem.device_ptr)

    def notify_shape(self, tensor_name, shape):
        self.tensor_shape = shape

class Runner:
    def __init__(self, logger=trt.Logger(min_severity=trt.ILogger.Severity.INFO)):
        self.builder = trt.Builder(logger)
        self.network = self.builder.create_network(flags=0)
        self.config = self.builder.create_builder_config()
        self.runtime = trt.Runtime(logger)
        self.inputs = {}
        self.outputs = {}
        self.expected = {}
        self.results = {}
        self.logger = logger
        self.atol = 0.1

def example(get_runner: Runner):
    network = get_runner.network
    inputs = get_runner.inputs
    outputs = get_runner.outputs
    expected = get_runner.expected

    # -------------------- Example Begin --------------------
    # Paste the code examples here
    # e.g. for Activation
    in1 = network.add_input("input1", dtype=trt.float32, shape=(2, 3))
    layer = network.add_activation(in1, type=trt.ActivationType.RELU)
    network.mark_output(layer.get_output(0))

    inputs[in1.name] = np.array([[-3.0, -2.0, -1.0], [0.0, 1.0, 2.0]])

    outputs[layer.get_output(0).name] = layer.get_output(0).shape

    expected[layer.get_output(0).name] = np.array([[0.0, 0.0, 0.0], [0.0, 1.0, 2.0]])
    # --------------------- Example End ---------------------

    return get_runner

def run_example():
    example_runner = Runner()
    inputs = example_runner.inputs
    outputs = example_runner.outputs
    expected = example_runner.expected
    builder = example_runner.builder
    config = example_runner.config
    runtime = example_runner.runtime
    results = example_runner.results

    def log_info(info):
        example_runner.logger.log(trt.ILogger.Severity.INFO, f"[Example] {info}")

    def log_error(info):
        example_runner.logger.log(trt.ILogger.Severity.ERROR, f"[Example] {info}")

    example_runner = example(example_runner)

    log_info("Building serialized network")
    serialized_engine = builder.build_serialized_network(example_runner.network, config)
    assert serialized_engine is not None

    log_info("Creating engine")
    engine = runtime.deserialize_cuda_engine(serialized_engine)
    context = engine.create_execution_context()

    # Allocate host and device buffers
    in_mem = dict()
    out_mem = dict()
    output_allocators = dict()
    tensor_names = [engine.get_tensor_name(i) for i in range(engine.num_io_tensors)]
    for tensor in tensor_names:
        dtype = np.dtype(trt.nptype(engine.get_tensor_dtype(tensor)))
        if engine.get_tensor_mode(tensor) == trt.TensorIOMode.INPUT:
            if engine.is_shape_inference_io(tensor):
                context.set_input_shape(tensor, inputs[tensor].shape)
                # Get input memory address from the numpy object.
                input_address = inputs[tensor].ctypes.data
                context.set_tensor_address(tensor, input_address)
            else:
                # Handle input tensors
                context.set_input_shape(tensor, inputs[tensor].shape)
                input_buffer = np.ascontiguousarray(inputs[tensor], dtype=dtype)
                input_memory = DeviceMem(input_buffer.nbytes)
                context.set_tensor_address(tensor, int(input_memory.device_ptr))
                in_mem[tensor] = (input_memory, input_buffer)
```

--------------------------------

### Get missing weights (C++)

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/refitting-engines.html

Two calls to getMissingWeights: first to get the count, then to get the names. The returned set is complete.

```cpp
int32_t const n = refitter->getMissingWeights(0, nullptr);
std::vector<const char*> weightsNames(n);
refitter->getMissingWeights(n, weightsNames.data());
```

--------------------------------

### Print full knob listing with --helpBuildRoute

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/tuning.html

Use this command to query the knob database and print all available internal builder knobs as JSON. This output serves as the authoritative reference for configuring --setBuildRoute or --tuneBuildRoutes. --helpBuildRoute does not require an ONNX model and ignores other build flags.

```bash
trtexec --helpBuildRoute
```

--------------------------------

### Quantize ONNX model to FP8 with ModelOpt CLI

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/benchmarking.html

Command-line example that loads an ONNX model, runs calibration with random data, inserts Quantize/Dequantize ops, and saves the quantized model. Use this for performance benchmarking where accuracy is not a concern.

```bash
python3 -m modelopt.onnx.quantization \
    --onnx_path vit.onnx \
    --quantize_mode fp8 \
    --output_path vit-quantized.onnx
```

--------------------------------

### Example JSON Output from getAllBuildRoutes

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/tuning.html

Shows two representative knobs from the tuner_options array: -slice_fusion and -kgen:codegen:cuda_tile. The full listing can be long; use trtexec --helpBuildRoute for the authoritative command-line view.

```json
{
  "tuner_version": "2.19.35",
  "tuner_options": [
    {
      "option": "-slice_fusion",
      "allowed_values": "-slice_fusion=[on|off]",
      "default_value": "on",
      "help": "Replace multiple Slice Ops with a single Split Op."
    },
    {
      "option": "-kgen:codegen:cuda_tile",
      "allowed_values": "-kgen:codegen:cuda_tile=[0|1|2|3]",
      "default_value": "1",
      "help": "CUDA Tile codegen. 0: disable, 1: where profitable, 2: supported kdags, 3: force all."
    }
  ]
}
```

--------------------------------

### C++: Using Transformer-Oriented APIs

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-fused-attention.html

C++ example showing how to create packed Q, K, V inputs, scale Q, and add an attention layer with packed NHD format. Uses addAttentionV2, setQueryForm, setKeyValueForm, setQueryLengths, setKeyValueLengths, and setDecomposable.

```cpp
// All inputs are packed: [totalTokens, numHeads, headSize]
ITensor* qInput = network->addInput("q", DataType::kHALF,
                                     Dims3{-1, numQHeads, headSize});   // [totalTokens, N_q, H]
ITensor* kInput = network->addInput("k", DataType::kHALF,
                                     Dims3{-1, numKvHeads, headSize});  // [totalTokens, N_kv, H]
ITensor* vInput = network->addInput("v", DataType::kHALF,
                                     Dims3{-1, numKvHeads, headSize});  // [totalTokens, N_kv, H]

// Cumulative token counts: [B + 1], first element is 0, last element is totalTokens
// For a batch of images with token counts [196, 256, 128]:
//   queryLengths = kvLengths = [0, 196, 452, 580]
ITensor* queryLengths = network->addInput("queryLengths", DataType::kINT32,
                                           Dims{1, {-1}});
ITensor* kvLengths = network->addInput("kvLengths", DataType::kINT32,
                                        Dims{1, {-1}});

// Scale Q
float qkScale = 1.0f / std::sqrt(static_cast<float>(headSize));
half_float::half scaleData{qkScale};
IConstantLayer* scaleConst = network->addConstant(Dims3{1, 1, 1},
                                                   Weights{DataType::kHALF, &scaleData, 1});

IElementWiseLayer* scaledQ = network->addElementWise(*qInput, *scaleConst->getOutput(0),
                                                      ElementWiseOperation::kPROD);
ITensor* scaledQTensor = scaledQ->getOutput(0);

// Add attention with fully packed Q, K, V
IAttention* attention = network->addAttentionV2(*scaledQTensor, *kInput, *vInput,
                                                 AttentionNormalizationOp::kSOFTMAX,
                                                 CausalMaskKind::kNONE);
attention->setQueryForm(AttentionIOForm::kPACKED_NHD);
attention->setKeyValueForm(AttentionIOForm::kPACKED_NHD);
attention->setQueryLengths(queryLengths);
attention->setKeyValueLengths(kvLengths);
attention->setDecomposable(true);

ITensor* attentionOutput = attention->getOutput(0);
attentionOutput->setName("attentionOutput");
network->markOutput(*attentionOutput);
```

--------------------------------

### Install Specific Version Using aptitude

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Alternative method using aptitude to install an older TensorRT version or with an older CUDA version. Requires installing aptitude first.

```bash
sudo apt-get install aptitude
sudo aptitude install -o "Aptitude::ProblemResolver::Hints::=reject tensorrt :UNINST" tensorrt=${version}
```

--------------------------------

### C++ Padded Format Attention Setup

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-fused-attention.html

Creates inputs for Q, K, V, KV cache, RoPE caches, position IDs, and KV lengths. Applies RoPE to Q and K, updates the KV cache, scales Q, and adds an attention layer with setKeyValueLengths and setDecomposable.

```cpp
ITensor* qInput = network->addInput("q", DataType::kHALF,
                        Dims4{-1, numQHeads, -1, headSize});
ITensor* kInput = network->addInput("k", DataType::kHALF,
                                   Dims4{-1, numKvHeads, -1, headSize});
ITensor* vInput = network->addInput("v", DataType::kHALF,
                                   Dims4{-1, numKvHeads, -1, headSize});

ITensor* kCacheInput = network->addInput("kCache", DataType::kHALF,
                                         Dims4{-1, numKvHeads, kvCacheCapacity, headSize});
ITensor* vCacheInput = network->addInput("vCache", DataType::kHALF,
                                         Dims4{-1, numKvHeads, kvCacheCapacity, headSize});
ITensor* ropeCosCache = network->addInput("ropeCosCache", DataType::kHALF,
                                         Dims2{maxPositionEmbeddings, headSize / 2});
ITensor* ropeSinCache = network->addInput("ropeSinCache", DataType::kHALF,
                                         Dims2{maxPositionEmbeddings, headSize / 2});
ITensor* positionIds = network->addInput("positionIds", DataType::kINT32,
                                         Dims2{-1, -1});
ITensor* kCacheIndices = network->addInput("kCacheIndices", DataType::kINT32,
                                           Dims{1, {-1}});
// Per-batch KV lengths: [B], each element is the number of valid KV tokens for that batch element
ITensor* kvLengths = network->addInput("kvLengths", DataType::kINT32,
                                       Dims{1, {-1}});

// Apply RoPE to Q and K
IRotaryEmbeddingLayer* ropeQ = network->addRotaryEmbedding(*qInput, *ropeCosCache, *ropeSinCache,
                                                           false, 0);
ropeQ->setInput(3, *positionIds);
IRotaryEmbeddingLayer* ropeK = network->addRotaryEmbedding(*kInput, *ropeCosCache, *ropeSinCache,
                                                           false, 0);
ropeK->setInput(3, *positionIds);

ITensor* ropeQTensor = ropeQ->getOutput(0);
ITensor* ropeKTensor = ropeK->getOutput(0);

// Update KV cache
KVCacheMode kvCacheMode = KVCacheMode::kLINEAR;
IKVCacheUpdateLayer* kCacheLayer = network->addKVCacheUpdate(*kCacheInput, *ropeKTensor,
                                                             *kCacheIndices, kvCacheMode);
IKVCacheUpdateLayer* vCacheLayer = network->addKVCacheUpdate(*vCacheInput, *vInput,
                                                             *kCacheIndices, kvCacheMode);

ITensor* kCacheOutput = kCacheLayer->getOutput(0);
ITensor* vCacheOutput = vCacheLayer->getOutput(0);

// Mark cache outputs
kCacheOutput->setName("kCacheOutput");
network->markOutput(*kCacheOutput);
vCacheOutput->setName("vCacheOutput");
network->markOutput(*vCacheOutput);

// Scale Q
float qkScale = 1.0f / std::sqrt(static_cast<float>(headSize));
half_float::half scaleData{qkScale};
IConstantLayer* scaleConst = network->addConstant(Dims4{1, 1, 1, 1},
                                                 Weights{DataType::kHALF, &scaleData, 1});

IElementWiseLayer* scaledQ = network->addElementWise(*ropeQTensor, *scaleConst->getOutput(0),
                                                     ElementWiseOperation::kPROD);
ITensor* scaledQTensor = scaledQ->getOutput(0);

// Add attention layer using the full cache as KV input.
// Use setKeyValueLengths to specify per-batch valid KV lengths instead of
// slicing the cache to extract the present KV.
IAttention* attention = network->addAttentionV2(*scaledQTensor, *kCacheOutput, *vCacheOutput,
                                               AttentionNormalizationOp::kSOFTMAX,
                                               CausalMaskKind::kNONE);
attention->setKeyValueLengths(kvLengths);
attention->setDecomposable(true);

ITensor* attentionOutput = attention->getOutput(0);
attentionOutput->setName("attentionOutput");
network->markOutput(*attentionOutput);
```

--------------------------------

### Benchmark Torch-TRT Module with CUDA Events

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/benchmarking.html

Python function to benchmark a compiled Torch-TRT module using CUDA events. It performs warm-up runs, records start and end events for each inference, and prints mean, median, min, max, P90, P95, and P99 latencies. The example uses a float16 input tensor of shape (128, 3, 224, 224) and calls the function with a precompiled module.

```python
import numpy as np

def benchmark_tensorrt(trt_compiled, input_tensor, num_warmup=10, num_runs=100):
    with torch.no_grad():
        # Warmup
        for _ in range(num_warmup):
            trt_compiled(input_tensor)
        torch.cuda.synchronize()

        # Benchmark
        start_events = [torch.cuda.Event(enable_timing=True) for _ in range(num_runs)]
        end_events = [torch.cuda.Event(enable_timing=True) for _ in range(num_runs)]
        for i in range(num_runs):
            start_events[i].record()
            trt_compiled(input_tensor)
            end_events[i].record()
        torch.cuda.synchronize()
        latencies = [s.elapsed_time(e) for s, e in zip(start_events, end_events)]

        latencies = np.array(latencies)
        print(f"\nLatency over {num_runs} runs (ms):")
        print(f"  Mean:   {np.mean(latencies):.2f}")
        print(f"  Median: {np.median(latencies):.2f}")
        print(f"  Min:    {np.min(latencies):.2f}")
        print(f"  Max:    {np.max(latencies):.2f}")
        print(f"  P90:    {np.percentile(latencies, 90):.2f}")
        print(f"  P95:    {np.percentile(latencies, 95):.2f}")
        print(f"  P99:    {np.percentile(latencies, 99):.2f}")

input_tensor = torch.rand((128, 3, 224, 224), dtype=torch.float16).cuda()
benchmark_tensorrt(trt_compiled, input_tensor)
```

--------------------------------

### Linear Resize Example

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/operators/Resize.html

Demonstrates how to create a resize layer with LINEAR interpolation mode, set the output shape to (1,1,5,5), and use ALIGN_CORNERS coordinate transformation. The example sets up the network, defines input data, and specifies expected output.

```Python
input = network.add_input("input", dtype=trt.float32, shape=(1, 1, 3, 3))
layer = network.add_resize(input)
layer.resize_mode = trt.InterpolationMode.LINEAR
layer.shape = (1, 1, 5, 5)
layer.coordinate_transformation = trt.ResizeCoordinateTransformation.ALIGN_CORNERS
network.mark_output(layer.get_output(0))

inputs[input.name] = np.array([[[[0, 1, 2], [3, 4, 5], [6, 7, 8]]]])

outputs[layer.get_output(0).name] = layer.get_output(0).shape
expected[layer.get_output(0).name] = np.array(
    [
        [
            [0, 0.5, 1, 1.5, 2],
            [1.5, 2, 2.5, 3, 3.5],
            [3, 3.5, 4, 4.5, 5],
            [4.5, 5, 5.5, 6, 6.5],
            [6, 6.5, 7, 7.5, 8],
        ]
    ]
)
```

--------------------------------

### Install dispatch runtime Python package

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the Python package for the dispatch runtime.

```bash
sudo dnf install python3-libnvinfer-dispatch
```

--------------------------------

### Install lean runtime Python package

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the Python package for the lean runtime.

```bash
sudo dnf install python3-libnvinfer-lean
```

--------------------------------

### Clone and Build Samples from GitHub (Release 11.4)

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/sample-cpp-building.html

Samples are not included in release packages; fetch them from GitHub and build. This example clones the release/11.4 branch and builds with the same CMake options.

```bash
git clone -b release/11.4 https://github.com/NVIDIA/TensorRT.git
cd TensorRT
mkdir build && cd build
cmake .. \
  -DTRT_LIB_DIR=$TRT_LIBPATH \
  -DTRT_OUT_DIR=`pwd`/out \
  -DBUILD_SAMPLES=ON \
  -DBUILD_PARSERS=OFF \
  -DBUILD_PLUGINS=OFF
cmake --build . --parallel 4
```

--------------------------------

### Install ONNX Graph Surgeon

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs numpy, onnx, and onnx-graphsurgeon for samples or projects.

```bash
python3 -m pip install numpy onnx onnx-graphsurgeon
```

--------------------------------

### Install dispatch runtime Python package

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the Python package for the dispatch runtime.

```bash
sudo apt-get install python3-libnvinfer-dispatch
```

--------------------------------

### Install lean runtime Python package

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-debian.html

Installs the Python package for the lean runtime.

```bash
sudo apt-get install python3-libnvinfer-lean
```

--------------------------------

### Install lean TensorRT runtime

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-pip.html

Installs only the lean runtime for running pre-built engines.

```bash
python3 -m pip install --upgrade tensorrt-lean
```

--------------------------------

### Verify TensorRT RPM Package Installation

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Use these commands to verify that TensorRT packages are installed. The expected output for the full TensorRT release is 'tensorrt-11.4.0.x-1.cuda-13.4.x86_64'. The lean/dispatch runtime command lists all installed libnvinfer* files.

```bash
rpm -q tensorrt
```

```bash
rpm -qa | grep nvinfer
```

--------------------------------

### Install full TensorRT runtime

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-pip.html

Installs the complete TensorRT package with builder and runtime. By default, installs the CUDA 13.x variant. The dependency names carry the CUDA major version (e.g., tensorrt-cu13-libs, tensorrt-cu13-bindings).

```bash
python3 -m pip install --upgrade tensorrt
```

--------------------------------

### Profile DLA with Nsight Systems

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/benchmarking.html

First command builds and serializes a DLA engine to dla.plan without inference. Second command profiles DLA using the nvmedia accelerator trace. Replace <dla-model>.onnx with a model whose every layer is supported on the target DLA platform. Windows on ARM does not support GPU fallback.

```bash
trtexec --onnx=<dla-model>.onnx --useDLACore=0 --saveEngine=dla.plan --skipInference
nsys profile -t cuda,nvtx,nvmedia,osrt --accelerator-trace=nvmedia --show-output=true trtexec --loadEngine=dla.plan --warmUp=0 --duration=0 --iterations=20
```

--------------------------------

### Install TensorRT C++ Development Package

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-rpm.html

Installs the TensorRT development package for C++ applications.

```bash
sudo dnf install tensorrt-devel
```

--------------------------------

### Add a plugin to a TensorRT network using addPluginV3() in C++

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/plugins-cpp.html

C++ example showing how to look up a plugin creator, populate plugin fields, create the plugin object, add it to the network, and delete it to avoid memory leaks. The plugin object is created on the heap and must be deleted after adding it to the network.

```cpp
// Look up the plugin in the registry
// Cast to appropriate child class of IPluginCreatorInterface
auto creator = static_cast<IPluginCreatorV3One*>(getPluginRegistry()->getCreator(pluginName, pluginVersion, pluginNamespace));
PluginFieldCollection const* pluginFC = creator->getFieldNames();
// Populate the field parameters for the plugin layer
// parseAndFillFields is application code that validates layerFields
// against pluginFC and owns the returned collection.
PluginFieldCollection *pluginData = parseAndFillFields(pluginFC, layerFields);
// Create the plugin object using the layerName and the plugin metadata for use by the TensorRT builder
IPluginV3 *pluginObj = creator->createPlugin(layerName, pluginData, TensorRTPhase::kBUILD);
// Add the plugin to the TensorRT network
auto layer = network->addPluginV3(inputs.data(), int(inputs.size()), shapeInputs.data(), int(shapeInputs.size()), *pluginObj);
//... (build rest of the network and serialize engine)
// Delete the plugin object
delete pluginObj;
// ... (free allocated pluginData)
```

--------------------------------

### Install ModelOpt with pip

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x.html

Install ModelOpt with all extras from NVIDIA's package index. Required before using AutoCast or Quantization features.

```bash
pip3 install --no-cache-dir --extra-index-url https://pypi.nvidia.com nvidia-modelopt[all]
```

--------------------------------

### Upgrade setuptools and packaging

Source: https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/install-pip.html

Updates setuptools and packaging Python modules to resolve TypeError during pip install.

```bash
python3 -m pip install --upgrade setuptools packaging
```
Tokens:
Raw