TensorRT LLM

https://nvidia.github.io/TensorRT-LLM/
Admin
TensorRT LLM is an open-source library for accelerating and optimizing the inference performance of...
Tokens:1,292,020
Snippets:8,212
Trust Score:9.7
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Quick start example without retention config

Source: https://nvidia.github.io/TensorRT-LLM/examples/kvcacheretentionconfig.html

Basic example from `examples/llm-api/quickstart_example.py` that generates text for four prompts using default KV cache retention priority (35).

```python
from tensorrt_llm import LLM, SamplingParams


def main():
    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = SamplingParams(max_tokens=32)

    llm = LLM(model='TinyLlama/TinyLlama-1.1B-Chat-v1.0')
    outputs = llm.generate(prompts, sampling_params)

    for i, output in enumerate(outputs):
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"[{i}] Prompt: {prompt!r}, Generated text: {generated_text!r}")


if __name__ == '__main__':
    main()

```

--------------------------------

### Generate a video with Wan 2.1 using quickstart_example.py

Source: https://nvidia.github.io/TensorRT-LLM/models/visual-generation.html

Run this command to generate a video with Wan 2.1. It executes the quickstart example script.

```bash
python examples/visual_gen/quickstart_example.py

```

--------------------------------

### Install Ray dependencies

Source: https://nvidia.github.io/TensorRT-LLM/features/ray-orchestrator.html

Install Ray and dependencies for the Ray orchestrator prototype. Run from the repository root before using the example scripts.

```bash
cd examples/ray_orchestrator
pip install -r requirements.txt

```

--------------------------------

### Quick start with quickstart_multimodal.py

Source: https://nvidia.github.io/TensorRT-LLM/features/multi-modality.html

Run the quickstart_multimodal.py script from the examples/llm-api directory. It imports helpers from quickstart_advanced.py, so you must clone the repository and run from that directory. The example uses the NVILA-8B model with image modality and disables KV cache reuse.

```bash
# Get the examples directory first; quickstart_multimodal.py imports from
# quickstart_advanced.py next to it.
git clone --depth 1 https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM/examples/llm-api
python3 quickstart_multimodal.py --model_dir Efficient-Large-Model/NVILA-8B --modality image --disable_kv_cache_reuse

```

--------------------------------

### Launch OpenAI-compatible server with trtllm-serve

Source: https://nvidia.github.io/TensorRT-LLM/features/multi-modality.html

Start an OpenAI-compatible server with multimodal support using the trtllm-serve command. The example uses the Qwen/Qwen2-VL-7B-Instruct model with the PyTorch backend. After launching, you can send OpenAI-compatible requests to the server endpoint.

```bash
trtllm-serve Qwen/Qwen2-VL-7B-Instruct  --backend pytorch

```

--------------------------------

### Benchmark output log

Source: https://nvidia.github.io/TensorRT-LLM/performance/perf-benchmarking.html

Example output from the trtllm-bench throughput command, showing the benchmark setup, warmup, request submission, and final performance statistics. The output includes engine details, world/runtime information, and performance overview with token throughput, request throughput, and total latency.

```text
[TensorRT-LLM] TensorRT-LLM version: 0.17.0
[01/18/2025-01:01:13] [TRT-LLM] [I] Preparing to run throughput benchmark...
[01/18/2025-01:01:13] [TRT-LLM] [I] Setting up throughput benchmark.

<snip verbose logging>

[01/18/2025-01:01:26] [TRT-LLM] [I] Setting up for warmup...
[01/18/2025-01:01:26] [TRT-LLM] [I] Running warmup.
[01/18/2025-01:01:26] [TRT-LLM] [I] Starting benchmarking async task.
[01/18/2025-01:01:26] [TRT-LLM] [I] Starting benchmark...
[01/18/2025-01:01:26] [TRT-LLM] [I] Request submission complete. [count=2, time=0.0000s, rate=2590780.97 req/s]
[01/18/2025-01:01:28] [TRT-LLM] [I] Benchmark complete.
[01/18/2025-01:01:28] [TRT-LLM] [I] Stopping LLM backend.
[01/18/2025-01:01:28] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[01/18/2025-01:01:28] [TRT-LLM] [I] All tasks cancelled.
[01/18/2025-01:01:28] [TRT-LLM] [I] LLM Backend stopped.
[01/18/2025-01:01:28] [TRT-LLM] [I] Warmup done.
[01/18/2025-01:01:28] [TRT-LLM] [I] Starting benchmarking async task.
[01/18/2025-01:01:28] [TRT-LLM] [I] Starting benchmark...
[01/18/2025-01:01:28] [TRT-LLM] [I] Request submission complete. [count=3000, time=0.0012s, rate=2590780.97 req/s]
[01/18/2025-01:01:42] [TRT-LLM] [I] Benchmark complete.
[01/18/2025-01:01:42] [TRT-LLM] [I] Stopping LLM backend.
[01/18/2025-01:01:42] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[01/18/2025-01:01:42] [TRT-LLM] [I] All tasks cancelled.
[01/18/2025-01:01:42] [TRT-LLM] [I] LLM Backend stopped.
[01/18/2025-01:01:42] [TRT-LLM] [I]

===========================================================
= ENGINE DETAILS
===========================================================
Model:                  meta-llama/Llama-3.1-8B
Engine Directory:       /tmp/meta-llama/Llama-3.1-8B/tp_1_pp_1
TensorRT-LLM Version:   0.17.0
Dtype:                  bfloat16
KV Cache Dtype:         FP8
Quantization:           FP8
Max Input Length:       256
Max Sequence Length:    256

===========================================================
= WORLD + RUNTIME INFORMATION
===========================================================
TP Size:                1
PP Size:                1
Max Runtime Batch Size: 4096
Max Runtime Tokens:     8192
Scheduling Policy:      Guaranteed No Evict
KV Memory Percentage:   90.00%
Issue Rate (req/sec):   5.0689E+14

===========================================================
= PERFORMANCE OVERVIEW
===========================================================
Number of requests:             3000
Average Input Length (tokens):  128.0000
Average Output Length (tokens): 128.0000
Token Throughput (tokens/sec):  28390.4265
Request Throughput (req/sec):   221.8002
Total Latency (ms):             13525.6862

===========================================================

[01/18/2025-01:01:42] [TRT-LLM] [I] Thread proxy_dispatch_result_thread stopped.
[TensorRT-LLM][INFO] Refreshed the MPI local session
```

--------------------------------

### Sparse attention example setup and argument parsing

Source: https://nvidia.github.io/TensorRT-LLM/examples/llm_sparse_attention.html

Initial portion of the llm_sparse_attention.py example. Imports TensorRT-LLM APIs including RocketSparseAttentionConfig and DeepSeekSparseAttentionConfig, defines a built-in needle-in-a-haystack prompt that exceeds the default 2,048-token prompt budget, and starts the argument parser with --model_path, --input_file, and --algo options.

```python
import argparse
import json

from tensorrt_llm import LLM, SamplingParams
from tensorrt_llm.llmapi import (CudaGraphConfig, DeepSeekSparseAttentionConfig,
                                 KvCacheConfig, MoeConfig,
                                 RocketSparseAttentionConfig)

# The built-in prompt follows a self-contained needle-in-a-haystack layout:
# 1. Cycle routine expedition log templates to form a deterministic haystack.
# 2. Replace one numbered log entry with a unique access-code needle.
# 3. Append a question that asks the model to retrieve the needle.
# With the default model, 128 entries produce about 2.8K tokens, exceeding the
# default 2,048-token prompt budget so that sparse attention is exercised.
_DEFAULT_LOG_TEMPLATES = (
    "The survey team checked the northern weather station and recorded normal "
    "temperature and pressure readings.",
    "Technicians inspected backup batteries, radio transmitters, and emergency "
    "lighting; all systems passed routine checks.",
    "Researchers cataloged soil samples, labeled storage containers, and "
    "updated the expedition inventory.",
    "The navigation group reviewed trail maps, satellite images, and the next "
    "day's travel schedule.",
    "At sunset, the field crew secured scientific instruments and uploaded the "
    "day's measurements.",
    "The medical officer reviewed first-aid supplies and confirmed the "
    "evacuation plan with base camp.",
    "Engineers calibrated wind sensors and verified timestamps against the "
    "observatory master clock.",
    "The logistics team counted food, water, fuel, and spare parts before "
    "closing the storage area.",
)
_DEFAULT_NUM_LOG_ENTRIES = 128
_DEFAULT_NEEDLE_INDEX = 87
_DEFAULT_ACCESS_CODE = "314159"


def _build_default_prompt():
    log_entries = []
    for index in range(_DEFAULT_NUM_LOG_ENTRIES):
        if index == _DEFAULT_NEEDLE_INDEX:
            message = (f"The expedition access code is {_DEFAULT_ACCESS_CODE}. "
                       "Remember this code for the final question.")
        else:
            message = _DEFAULT_LOG_TEMPLATES[index %
                                             len(_DEFAULT_LOG_TEMPLATES)]
        log_entries.append(f"Log entry {index}: {message}")

    return (
        "Read the following expedition field log carefully. One entry contains "
        "an access code that you will need to recall.\n\n" +
        "\n".join(log_entries) + "\n\nWhat is the expedition access code?")


DEFAULT_PROMPTS = [_build_default_prompt()]


def read_input(input_file):
    results = []
    with open(input_file, 'r') as f:
        for line in f:
            ret = json.loads(line)
            results.append(ret)
    return results


def parse_arguments():
    parser = argparse.ArgumentParser()
    parser.add_argument('--model_path',
                        type=str,
                        default="nvidia/Llama-3.1-8B-Instruct-FP8",
                        help="The local path or Hugging Face ID of the model.")
    parser.add_argument(
        '--input_file',
        type=str,
        default=None,
        help="Optional path to a JSONL input file. The built-in "
        "long prompt is used when omitted.")

    # Build config
    parser.add_argument('--algo',
                        type=str,
                        default='ROCKETKV',
```

--------------------------------

### Uninstall previous CUTLASS DSL installation

Source: https://nvidia.github.io/TensorRT-LLM/installation/installation-guide.html

Before installing the latest TensorRT-LLM, uninstall any previous CUTLASS DSL installation as described in the CUTLASS DSL installation guide.

```bash
pip3 uninstall nvidia-cutlass-dsl nvidia-cutlass-dsl-libs-base \
    nvidia-cutlass-dsl-libs-core nvidia-cutlass-dsl-libs-cu12 nvidia-cutlass-dsl-libs-cu13
```

--------------------------------

### Enable guided decoding with xgrammar backend

Source: https://nvidia.github.io/TensorRT-LLM/features/guided-decoding.html

Create an LLM instance with `guided_decoding_backend="xgrammar"` to enable guided decoding. This is the prerequisite for all guided decoding examples on this page.

```python
from tensorrt_llm import LLM

llm = LLM("nvidia/Llama-3.1-8B-Instruct-FP8", guided_decoding_backend="xgrammar")

```

--------------------------------

### Complete example with configuration options

Source: https://nvidia.github.io/TensorRT-LLM/features/auto_deploy/advanced/example_run.html

A more complete example that configures world size, runtime, compile backend, attention backend, and enables the built-in benchmark.

```bash
cd examples/auto_deploy
python build_and_run_ad.py \
--model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" \
--args.world-size 2 \
--args.runtime "demollm" \
--args.compile-backend "torch-compile" \
--args.attn-backend "flashinfer" \
--benchmark.enabled True

```

--------------------------------

### Start multimodal server with configuration file

Source: https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve/trtllm-serve.html

Starts the server for a multimodal model (Qwen/Qwen2-VL-7B-Instruct) using the configuration file created earlier.

```bash
trtllm-serve Qwen/Qwen2-VL-7B-Instruct \
    --config ./config.yml

```

--------------------------------

### Run DeepSeek models with MTP using quickstart_advanced.py

Source: https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog02_DeepSeek_R1_MTP_Implementation_and_Optimization.html

Run DeepSeek-V3/R1 models with MTP using the quickstart_advanced.py example. Specify the model directory, set --spec_decode_algo to MTP, and set --spec_decode_max_draft_len to the desired draft length N.

```bash
cd examples/llm-api
python quickstart_advanced.py --model_dir <YOUR_MODEL_DIR> --spec_decode_algo MTP --spec_decode_max_draft_len N

```

--------------------------------

### Example .devcontainer/docker-compose.override.yml

Source: https://nvidia.github.io/TensorRT-LLM/developer-guide/dev-containers.html

Example override file to add custom volume mounts. The file is merged with the base docker-compose.yml when starting the container. Uncomment the volume lines to mount a data directory.

```yaml
# Example .devcontainer/docker-compose.override.yml
version: "3.9"
services:
  tensorrt_llm-dev:
    volumes:
      # Uncomment the following lines to enable
      # # Mount TRTLLM data volume:
      # - /home/scratch.trt_llm_data_ci/:/home/scratch.trt_llm_data_ci/:ro

```

--------------------------------

### Get decoder prompt token IDs with forced BOS prefix

Source: https://nvidia.github.io/TensorRT-LLM/_modules/tensorrt_llm/llmapi/llm.html

Returns a list starting with the decoder start token ID followed by the decoder output token prefix from sampling parameters. Used to construct the prompt for decoder-only models.

```python
    def _get_decoder_prompt_token_ids(
            self, sampling_params: SamplingParams) -> List[int]:
        return [
            self._get_decoder_start_token_id(),
            *sampling_params._decoder_output_token_prefix,
        ]
```

--------------------------------

### Run the KV cache connector example with Llama-3.1-8B-Instruct

Source: https://nvidia.github.io/TensorRT-LLM/examples/llm_kv_cache_connector.html

Example command that runs the script with the meta-llama/Llama-3.1-8B-Instruct model.

```bash
python llm_kv_cache_connector.py meta-llama/Llama-3.1-8B-Instruct
```

--------------------------------

### Start and stop profiling with curl

Source: https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve/trtllm-serve.html

Example workflow using curl to start a 30-iteration profile window, stop it if needed, and inspect the resulting chrome traces. The traces are written to /tmp/traces/ with filenames like trtllm-trace-<profile_id>-rank-<N>.json.

```bash
# Start a 30-iteration profile window into /tmp/traces.
curl -s -X POST http://127.0.0.1:8000/start_profile \
    -H "Content-Type: application/json" \
    -d '{"output_dir": "/tmp/traces", "num_steps": 30}'

# ... drive load via /v1/completions or /v1/chat/completions ...

# If you started the window without num_steps, stop it:
curl -s -X POST http://127.0.0.1:8000/stop_profile

# Inspect the resulting chrome traces.
ls /tmp/traces/
# trtllm-trace-1778480665-7993d824-rank-0.json
# trtllm-trace-1778480665-7993d824-rank-1.json   (TP=2)
```

--------------------------------

### Run the KV cache connector example

Source: https://nvidia.github.io/TensorRT-LLM/examples/llm_kv_cache_connector.html

Command to run the example script with a model path. The example uses meta-llama/Llama-3.1-8B-Instruct as the model.

```bash
python llm_kv_cache_connector.py <model_path>
```

--------------------------------

### Run release container with make

Source: https://nvidia.github.io/TensorRT-LLM/installation/containers.html

Runs the release container. Supports LOCAL_USER=1 to switch to local user account. Examples are installed in /app/tensorrt_llm/examples.

```bash
make -C docker release_run

```

--------------------------------

### Run AutoDeploy example with build_and_run_ad.py

Source: https://nvidia.github.io/TensorRT-LLM/features/auto_deploy/advanced/example_run.html

Basic command to build and run an AutoDeploy example using the TinyLlama model. Navigate to the examples/auto_deploy directory first.

```bash
cd examples/auto_deploy
python build_and_run_ad.py --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

```

--------------------------------

### Single-session replay with run_trace_replay.py

Source: https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog27_Evaluating_Agentic_Serving_with_Trace_Replay_and_Job_Level_Metrics.html

Replay a trace against any OpenAI-compatible endpoint, typically trtllm-serve. Use this to verify the setup with the shipped example trace before collecting your own.

```bash
python examples/scaffolding/trace_replay/run_trace_replay.py <trace>.trace.json \
  --model <model> --openai-base-url http://127.0.0.1:8000/v1

```

--------------------------------

### Run quick-start job with sbatch

Source: https://nvidia.github.io/TensorRT-LLM/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.html

Runs the ready-made multi-node quick-start job without a server. Loads the model once, runs four prompts, and reports whether each response contains expected text (successful run reports True for all four checks). See examples/kimi_k3/README.md for overriding Slurm partition, account, and time limit.

```bash
sbatch examples/kimi_k3/quick_start_kimi_k3.sbatch \
    --model /path/to/kimi-k3-checkpoint \
    --image /path/to/tensorrt-llm-container.sqsh
```

--------------------------------

### Python example: structural_tag guided decoding with two tools

Source: https://nvidia.github.io/TensorRT-LLM/features/guided-decoding.html

Configures a TensorRT-LLM LLM with the xgrammar guided decoding backend and creates a structural_tag that triggers on '<function=' and requires JSON objects matching the tool parameters, producing output that calls both tools in the specified format.

```python
import json
from tensorrt_llm import LLM
from tensorrt_llm.sampling_params import SamplingParams, GuidedDecodingParams

if __name__ == "__main__":
    llm = LLM("nvidia/Llama-3.1-8B-Instruct-FP8", guided_decoding_backend="xgrammar")

    tool_get_current_weather = {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "The city to find the weather for, e.g. 'San Francisco'",
                    },
                    "state": {
                        "type": "string",
                        "description": "the two-letter abbreviation for the state that the city is in, e.g. 'CA' which would mean 'California'",
                    },
                    "unit": {
                        "type": "string",
                        "description": "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["city", "state", "unit"],
            },
        },
    }

    tool_get_current_date = {
        "type": "function",
        "function": {
            "name": "get_current_date",
            "description": "Get the current date and time for a given timezone",
            "parameters": {
                "type": "object",
                "properties": {
                    "timezone": {
                        "type": "string",
                        "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'",
                    }
                },
                "required": ["timezone"],
            },
        },
    }

    system_prompt = f"""# Tool Instructions
- Always execute python code in messages that you share.
- When looking for real time information use relevant functions if available else fallback to brave_search
You have access to the following functions:
Use the function 'get_current_weather' to: Get the current weather in a given location
{tool_get_current_weather["function"]}
Use the function 'get_current_date' to: Get the current date and time for a given timezone
{tool_get_current_date["function"]}
If a you choose to call a function ONLY reply in the following format:
<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}
where
start_tag => `<function`
parameters => a JSON dict with the function argument name as key and function argument value as value.
end_tag => `</function>`
Here is an example,
<function=example_function_name>{{"example_name": "example_value"}}</function>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- Always add your sources when using search results to answer the user query
You are a helpful assistant."""
    user_prompt = "You are in New York. Please get the current date and time, and the weather."
    structural_tag = {
        "type": "structural_tag",
        "format": {
            "type": "triggered_tags",
            "triggers": ["<function="],
            "tags": [
                {
                    "begin": "<function=get_current_weather>",
                    "content": {
                        "type": "json_schema",
                        "json_schema": tool_get_current_weather["function"]["parameters"]
                    },
                    "end": "</function>",
                },
                {
                    "begin": "<function=get_current_date>",
                    "content": {
                        "type": "json_schema",
                        "json_schema": tool_get_current_date["function"]["parameters"]
                    },
                    "end": "</function>",
                },
            ],
        },
    }

    messages = [
        {
            "role": "system",
            "content": system_prompt,
        },
        {
            "role": "user",
            "content": user_prompt,
        },
    ]
    prompt = llm.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

    output = llm.generate(
        prompt,
        sampling_params=SamplingParams(max_tokens=256, guided_decoding=GuidedDecodingParams(structural_tag=json.dumps(structural_tag))),
    )
    print(output.outputs[0].text)

```
Tokens:
Raw