Triton Inference Server

https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/
Admin
Triton Inference Server is an open-source inference serving software that streamlines AI inferencing...
Tokens:208,894
Snippets:1,658
Trust Score:10
Update:3 days ago
Show doc for...
Tokens:
Raw
### Build and Install Custom Batching Library

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html

Commands to build the volume_batching example library and install it to a local directory.

```bash
$ cd batch_strategies/volume_batching
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
$ make install
```

--------------------------------

### Install required packages for Stable Diffusion optimization

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_navigator/README.html

Install the necessary dependencies before running the Stable Diffusion optimization example.

```bash
pip install transformers diffusers torch
```

--------------------------------

### Build and install the minimal backend

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html

Commands to build the minimal backend and install it into a local directory.

```bash
$ cd backends/minimal
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
$ make install
```

--------------------------------

### Setup mock gRPC service

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/docs/benchmarking.html

Creates a simple Protobuf definition, compiles it, and starts a Python gRPC server that echoes received messages.

```bash
# Define simple gRPC service
cat <<EOF > simple.proto
syntax = "proto3";

package v1;

service Simple {
  // Simple bidirectional streaming RPC that echoes request message
  rpc Echo(stream Request) returns (stream Response) {}
}

message Request {
  string message = 1;
}

message Response {
  string message = 1;
}
EOF

# compile protobuf and generate gRPC stubs
pip install grpcio-tools
python -m grpc_tools.protoc --proto_path=. \
	--python_out=. --grpc_python_out=. \
	simple.proto

# run gRPC server
cat <<EOF > simple_server.py
from concurrent import futures
import grpc
import simple_pb2
import simple_pb2_grpc

class SimpleServicer(simple_pb2_grpc.SimpleServicer):

    def Echo(self, request_iterator, context):
        for request in request_iterator:
            print(f"Received: {request.message}")
            yield simple_pb2.Response(message=request.message)

if __name__ == "__main__":
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    simple_pb2_grpc.add_SimpleServicer_to_server(SimpleServicer(), server)
    server.add_insecure_port(f"[::]:8001")
    server.start()

    print(f"Started gRPC service at 127.0.0.1:8001")
    server.wait_for_termination()
EOF

python simple_server.py
# Output: Started gRPC service at 127.0.0.1:8001
```

--------------------------------

### Create and Start a Server Instance

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html

Initializes and starts the server for diffusion models.

```python
import tritonserver
import numpy
from PIL import Image
server = tritonserver.Server(model_repository="/workspace/diffusion-models")
server.start()
```

--------------------------------

### Build Java Examples with Maven

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/java/README.html

Commands to navigate to the Java client directory and compile the examples using Maven.

```bash
$ cd client/src/java
$ mvn clean install -Ddir=examples
```

--------------------------------

### Run the Java example client

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/java/README.html

Build the examples project and execute the SimpleJavaClient. Provide the host and port (default 8001) as arguments.

```bash
$ cd examples

$ mvn clean install 

$ mvn exec:java -Dexec.mainClass=clients.SimpleJavaClient -Dexec.args="<host> <port>"
```

--------------------------------

### Install Pip

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html

Install python3-pip using apt-get.

```bash
sudo apt-get update && sudo apt-get install python3-pip
```

--------------------------------

### Install wkhtmltopdf

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html

Install wkhtmltopdf to enable PDF report generation.

```bash
sudo apt-get update && sudo apt-get install wkhtmltopdf
```

--------------------------------

### Start Triton Server with tritonserver

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/tritonfrontend.html

Constructs server options and starts the Triton server instance.

```python
import tritonserver

# Constructing path to Model Repository
model_path = f"server/src/python/examples/example_model_repository"

server_options = tritonserver.Options(
    server_id="ExampleServer",
    model_repository=model_path,
    log_error=True,
    log_warn=True,
    log_info=True,
)
server = tritonserver.Server(server_options).start(wait_until_ready=True)
```

--------------------------------

### Create and Start a Server Instance

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html

Initializes and starts the Triton server instance pointing to the model repository.

```python
import tritonserver
server = tritonserver.Server(model_repository="/workspace/identity-models")
server.start()
```

--------------------------------

### Build and install backend utilities

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/README.html

Commands to create a build directory, configure the build with CMake, and install the backend utilities library to a local directory.

```bash
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
$ make install
```

--------------------------------

### Install dependencies

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/javascript/README.html

Install the required Node.js packages.

```bash
npm install
```

--------------------------------

### Download examples via sparse checkout

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/bls_quick_start.html

Enable sparse checkout to download only the examples directory containing the bls and add models.

```bash
git config core.sparseCheckout true && \
echo 'examples' >> .git/info/sparse-checkout && \
git pull origin main
```

--------------------------------

### Install Protobuf Compiler

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/rust/triton-client/README.html

Install the required Protocol Buffers compiler on macOS or Ubuntu/Debian systems before building.

```bash
# macOS
brew install protobuf

# Ubuntu/Debian
apt-get install protobuf-compiler
```

--------------------------------

### Install Built Wheel

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html

Install the generated wheel file using pip.

```bash
pip3 install wheels/triton-model-analyzer-*.whl
```

--------------------------------

### Copy generated stubs to examples folder

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/java/README.html

Copy the generated Java and gRPC stubs into the examples project directory.

```bash
$ cd ..

$ cp -R library/target/generated-sources/protobuf/java/inference  examples/src/main/java/inference

$ cp -R library/target/generated-sources/protobuf/grpc-java/inference/*.java  examples/src/main/java/inference/
```

--------------------------------

### Start server with LoRA support

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html

Command to start the server with a specified model and LoRA separator.

```bash
python3 openai_frontend/main.py --lora-separator=_lora_ ...
```

--------------------------------

### Model Configuration Parameters Example

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config.html

Example of defining model configuration parameters, showing how to specify lists for sweeping through values.

```yaml
model_config_parameters:
 max_batch_size: [6, 8]
 dynamic_batching:
 max_queue_delay_microseconds: [200, 300]
 instance_group:
 - kind: KIND_GPU
 count: [1, 2]
```

--------------------------------

### is_server_ready

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html

Contact the inference server and get readiness.

```APIDOC
## is_server_ready

### Description
Contact the inference server and get readiness.

### Method
is_server_ready

### Parameters
- **headers** (_dict_) - Optional dictionary specifying additional HTTP headers to include in the request.
- **client_timeout** (_float_) - The maximum end-to-end time, in seconds, the request is allowed to take. The client will abort request and raise InferenceServerExeption with message "Deadline Exceeded" when the specified time elapses. The default value is None which means client will wait for the response from the server.

### Returns
True if server is ready, False if server is not ready.

### Return type
bool

### Raises
- **InferenceServerException** - If unable to get readiness or has timed out.
```

--------------------------------

### Create and initialize directory

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/quick_start.html

Create a new directory and initialize a git repository to prepare for downloading the examples.

```bash
mkdir <new_dir> && cd <new_dir>
```

```bash
git init && git remote add -f origin https://github.com/triton-inference-server/model_analyzer.git
```

--------------------------------

### Run NVIDIA CUDA Container

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html

Starts a Docker container with NVIDIA GPU support for environment setup.

```bash
docker run --rm --ipc=host --runtime=nvidia --gpus '"device=0"' --entrypoint /bin/bash -it nvidia/cuda:12.4.1-devel-ubuntu22.04
```

--------------------------------

### Fixed schedule mode: input_data.json

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/docs/inference_load_modes.html

JSON input data file for fixed schedule mode. Each entry has a payload and a timestamp in milliseconds. The timestamp tells Perf Analyzer when to send the request relative to benchmark start. This example sends two requests at 1000ms and 2000ms.

```json
# input_data.json
{
 "data": [
 {
 "payload": [{
 "model": "facebook/opt-125m",
 "messages": [{"role": "user","content": "What is the capital of France?"}],
 "max_completion_tokens": 1
 }],
 "timestamp": [1000]
 },
 {
 "payload": [{
 "model": "facebook/opt-125m",
 "messages": [{"role": "user","content": "What is 1+1?"}],
 "max_completion_tokens": 1
 }],
 "timestamp": [2000]
 }
 ]
}
```

--------------------------------

### Prepare proto files

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/javascript/README.html

Create a proto directory and copy the required protobuf files from the cloned repository.

```bash
mkdir proto
cp common-repo/protobuf/*.proto ./proto/
```

--------------------------------

### Run Go gRPC Client Example

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/go/README.html

Steps to prepare the model repository, launch the Triton server via Docker, and run the Go client.

```bash
# Clone the repo containing the example model, from within client/src/grpc_generated/go/.
git clone https://github.com/triton-inference-server/server.git

# Setup "simple" model from example model_repository
cd server/docs/examples
./fetch_models.sh

# Launch (detached) Triton
docker run -d -p8000:8000 -p8001:8001 -p8002:8002 -it -v $(pwd)/model_repository:/models nvcr.io/nvidia/tritonserver:22.11-py3 tritonserver --model-store=/models

# Use client
cd ../../../
go run grpc_simple_client.go
```

--------------------------------

### Install tritonclient HTTP support only

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/README.html

Installs only the HTTP/REST client library.

```bash
$ pip install tritonclient[http]
```

--------------------------------

### Run the Scala example client

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/java/README.html

Execute the SimpleClient for Scala, which includes additional checks for server and model readiness.

```bash
$ mvn exec:java -Dexec.mainClass=clients.SimpleClient -Dexec.args="<host> <port>"
```

--------------------------------

### Start Stable Diffusion Container

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html

Starts the container with the diffusion framework.

```bash
./run.sh --framework diffusion
```

--------------------------------

### Install Model Analyzer via Pip

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html

Install the Model Analyzer package.

```bash
pip3 install triton-model-analyzer
```

--------------------------------

### Install Maven

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inprocess_java_api.html

Commands to download, extract, and add Maven to the system path for building bindings manually.

```bash
$ cd /opt/tritonserver
 $ wget https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz
 $ tar zxvf apache-maven-3.8.4-bin.tar.gz
 $ export PATH=/opt/tritonserver/apache-maven-3.8.4/bin:$PATH
```

--------------------------------

### Install TensorRT-LLM Dependencies and Package

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html

Installs system dependencies and the specific version of TensorRT-LLM.

```bash
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev git git-lfs
```

```bash
pip3 install tensorrt_llm==0.11.0 --extra-index-url https://pypi.nvidia.com
```

```bash
python3 -c "import tensorrt_llm"
```

--------------------------------

### Start Triton Server

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/README.html

Launch the Triton server and verify it is ready to accept requests.

```bash
# inside triton container
tritonserver --model-repository $(pwd)/model_repository &> server.log &

# confirm server is ready, look for 'HTTP/1.1 200 OK'
curl -v localhost:8000/v2/health/ready

# detach (CTRL-p CTRL-q)
```

--------------------------------

### Example JSON Response

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html

An example of the JSON response received from the chat completion request.

```json
{
 "id": "cmpl-704c758c-8a84-11ef-b106-107c6149ca79",
 "choices": [
 {
 "finish_reason": "stop",
 "index": 0,
 "message": {
 "content": "It looks like you're testing the system!",
 "tool_calls": null,
 "role": "assistant",
 "function_call": null
 },
 "logprobs": null
 }
 ],
 "created": 1728948689,
 "model": "llama-3-8b-instruct",
 "system_fingerprint": null,
 "object": "chat.completion",
 "usage": null
}
```

--------------------------------

### Install JDK

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inprocess_java_api.html

Command to update package lists and install OpenJDK 11 within the container.

```bash
$ apt update && apt install -y openjdk-11-jdk
```

--------------------------------

### Benchmark OpenAI-compatible server with Perf Analyzer

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/docs/benchmarking.html

This workflow demonstrates setting up a vLLM server, creating input data, and running Perf Analyzer with the --service-kind=openai flag. It includes the expected console output for the benchmark run.

```bash
# get chat template required for facebook/opt-125m model
curl -o template_chatml.jinja https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/examples/template_chatml.jinja

# start vllm, an OpenAI API-compatible server
vllm serve facebook/opt-125m --chat-template=template_chatml.jinja > server.log 2>&1 &

# wait for server to be ready
while [ "$(curl -s -o /dev/null -w "%{http_code}" localhost:8000/v1/models)" != "200" ]; do sleep 1; done

# create simple input data JSON for Perf Analyzer
cat > input_data.json <<EOF
{
  "data": [
    {
      "payload": [
        {
          "model": "facebook/opt-125m",
          "messages": [
            {"role": "user", "content": "Who wrote the play Romeo and Juliet?"}
          ],
          "max_tokens": 32
        }
      ]
    }
  ]
}
EOF

# run Perf Analyzer
perf_analyzer \
  -m facebook/opt-125m \
  --input-data=input_data.json \
  --service-kind=openai \
  --endpoint=v1/chat/completions \
  --async

#  Successfully read data for 1 stream/streams with 1 step/steps.
# *** Measurement Settings ***
#   Service Kind: OPENAI
#   Using "time_windows" mode for stabilization
#   Stabilizing using average throughput
#   Measurement window: 5000 msec
#   Using asynchronous calls for inference

# Request concurrency: 1
#   Client:
#     Request count: 89
#     Throughput: 4.94426 infer/sec
#     Avg latency: 200467 usec (standard deviation 17124 usec)
#     p50 latency: 204443 usec
#     p90 latency: 205549 usec
#     p95 latency: 205706 usec
#     p99 latency: 206259 usec
#     Avg HTTP time: 200461 usec (send/recv 169 usec + response wait 200292 usec)
# Inferences/Second vs. Client Average Batch Latency
# Concurrency: 1, throughput: 4.94426 infer/sec, latency 200467 usec
```

--------------------------------

### Triton Server Startup Output

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/quickstart.html

Example console output indicating that models are loaded and the server is ready to accept requests.

```text
+----------------------+---------+--------+
| Model                | Version | Status |
+----------------------+---------+--------+
| <model_name>         | <v>     | READY  |
| ..                   | .       | ..     |
| ..                   | .       | ..     |
+----------------------+---------+--------+
...
...
...
I1002 21:58:57.891440 62 grpc_server.cc:3914] Started GRPCInferenceService at 0.0.0.0:8001
I1002 21:58:57.893177 62 http_server.cc:2717] Started HTTPService at 0.0.0.0:8000
I1002 21:58:57.935518 62 http_server.cc:2736] Started Metrics Service at 0.0.0.0:8002
```

--------------------------------

### Start triton-python-api:r24.01 Container

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html

Starts a container and volume mounts the current directory as workspace.

```bash
./run.sh
```

--------------------------------

### Install Python Bindings and Requirements

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html

Install the necessary Triton server Python bindings and application dependencies.

```bash
pip install /opt/tritonserver/python/triton*.whl
```

```bash
git clone https://github.com/triton-inference-server/server.git
cd server/python/openai/
pip install -r requirements.txt
```

--------------------------------

### Launch Triton Inference Server (py3-sdk)

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html

Starts the Triton Inference Server SDK container for benchmarking.

```bash
export RELEASE="24.07"
docker run -it --net=host --gpus '"device=0"'  nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk
```

--------------------------------

### Run the QA Docker container

Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/test.html

Start the QA container and mount the model repositories from the host.

```bash
$ docker run --gpus=all -it --rm -v/tmp:/data/inferenceserver tritonserver_qa
```
Tokens:
Raw