### Build and Install Custom Batching Library Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html Commands to build the volume_batching example library and install it to a local directory. ```bash $ cd batch_strategies/volume_batching $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install .. $ make install ``` -------------------------------- ### Install required packages for Stable Diffusion optimization Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_navigator/README.html Install the necessary dependencies before running the Stable Diffusion optimization example. ```bash pip install transformers diffusers torch ``` -------------------------------- ### Build and install the minimal backend Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html Commands to build the minimal backend and install it into a local directory. ```bash $ cd backends/minimal $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install .. $ make install ``` -------------------------------- ### Setup mock gRPC service Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/docs/benchmarking.html Creates a simple Protobuf definition, compiles it, and starts a Python gRPC server that echoes received messages. ```bash # Define simple gRPC service cat < simple.proto syntax = "proto3"; package v1; service Simple { // Simple bidirectional streaming RPC that echoes request message rpc Echo(stream Request) returns (stream Response) {} } message Request { string message = 1; } message Response { string message = 1; } EOF # compile protobuf and generate gRPC stubs pip install grpcio-tools python -m grpc_tools.protoc --proto_path=. \ --python_out=. --grpc_python_out=. \ simple.proto # run gRPC server cat < simple_server.py from concurrent import futures import grpc import simple_pb2 import simple_pb2_grpc class SimpleServicer(simple_pb2_grpc.SimpleServicer): def Echo(self, request_iterator, context): for request in request_iterator: print(f"Received: {request.message}") yield simple_pb2.Response(message=request.message) if __name__ == "__main__": server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) simple_pb2_grpc.add_SimpleServicer_to_server(SimpleServicer(), server) server.add_insecure_port(f"[::]:8001") server.start() print(f"Started gRPC service at 127.0.0.1:8001") server.wait_for_termination() EOF python simple_server.py # Output: Started gRPC service at 127.0.0.1:8001 ``` -------------------------------- ### Create and Start a Server Instance Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html Initializes and starts the server for diffusion models. ```python import tritonserver import numpy from PIL import Image server = tritonserver.Server(model_repository="/workspace/diffusion-models") server.start() ``` -------------------------------- ### Build Java Examples with Maven Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/java/README.html Commands to navigate to the Java client directory and compile the examples using Maven. ```bash $ cd client/src/java $ mvn clean install -Ddir=examples ``` -------------------------------- ### Run the Java example client Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/java/README.html Build the examples project and execute the SimpleJavaClient. Provide the host and port (default 8001) as arguments. ```bash $ cd examples $ mvn clean install $ mvn exec:java -Dexec.mainClass=clients.SimpleJavaClient -Dexec.args=" " ``` -------------------------------- ### Install Pip Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html Install python3-pip using apt-get. ```bash sudo apt-get update && sudo apt-get install python3-pip ``` -------------------------------- ### Install wkhtmltopdf Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html Install wkhtmltopdf to enable PDF report generation. ```bash sudo apt-get update && sudo apt-get install wkhtmltopdf ``` -------------------------------- ### Start Triton Server with tritonserver Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/tritonfrontend.html Constructs server options and starts the Triton server instance. ```python import tritonserver # Constructing path to Model Repository model_path = f"server/src/python/examples/example_model_repository" server_options = tritonserver.Options( server_id="ExampleServer", model_repository=model_path, log_error=True, log_warn=True, log_info=True, ) server = tritonserver.Server(server_options).start(wait_until_ready=True) ``` -------------------------------- ### Create and Start a Server Instance Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html Initializes and starts the Triton server instance pointing to the model repository. ```python import tritonserver server = tritonserver.Server(model_repository="/workspace/identity-models") server.start() ``` -------------------------------- ### Build and install backend utilities Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/README.html Commands to create a build directory, configure the build with CMake, and install the backend utilities library to a local directory. ```bash $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install .. $ make install ``` -------------------------------- ### Install dependencies Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/javascript/README.html Install the required Node.js packages. ```bash npm install ``` -------------------------------- ### Download examples via sparse checkout Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/bls_quick_start.html Enable sparse checkout to download only the examples directory containing the bls and add models. ```bash git config core.sparseCheckout true && \ echo 'examples' >> .git/info/sparse-checkout && \ git pull origin main ``` -------------------------------- ### Install Protobuf Compiler Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/rust/triton-client/README.html Install the required Protocol Buffers compiler on macOS or Ubuntu/Debian systems before building. ```bash # macOS brew install protobuf # Ubuntu/Debian apt-get install protobuf-compiler ``` -------------------------------- ### Install Built Wheel Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html Install the generated wheel file using pip. ```bash pip3 install wheels/triton-model-analyzer-*.whl ``` -------------------------------- ### Copy generated stubs to examples folder Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/java/README.html Copy the generated Java and gRPC stubs into the examples project directory. ```bash $ cd .. $ cp -R library/target/generated-sources/protobuf/java/inference examples/src/main/java/inference $ cp -R library/target/generated-sources/protobuf/grpc-java/inference/*.java examples/src/main/java/inference/ ``` -------------------------------- ### Start server with LoRA support Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Command to start the server with a specified model and LoRA separator. ```bash python3 openai_frontend/main.py --lora-separator=_lora_ ... ``` -------------------------------- ### Install tritonclient HTTP support only Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/README.html Installs only the HTTP/REST client library. ```bash $ pip install tritonclient[http] ``` -------------------------------- ### Create and initialize directory Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/quick_start.html Create a new directory and initialize a git repository to prepare for downloading the examples. ```bash mkdir && cd ``` ```bash git init && git remote add -f origin https://github.com/triton-inference-server/model_analyzer.git ``` -------------------------------- ### Start Stable Diffusion Container Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html Starts the container with the diffusion framework. ```bash ./run.sh --framework diffusion ``` -------------------------------- ### Model Configuration Parameters Example Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config.html Example of defining model configuration parameters, showing how to specify lists for sweeping through values. ```yaml model_config_parameters: max_batch_size: [6, 8] dynamic_batching: max_queue_delay_microseconds: [200, 300] instance_group: - kind: KIND_GPU count: [1, 2] ``` -------------------------------- ### Start Triton Server Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/README.html Launch the Triton server and verify it is ready to accept requests. ```bash # inside triton container tritonserver --model-repository $(pwd)/model_repository &> server.log & # confirm server is ready, look for 'HTTP/1.1 200 OK' curl -v localhost:8000/v2/health/ready # detach (CTRL-p CTRL-q) ``` -------------------------------- ### Install Maven Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inprocess_java_api.html Commands to download, extract, and add Maven to the system path for building bindings manually. ```bash $ cd /opt/tritonserver $ wget https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz $ tar zxvf apache-maven-3.8.4-bin.tar.gz $ export PATH=/opt/tritonserver/apache-maven-3.8.4/bin:$PATH ``` -------------------------------- ### Start Services with tritonfrontend Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/tritonfrontend.html Initializes and starts the KServe HTTP, gRPC, and Metrics services using the running server instance. ```python from tritonfrontend import KServeHttp, KServeGrpc, Metrics http_options = KServeHttp.Options(thread_count=5) http_service = KServeHttp(server, http_options) http_service.start() # Default options (if none provided) grpc_service = KServeGrpc(server) grpc_service.start() # Can start metrics service as well metrics_service = Metrics(server) metrics_service.start() ``` -------------------------------- ### Start triton-python-api:r24.01 Container Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html Starts a container and volume mounts the current directory as workspace. ```bash ./run.sh ``` -------------------------------- ### Run the Scala example client Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/java/README.html Execute the SimpleClient for Scala, which includes additional checks for server and model readiness. ```bash $ mvn exec:java -Dexec.mainClass=clients.SimpleClient -Dexec.args=" " ``` -------------------------------- ### is_server_ready Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Contact the inference server and get readiness. ```APIDOC ## is_server_ready ### Description Contact the inference server and get readiness. ### Method is_server_ready ### Parameters - **headers** (_dict_) - Optional dictionary specifying additional HTTP headers to include in the request. - **client_timeout** (_float_) - The maximum end-to-end time, in seconds, the request is allowed to take. The client will abort request and raise InferenceServerExeption with message "Deadline Exceeded" when the specified time elapses. The default value is None which means client will wait for the response from the server. ### Returns True if server is ready, False if server is not ready. ### Return type bool ### Raises - **InferenceServerException** - If unable to get readiness or has timed out. ``` -------------------------------- ### Run the QA Docker container Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/test.html Start the QA container and mount the model repositories from the host. ```bash $ docker run --gpus=all -it --rm -v/tmp:/data/inferenceserver tritonserver_qa ``` -------------------------------- ### Install Java dependencies on Ubuntu Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/README.html Install the required JDK and Maven packages for building the Java client on Ubuntu. ```bash $ apt-get install default-jdk maven ``` -------------------------------- ### Install JDK Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inprocess_java_api.html Command to update package lists and install OpenJDK 11 within the container. ```bash $ apt update && apt install -y openjdk-11-jdk ``` -------------------------------- ### Launch Triton Inference Server (py3-sdk) Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html Starts the Triton Inference Server SDK container for benchmarking. ```bash export RELEASE="24.07" docker run -it --net=host --gpus '"device=0"' nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk ``` -------------------------------- ### Install Python Bindings and Requirements Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Install the necessary Triton server Python bindings and application dependencies. ```bash pip install /opt/tritonserver/python/triton*.whl ``` ```bash git clone https://github.com/triton-inference-server/server.git cd server/python/openai/ pip install -r requirements.txt ``` -------------------------------- ### Triton Server Startup Output Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/quickstart.html Example console output indicating that models are loaded and the server is ready to accept requests. ```text +----------------------+---------+--------+ | Model | Version | Status | +----------------------+---------+--------+ | | | READY | | .. | . | .. | | .. | . | .. | +----------------------+---------+--------+ ... ... ... I1002 21:58:57.891440 62 grpc_server.cc:3914] Started GRPCInferenceService at 0.0.0.0:8001 I1002 21:58:57.893177 62 http_server.cc:2717] Started HTTPService at 0.0.0.0:8000 I1002 21:58:57.935518 62 http_server.cc:2736] Started Metrics Service at 0.0.0.0:8002 ``` -------------------------------- ### Start OpenAI Frontend Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Command to launch the OpenAI frontend server, specifying the model repository and tokenizer. ```bash python3 openai_frontend/main.py --model-repository tests/vllm_models --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct ``` -------------------------------- ### List checkpoint files Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/checkpoints.html Example output showing the contents of the checkpoints directory after a profiling run. ```bash $ ls -l checkpoints -rw-r--r-- 1 root root 11356 May 11 20:00 0.ckpt -rw-r--r-- 1 root root 11356 May 13 19:58 1.ckpt ``` -------------------------------- ### Install tritonclient with HTTP and CUDA support Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/README.html Installs the HTTP client library along with CUDA shared memory support. ```bash $ pip install tritonclient[http, cuda] ``` -------------------------------- ### Install tritonclient via pip Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/README.html Installs the full tritonclient package including both HTTP and GRPC support. ```bash $ pip install tritonclient[all] ``` -------------------------------- ### Profile models command Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/checkpoints.html Example command to initiate profiling for specified models in a repository. ```bash $ model-analyzer profile -m example_model_repo --profile-models example_model_1,example_model_2 2021-05-13 19:57:05.87 INFO[entrypoint.py:98] Starting a local Triton Server... 2021-05-13 19:57:05.92 INFO[server_local.py:64] Triton Server started. 2021-05-13 19:57:09.234 INFO[server_local.py:81] Triton Server stopped. 2021-05-13 19:57:09.235 INFO[analyzer_state_manager.py:118] No checkpoint file found, starting a fresh run. . . . 2021-05-13 19:58:01.625 INFO[analyzer.py:110] Finished profiling. Obtained measurements for models: ['example_model_1', 'example_model_2'] ``` -------------------------------- ### Start Triton Container Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/README.html Pull and run the Triton Inference Server container with GPU support. ```bash export RELEASE= # e.g. to use the release from the end of February of 2023, do `export RELEASE=23.02` docker pull nvcr.io/nvidia/tritonserver:${RELEASE}-py3 docker run --gpus all --rm -it --net host nvcr.io/nvidia/tritonserver:${RELEASE}-py3 ``` -------------------------------- ### is_server_ready(headers=None, query_params=None) Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.http.html Contacts the inference server and gets readiness. Returns True if server is ready, False otherwise. ```APIDOC ## is_server_ready(headers=None, query_params=None) ### Description Contact the inference server and get readiness. ### Method N/A (Python SDK method) ### Endpoint N/A (Python SDK method) ### Parameters - **headers** (dict) - Optional - Dictionary specifying additional HTTP headers to include in the request. - **query_params** (dict) - Optional - URL query parameters to use in network transaction. ### Returns - **bool** - True if server is ready, False if server is not ready. ### Raises - **Exception** - If unable to get readiness. ``` -------------------------------- ### Named Tool Calling Request Setup Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Python setup for making a named tool calling request, defining multiple tools for selection. ```python import json from openai import OpenAI def get_current_weather(city: str, state: str, unit: "str"): return ( "The weather in Dallas, Texas is 85 degrees fahrenheit. It is " "partly cloudly, with highs in the 90's." ) def get_n_day_weather_forecast(city: str, state: str, unit: str, num_days: int): return ( f"The weather in Dallas, Texas is 85 degrees fahrenheit in next {num_days} days." ) available_tools = {"get_current_weather": get_current_weather, "get_n_day_weather_forecast": get_n_day_weather_forecast} ``` -------------------------------- ### Build ONNX Runtime Backend Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/onnxruntime_backend/README.html Standard commands to build and install the ONNX Runtime backend in a local directory. ```bash $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install -DTRITON_BUILD_ONNXRUNTIME_VERSION=1.14.1 -DTRITON_BUILD_CONTAINER_VERSION=23.04 .. $ make install ``` -------------------------------- ### Start Triton SDK Container Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/README.html Pull and run the Triton SDK container to access the performance analyzer tools. ```bash docker pull nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk docker run --gpus all --rm -it --net host nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk ``` -------------------------------- ### Configure Execution Parameters Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/onnxruntime_backend/README.html Example configuration for setting thread counts and execution mode parameters. ```protobuf parameters { key: "intra_op_thread_count" value: { string_value: "0" } } parameters { key: "execution_mode" value: { string_value: "0" } } parameters { key: "inter_op_thread_count" value: { string_value: "0" } } ``` -------------------------------- ### Partial configuration file example Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/dali_backend/README.html You can override values like max_batch_size or rename outputs in the configuration file even if they are defined in the pipeline. ```text name: "dali" backend: "dali" max_batch_size: 128 output [ { name: "DALI_OUTPUT_0" dims: [ 224, 224, 3 ] } ] ``` -------------------------------- ### Install ONNXRuntime with CUDA 11 support Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_navigator/README.html Use this extra index URL to install ONNXRuntime with CUDA 11 support, as the default version is CUDA 12. ```bash --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/ ``` -------------------------------- ### Initialize git repository Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/bls_quick_start.html Start a git repository and add the remote origin for the model analyzer repository. ```bash git init && git remote add -f origin https://github.com/triton-inference-server/model_analyzer.git ``` -------------------------------- ### Start Triton Server with Docker Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/fil_backend/README.html Command to launch the Triton Inference Server container with the model repository mounted. ```bash docker run -p 8000:8000 -p 8001:8001 --gpus all \ -v ${PWD}/model_repository:/models \ nvcr.io/nvidia/tritonserver:23.09-py3 \ tritonserver --model-repository=/models ``` -------------------------------- ### Run NVIDIA CUDA Container Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html Starts a Docker container with NVIDIA GPU support for environment setup. ```bash docker run --rm --ipc=host --runtime=nvidia --gpus '"device=0"' --entrypoint /bin/bash -it nvidia/cuda:12.4.1-devel-ubuntu22.04 ``` -------------------------------- ### Tool Calling Output Example Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Displays the expected output format after executing the tool calling request. ```text function name: get_n_day_weather_forecast function arguments: {"city": "Dallas", "state": "TX", "unit": "fahrenheit", num_days: 1} tool calling result: The weather in Dallas, Texas is 85 degrees fahrenheit in next 1 days. ``` -------------------------------- ### Complete Model Analyzer Configuration Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config.html A complete configuration example showing model repository path and model-specific parameters. ```yaml model_repository: /path/to/model-repository profile_models: model_1: parameters: concurrency: start: 2 stop: 64 step: 8 batch_sizes: 1,2,3 ``` -------------------------------- ### Basic Optuna Search YAML Configuration Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config_search.html A basic example of a YAML configuration file set to perform an Optuna Search. ```yaml model_repository: /path/to/model/repository/ run_config_search_mode: optuna profile_models: - model_A ``` -------------------------------- ### Fixed schedule mode: input_data.json Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/docs/inference_load_modes.html JSON input data file for fixed schedule mode. Each entry has a payload and a timestamp in milliseconds. The timestamp tells Perf Analyzer when to send the request relative to benchmark start. This example sends two requests at 1000ms and 2000ms. ```json # input_data.json { "data": [ { "payload": [{ "model": "facebook/opt-125m", "messages": [{"role": "user","content": "What is the capital of France?"}], "max_completion_tokens": 1 }], "timestamp": [1000] }, { "payload": [{ "model": "facebook/opt-125m", "messages": [{"role": "user","content": "What is 1+1?"}], "max_completion_tokens": 1 }], "timestamp": [2000] } ] } ``` -------------------------------- ### Prepare proto files Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/javascript/README.html Create a proto directory and copy the required protobuf files from the cloned repository. ```bash mkdir proto cp common-repo/protobuf/*.proto ./proto/ ``` -------------------------------- ### Run Go gRPC Client Example Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/go/README.html Steps to prepare the model repository, launch the Triton server via Docker, and run the Go client. ```bash # Clone the repo containing the example model, from within client/src/grpc_generated/go/. git clone https://github.com/triton-inference-server/server.git # Setup "simple" model from example model_repository cd server/docs/examples ./fetch_models.sh # Launch (detached) Triton docker run -d -p8000:8000 -p8001:8001 -p8002:8002 -it -v $(pwd)/model_repository:/models nvcr.io/nvidia/tritonserver:22.11-py3 tritonserver --model-store=/models # Use client cd ../../../ go run grpc_simple_client.go ``` -------------------------------- ### Launch OpenAI Server Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Start the OpenAI server. Adjust the --tokenizer argument based on the specific model being used. ```bash python3 openai_frontend/main.py --model-repository path/to/models --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct ``` -------------------------------- ### Deploy Model Analyzer with Helm Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/kubernetes_deploy.html Command to install the Model Analyzer helm chart from the root directory. ```bash ~/model_analyzer$ helm install model-analyzer helm-chart NAME: model-analyzer LAST DEPLOYED: Mon Dec 7 15:09:14 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None ``` -------------------------------- ### Triton Server Model Loading Output Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html Example console output confirming that the batching and nonbatching models have loaded successfully. ```text I1215 23:46:00.250284 68 server.cc:589] +-------------+---------+--------+ | Model | Version | Status | +-------------+---------+--------+ | batching | 1 | READY | | nonbatching | 1 | READY | +-------------+---------+--------+ ``` -------------------------------- ### Run Triton with Minimal Backend Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html Command to start the Triton server using a Docker image with the minimal backend loaded. ```bash $ docker run --rm -it --net=host -v/path/to/model_repos/minimal_models:/models tritonserver --model-repository=/models ``` -------------------------------- ### Running Tests Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Commands to install test requirements and execute tests for the OpenAI integration. ```bash cd /opt/tritonserver/python/openai/ pip install -r requirements-test.txt pytest -v tests/ ``` -------------------------------- ### Install Model Analyzer via Pip Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html Install the Model Analyzer package. ```bash pip3 install triton-model-analyzer ``` -------------------------------- ### Run tritonserver_buildbase container for Triton Core Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/build.html Starts the build container with the host server source directory mapped to /server. ```bash $ docker run -it --rm -v/home/me/server:/server tritonserver_buildbase bash ``` -------------------------------- ### Launch OpenAI Frontend with Tool Call Parser Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html Command to start the OpenAI frontend with a specific tool-call parser enabled. ```bash python3 openai_frontend/main.py \ --model-repository tests/vllm_models \ --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct \ --tool-call-parser llama3 ``` -------------------------------- ### Install TensorRT-LLM Dependencies and Package Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html Installs system dependencies and the specific version of TensorRT-LLM. ```bash apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev git git-lfs ``` ```bash pip3 install tensorrt_llm==0.11.0 --extra-index-url https://pypi.nvidia.com ``` ```bash python3 -c "import tensorrt_llm" ``` -------------------------------- ### Example JSON Response Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html An example of the JSON response received from the chat completion request. ```json { "id": "cmpl-704c758c-8a84-11ef-b106-107c6149ca79", "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "It looks like you're testing the system!", "tool_calls": null, "role": "assistant", "function_call": null }, "logprobs": null } ], "created": 1728948689, "model": "llama-3-8b-instruct", "system_fingerprint": null, "object": "chat.completion", "usage": null } ``` -------------------------------- ### Benchmark OpenAI-compatible server with Perf Analyzer Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/docs/benchmarking.html This workflow demonstrates setting up a vLLM server, creating input data, and running Perf Analyzer with the --service-kind=openai flag. It includes the expected console output for the benchmark run. ```bash # get chat template required for facebook/opt-125m model curl -o template_chatml.jinja https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/examples/template_chatml.jinja # start vllm, an OpenAI API-compatible server vllm serve facebook/opt-125m --chat-template=template_chatml.jinja > server.log 2>&1 & # wait for server to be ready while [ "$(curl -s -o /dev/null -w "%{http_code}" localhost:8000/v1/models)" != "200" ]; do sleep 1; done # create simple input data JSON for Perf Analyzer cat > input_data.json <-py3-sdk ``` ```bash $ docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:-py3-sdk ``` -------------------------------- ### Triton Server Console Output Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html Example of the console output indicating that the batching model has loaded successfully. ```text I1215 23:46:00.250284 68 server.cc:589] +-------------+---------+--------+ | Model | Version | Status | +-------------+---------+--------+ | batching | 1 | READY | +-------------+---------+--------+ ``` -------------------------------- ### Configure Model Weighting in YAML Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config_search.html Example configuration showing how to assign weights to models to prioritize specific objectives during multi-model search. ```yaml model_repository: /path/to/model/repository/ run_config_profile_models_concurrently_enable: true profile_models: model_A: weighting: 3 objectives: perf_latency_p99: 1 model_B: weighting: 1 objectives: perf_throughput: 1 ``` -------------------------------- ### Quick Search YAML Configuration Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config_search.html An example YAML configuration for Model Analyzer performing a Quick Search with constrained search parameters. ```yaml model_repository: /path/to/model/repository/ run_config_search_mode: quick run_config_search_max_instance_count: 4 run_config_search_max_concurrency: 16 run_config_search_min_model_batch_size: 8 profile_models: - model_A ``` -------------------------------- ### Run Triton Docker container Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inprocess_java_api.html Command to start the Triton Inference Server container with GPU support and a volume mount. ```bash $ docker run -it --gpus=all -v ${pwd}:/workspace nvcr.io/nvidia/tritonserver:-py3 bash ``` -------------------------------- ### Clone the Triton Tutorials Repository Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html Clone the repository containing the Python API tutorials and navigate to the project directory. ```bash git clone https://github.com/triton-inference-server/tutorials.git cd tutorials/Triton_Inference_Server_Python_API ``` -------------------------------- ### Triton Model Configuration (config.pbtxt) Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/dali_backend/README.html Example configuration file for the DALI backend, specifying input and output tensors and the backend type. ```text name: "dali" backend: "dali" max_batch_size: 256 input [ { name: "DALI_INPUT_0" data_type: TYPE_UINT8 dims: [ -1 ] } ] output [ { name: "DALI_OUTPUT_0" data_type: TYPE_UINT8 dims: [ 224, 224, 3 ] } ] ``` -------------------------------- ### Quick Search Mode YAML Configuration Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config_search.html Example YAML configuration for Model Analyzer to perform a Quick Search on a specified model. ```yaml model_repository: /path/to/model/repository/ run_config_search_mode: quick profile_models: - model_A ``` -------------------------------- ### Basic Triton client example: connect, health check, metadata, and inference Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/rust/triton-client/README.html Shows the core workflow: connect to Triton at localhost:8001, verify server liveness and readiness, query server metadata, build an inference request with InferInput and InferRequestBuilder, run inference, and read the output as f32. Uses the prelude for convenience. ```rust use triton_client::prelude::*; #[tokio::main] async fn main() -> triton_client::error::Result<()> { // Connect to Triton let client = TritonClient::connect("http://localhost:8001").await?; // Check health assert!(client.is_server_live().await?); assert!(client.is_server_ready().await?); // Query server metadata let metadata = client.server_metadata().await?; println!("Server: {} v{}", metadata.name, metadata.version); // Build an inference request let input = InferInput::new("input0", vec![1, 16], DataType::Fp32) .with_data_f32(&[0.0; 16]); let request = InferRequestBuilder::new("my_model") .model_version("1") .input(input) .output("output0") .build(); // Run inference let response = client.infer(request).await?; let output = response.output_as_f32(0)?; println!("Output: {:?}", output); Ok(()) } ``` -------------------------------- ### Profile model with quick search mode Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/cli.html Run quick search on a model called add_sub located in examples/quick-start. ```bash $ model-analyzer profile -m examples/quick-start --profile-models add_sub --run-config-search-mode quick ``` -------------------------------- ### InferInput tensor builder examples Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/rust/triton-client/README.html Examples of using the InferInput builder to create input tensors for numeric, string/bytes, and raw data types. ```rust // Numeric data InferInput::new("input", vec![1, 4], DataType::Fp32).with_data_f32(&[1.0, 2.0, 3.0, 4.0]) InferInput::new("input", vec![1, 4], DataType::Int64).with_data_i64(&[1, 2, 3, 4]) // String / bytes data InferInput::new("text", vec![1, 2], DataType::Bytes).with_data_bytes(&[b"hello", b"world"]) // Raw data (FP16, BF16, or any format) InferInput::new("input", vec![1, 2], DataType::Fp16).with_data_raw(raw_bytes) ``` -------------------------------- ### Build DALI Backend from source Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/dali_backend/README.html Commands to clone the repository with submodules, create a build directory, and compile the backend using CMake. ```bash git clone --recursive https://github.com/triton-inference-server/dali_backend.git cd dali_backend mkdir build cd build cmake .. make ``` -------------------------------- ### Minimal Triton Java Inference Example Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/java/README.html A basic example demonstrating how to define model inputs, request outputs, and perform synchronous inference using the InferenceServerClient. ```java package triton.client.example; import java.util.Arrays; import java.util.List; import com.google.common.collect.Lists; import triton.client.InferInput; import triton.client.InferRequestedOutput; import triton.client.InferResult; import triton.client.InferenceServerClient; import triton.client.pojo.DataType; public class MinExample { public static void main(String[] args) throws Exception { boolean isBinary = true; InferInput inputIds = new InferInput("input_ids", new long[] {1L, 32}, DataType.INT32); int[] inputIdsData = new int[32]; Arrays.fill(inputIdsData, 1); // fill with some data. inputIds.setData(inputIdsData, isBinary); InferInput inputMask = new InferInput("input_mask", new long[] {1, 32}, DataType.INT32); int[] inputMaskData = new int[32]; Arrays.fill(inputMaskData, 1); inputMask.setData(inputMaskData, isBinary); InferInput segmentIds = new InferInput("segment_ids", new long[] {1, 32}, DataType.INT32); int[] segmentIdsData = new int[32]; Arrays.fill(segmentIdsData, 0); segmentIds.setData(segmentIdsData, isBinary); List inputs = Lists.newArrayList(inputIds, inputMask, segmentIds); List outputs = Lists.newArrayList(new InferRequestedOutput("logits", isBinary)); InferenceServerClient client = new InferenceServerClient("0.0.0.0:8000", 5000, 5000); InferResult result = client.infer("roberta", inputs, outputs); float[] logits = result.getOutputAsFloat("logits"); System.out.println(Arrays.toString(logits)); } } ``` -------------------------------- ### InferenceServerException.status() Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Get the status of the exception. ```APIDOC ## InferenceServerException.status() ### Description Get the status of the exception. ### Returns - **status** (str) - The status of the exception. ### Return Type str ``` -------------------------------- ### InferenceServerException.message() Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Get the exception message. ```APIDOC ## InferenceServerException.message() ### Description Get the exception message. ### Returns - **message** (str) - The message associated with this exception, or None if no message. ### Return Type str ``` -------------------------------- ### Configure Build with CMake Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/README.html Commands to create a build directory and configure the client build using CMake. ```bash $ mkdir build $ cd build $ cmake -DVCPKG_TARGET_TRIPLET=x64-windows -DCMAKE_TOOLCHAIN_FILE='/vcpkg/scripts/buildsystems/vcpkg.cmake' -DCMAKE_INSTALL_PREFIX=install -DTRITON_ENABLE_CC_GRPC=ON -DTRITON_ENABLE_PYTHON_GRPC=ON -DTRITON_ENABLE_GPU=OFF -DTRITON_ENABLE_EXAMPLES=ON -DTRITON_ENABLE_TESTS=ON .. ``` -------------------------------- ### is_server_live Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Contact the inference server and get liveness. ```APIDOC ## is_server_live ### Description Contact the inference server and get liveness. ### Method is_server_live ### Parameters - **headers** (_dict_) - Optional dictionary specifying additional HTTP headers to include in the request. - **client_timeout** (_float_) - The maximum end-to-end time, in seconds, the request is allowed to take. The client will abort request and raise InferenceServerExeption with message "Deadline Exceeded" when the specified time elapses. The default value is None which means client will wait for the response from the server. ### Returns True if server is live, False if server is not live. ### Return type bool ### Raises - **InferenceServerException** - If unable to get liveness or has timed out. ``` -------------------------------- ### InferRequestedOutput.name Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.http.html Gets the name of the output associated with this object. ```APIDOC ## InferRequestedOutput.name ### Description Get the name of output associated with this object. ### Method name ### Returns The name of output (str) ``` -------------------------------- ### InferInput.shape Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.http.html Gets the shape of the input associated with this object. ```APIDOC ## InferInput.shape ### Description Get the shape of input associated with this object. ### Method shape ### Returns The shape of input (list) ``` -------------------------------- ### Go Client Sample Output Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client/src/grpc_generated/go/README.html Expected output when running the grpc_simple_client.go script. ```text $ go run grpc_simple_client.go FLAGS: {simple 1 localhost:8001} Triton Health - Live: true Triton Health - Ready: true name:"simple" versions:"1" platform:"tensorflow_graphdef" inputs:{name:"INPUT0" datatype:"INT32" shape:-1 shape:16} inputs:{name:"INPUT1" datatype:"INT32" shape:-1 shape:16} outputs:{name:"OUTPUT0" datatype:"INT32" shape:-1 shape:16} outputs:{name:"OUTPUT1" datatype:"INT32" shape:-1 shape:16} Checking Inference Outputs -------------------------- 0 + 1 = 1 0 - 1 = -1 1 + 1 = 2 1 - 1 = 0 2 + 1 = 3 2 - 1 = 1 3 + 1 = 4 3 - 1 = 2 4 + 1 = 5 4 - 1 = 3 5 + 1 = 6 5 - 1 = 4 6 + 1 = 7 6 - 1 = 5 7 + 1 = 8 7 - 1 = 6 8 + 1 = 9 8 - 1 = 7 9 + 1 = 10 9 - 1 = 8 10 + 1 = 11 10 - 1 = 9 11 + 1 = 12 11 - 1 = 10 12 + 1 = 13 12 - 1 = 11 13 + 1 = 14 13 - 1 = 12 14 + 1 = 15 14 - 1 = 13 15 + 1 = 16 15 - 1 = 14 ``` -------------------------------- ### InferRequestedOutput.name() Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Get the name of output associated with this object. ```APIDOC ## InferRequestedOutput.name() ### Description Get the name of output associated with this object. ### Method name() ### Returns The name of output ### Return Type str ``` -------------------------------- ### InferInput.shape() Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Get the shape of input associated with this object. ```APIDOC ## InferInput.shape() ### Description Get the shape of input associated with this object. ### Method shape() ### Returns The shape of input ### Return Type list ``` -------------------------------- ### tritonclient.grpc.InferInput.name Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Get the name of input associated with this object. ```APIDOC ## tritonclient.grpc.InferInput.name ### Description Get the name of input associated with this object. ### Returns - **str** - The name of input ``` -------------------------------- ### tritonclient.grpc.InferInput.datatype Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Get the datatype of input associated with this object. ```APIDOC ## tritonclient.grpc.InferInput.datatype ### Description Get the datatype of input associated with this object. ### Returns - **str** - The datatype of input ``` -------------------------------- ### List Models Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/python_readme.html Lists available models for the diffusion setup. ```python server.models() ``` ```text {('stable_diffusion', 1): {'name': 'stable_diffusion', 'version': 1, 'state': 'READY'}, ('text_encoder', 1): {'name': 'text_encoder', 'version': 1, 'state': 'READY'}, ('vae', 1): {'name': 'vae', 'version': 1, 'state': 'READY'}} ``` -------------------------------- ### View Model Analyzer Help Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/cli.html Use the -h or --help flag to display the command line interface description. ```bash $ model-analyzer -h ``` -------------------------------- ### InferenceServerException.debug_details Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.grpc.html Get the detailed information about the exception for debugging purposes. ```APIDOC ## InferenceServerException.debug_details ### Description Get the detailed information about the exception for debugging purposes. ### Returns Returns the exception details. ### Return Type str ``` -------------------------------- ### Upgrade Pip Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/install.html Upgrade pip if dependency installation errors occur. ```bash pip3 install --upgrade pip ``` -------------------------------- ### Execute recommended_client Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/examples/README.html Run the recommended_client Python script to send inference requests to the recommended backend. ```bash $ clients/recommended_client ``` -------------------------------- ### serialized_byte_size(tensor_value) Source: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_reference/tritonclient/tritonclient.utils.html Gets the underlying number of bytes for a numpy ndarray. ```APIDOC ## serialized_byte_size(tensor_value) ### Description Gets the underlying number of bytes for a numpy ndarray. ### Parameters - **tensor_value** - The numpy ndarray. ### Returns The number of bytes. ```