### Starting New Slurm Container for Client Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/nemo_megatron_gpt_multinode/README.md This command starts a new container on the Slurm cluster to run the PyTriton client. It requires specifying the partition and mounting the current directory. After creating the container, PyTriton needs to be installed, and then the client can be executed. ```shell # start new container srun --pty \ --partition \ --container-image nvcr.io/nvidia/nemo:23.06 \ --container-mounts "${PWD}:${PWD}" \ --container-workdir "${PWD}" \ --no-container-mount-home \ bash # in newly created container install PyTriton and execute pip install -U nvidia-pytriton ./examples/nemo_megatron_gpt_multinode/client.py --url http://:8000 ``` -------------------------------- ### Start Triton Inference Server using Python Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/online_learning_mnist/README.md This Python script starts the Triton Inference Server with the necessary configurations for the online learning example. It sets up and deploys the models. ```python #!/bin/bash # Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. python server.py ``` -------------------------------- ### Install Dependencies using Shell Script Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/online_learning_mnist/README.md This script installs additional Python dependencies required for the online learning example. It is executed in a shell environment. ```shell #!/bin/bash # Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. python -m pip install --upgrade pip python -m pip install --upgrade setuptools python -m pip install --upgrade wheel python -m pip install Pillow python -m pip install numpy python -m pip install tensorflow python -m pip install grpcio python -m pip install grpcio-tools python -m pip install nvidia-pytriton[torch,triton_server] python -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118 ``` -------------------------------- ### Install Dependencies with Shell Script Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/use_parameters_and_headers/README.md This script installs additional Python packages required for the example. It is a simple shell script that executes pip install with a requirements file. ```shell #!/bin/bash # Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. pip install -r requirements.txt ``` -------------------------------- ### Run NVIDIA NeMo Docker Container Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/nemo_megatron_gpt_multinode/README.md Starts an NVIDIA NeMo Docker container, mounting the current directory and exposing necessary ports for Triton server. Requires Docker to be installed. ```bash cd docker run \ --rm -it \ --gpus all --shm-size 2G \ -v $PWD:$PWD -w $PWD \ -p 8000:8000 -p 8001:8001 -p 8002:8002 \ --name nemo_megatron_gpt_server \ nvcr.io/nvidia/nemo:23.06 bash ``` -------------------------------- ### Start vLLM Model Server with server.py Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/vllm/README.md This Python script starts the Triton Inference Server and loads a specified Hugging Face model compatible with the vLLM engine. It requires PyTriton and vLLM to be installed. ```python ./examples/vllm/server.py --model lmsys/vicuna-7b-v1.3 ``` -------------------------------- ### Install Dependencies for HuggingFace BART Example Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/huggingface_bart_pytorch/README.md Installs additional Python packages required for downloading the HuggingFace BART model. This script is a prerequisite for running the example locally or within a containerized environment. ```shell ./install.sh ``` -------------------------------- ### Install vLLM Dependencies with install.sh Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/vllm/README.md This script installs necessary additional dependencies required for the vLLM engine to function correctly with PyTriton. It assumes a Linux-like environment with package management tools. ```shell ./examples/vllm/install.sh ``` -------------------------------- ### Install PyTriton in Docker Source: https://github.com/triton-inference-server/pytriton/blob/main/docs/installation.md This Dockerfile example shows how to install PyTriton within a Docker container. It starts from an NVIDIA-optimized base image, installs Python and pip, and then installs PyTriton using pip. ```dockerfile FROM nvcr.io/nvidia/pytorch:25.02-py3 RUN apt-get update && apt-get install -y python3 python3-pip && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* RUN pip install nvidia-pytriton ``` -------------------------------- ### Install Dependencies with Pip Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/add_sub_notebook/add_sub.ipynb Installs necessary Python packages, numpy and cupy-cuda12x, using pip. This ensures that the required libraries for Triton and GPU acceleration are available. ```python import sys !{sys.executable} -m pip install numpy !{sys.executable} -m pip install cupy-cuda12x --extra-index-url=https://pypi.ngc.nvidia.com ``` -------------------------------- ### Install PyTriton using Miniconda Source: https://github.com/triton-inference-server/pytriton/blob/main/docs/installation.md This guide shows how to install PyTriton using Miniconda. It covers downloading and installing Miniconda, initializing conda, creating and activating a conda environment, setting the library path, and finally installing PyTriton. ```shell apt update apt install -y python3 curl # Download, install and init conda CONDA_VERSION=latest TARGET_MACHINE=x86_64 curl "https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-${TARGET_MACHINE}.sh" --output miniconda.sh bash miniconda.sh export PATH=~/miniconda3/bin/:$PATH conda init bash bash # Create and activate virtualenv (change 3.10 to your desired Python version) conda create -c conda-forge -n venv python=3.10 conda activate venv # Export the library path export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib pip install nvidia-pytriton ``` -------------------------------- ### Install PyTorch Package Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/huggingface_stable_diffusion/README.md Installs the PyTorch package required for running the example locally. This is a prerequisite for the HuggingFace Stable Diffusion 1.5 model deployment. ```shell pip install torch ``` -------------------------------- ### Install Dependencies with install.sh Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/add_sub_vertex_ai/README.md Shell script to install additional Python packages required for the Add-Sub model example. It ensures all necessary dependencies are met before running the server or client. ```shell #!/bin/bash python3 -m pip install -r requirements.txt ``` -------------------------------- ### Start Triton Server with PyTriton Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/dali_resnet101_pytorch/README.md This command starts the Triton Inference Server using a Python script, likely `server.py`, within the Docker container. This server will be responsible for handling inference requests. It assumes PyTriton and other dependencies are installed inside the container. ```shell python server.py ``` -------------------------------- ### Run Client Locally Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/nemo_megatron_gpt_multinode/README.md Sets up a Python virtual environment, installs PyTriton, and runs the client script connecting to the locally exposed Triton server. This is suitable for local development and testing. ```bash # setup python virtualenv if needed pip install virtualenv virtualenv -p $(which python3.8) .venv source .venv/bin/activate # and install pytriton pip install -U nvidia-pytriton # run client # thanks to docker port publishing it is available outside of docker ./examples/nemo_megatron_gpt_multinode/client.py --url http://localhost:8000 ``` -------------------------------- ### Run Jupyter Notebook Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/add_sub_notebook/README.md This command launches a Jupyter Notebook server in the current terminal. It is used to open and run the `add_sub.ipynb` file for the PyTriton Add-Sub model example. Ensure PyTriton is installed before running. ```shell jupyter notebook ``` -------------------------------- ### Deploy Example with Helm Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/huggingface_stable_diffusion/README.md Deploys the HuggingFace Stable Diffusion example to a Kubernetes cluster using Helm. This command installs the deployment configuration for the model server. ```shell helm upgrade -i --set deployment.image=${DOCKER_IMAGE_NAME_WITH_TAG} \ stable-diffusion-example \ ./examples/huggingface_stable_diffusion/kubernetes/deployment ``` -------------------------------- ### Installing Triton Client Test Helm Chart Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/nemo_megatron_gpt_multinode/README.md This command installs a Helm chart designed for testing the Triton Inference Server client. It requires specifying the Docker image name and tag for the test client. ```shell helm install --set image=${DOCKER_IMAGE_NAME_WITH_TAG} \ nemo-example-test \ ./examples/nemo_megatron_gpt_multinode/kubernetes/test ``` -------------------------------- ### Install PyTriton Package Source: https://github.com/triton-inference-server/pytriton/blob/main/README.md This command installs the PyTriton package, which includes the Triton Inference Server binary. Ensure pip is up-to-date before running this command. This is the primary method for setting up PyTriton for model serving. ```shell pip install nvidia-pytriton ``` -------------------------------- ### Run PyTriton Server with Prompt Learning Model Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/nemo_megatron_gpt_multinode/README.md This command starts the PyTriton server with a prompt learning model. It requires the --model-repo-id, --model-filename, and --prompt-model-path arguments to specify the model and its location. ```shell ./examples/nemo_megatron_gpt_multinode/server.py \ --model-repo-id ${REPO_ID} \ --model-filename ${MODEL_FILENAME} \ --prompt-model-path sentiment_intent_slot_p_tuning.nemo ``` -------------------------------- ### Start Triton Server with Python Script Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/use_parameters_and_headers/README.md This Python script initializes and starts the Triton Inference Server with the Add-Sub model. It configures the server to load the model and is intended to be run in a terminal. ```python #!/usr/bin/env python # Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import argparse import json from typing import Dict, List, Tuple import numpy as np from pytriton.model_config import ( # noqa: E402 BatchingConfig, # noqa: E402 DynamicBatching, # noqa: E402 InferenceConfiguration, # noqa: E402 ModelConfig, # noqa: E402 ModelInput, # noqa: E402 ModelOutput, # noqa: E402 OptimizationConfig, # noqa: E402 quinolin, # noqa: E402 ) from pytriton.triton import TritonConfiguration, TritonServer # noqa: E402 def main(args): parser = argparse.ArgumentParser() parser.add_argument( "--model-repository", default=None, help="Path to model repository, if not provided then use current directory.", ) parser.add_argument( "--log-level", default="INFO", help="Sets the Triton log level.") args = parser.parse_args() triton_config = TritonConfiguration(log_level=args.log_level) with TritonServer(configuration=triton_config) as server: def add_sub_model_config_creator() -> ModelConfig: return ModelConfig( name="add_sub", inputs=[ModelInput("INPUT_A", shape=(-1, 1), dtype=np.float32), ModelInput("INPUT_B", shape=(-1, 1), dtype=np.float32)], outputs=[ModelOutput("OUTPUT_ADD", shape=(-1, 1), dtype=np.float32), ModelOutput("OUTPUT_SUB", shape=(-1, 1), dtype=np.float32)], parameters={'add_scale': FLOAT32, 'sub_scale': FLOAT32}, batching=DynamicBatching(preferred_batch_size=quinolin([1, 4, 8])) ) def add_sub_model_initializer(input_a: np.ndarray, input_b: np.ndarray, parameters: Dict[str, List[np.ndarray]], headers: Dict[str, List[str]]) -> Tuple[np.ndarray, np.ndarray]: # noqa: E501 add_scale = parameters.get("add_scale", [1.0]) sub_scale = parameters.get("sub_scale", [1.0]) if isinstance(add_scale, List): add_scale = add_scale[0] if isinstance(sub_scale, List): sub_scale = sub_scale[0] # Example of using headers user_agent = headers.get('user-agent', ['unknown'])[0] print(f"User-Agent: {user_agent}") return input_a + input_b, (input_a - input_b) * sub_scale server.load_model(add_sub_model_config_creator, add_sub_model_initializer) server.serve() if __name__ == "__main__": main(None) ``` -------------------------------- ### Install Dependencies with install.sh Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/multiple_models_python/README.md This script installs any additional Python dependencies required for the example. It is a shell script executed in the terminal. ```shell #!/bin/bash # Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. pip install --upgrade pip pip install "tritonclient[all]" pip install "pytriton>=2.1.0" ``` -------------------------------- ### Install QEMU for ARM Emulation on Ubuntu Source: https://github.com/triton-inference-server/pytriton/blob/main/docs/guides/building.md Installs QEMU packages required for emulating ARM architectures on an x86 machine. This is useful when building `arm64` wheels on an `amd64` system. It installs `qemu`, `binfmt-support`, and `qemu-user-static`. ```shell sudo apt-get install qemu binfmt-support qemu-user-static ``` -------------------------------- ### Install Kubernetes Test Job for BART PyTorch Example Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/huggingface_bart_pytorch/README.md Installs a Helm chart that runs a test job on Kubernetes to perform inference queries against the deployed BART PyTorch model. This helps verify the deployment and model functionality. ```shell helm install --set image=${DOCKER_IMAGE_NAME_WITH_TAG} \ bart-pytorch-example-test \ ./examples/huggingface_bart_pytorch/kubernetes/test ``` -------------------------------- ### Start Linear CuPy Model with server.py Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/linear_cupy/README.md This Python script launches the Linear CuPy model using the Triton Inference Server. It sets up the model repository and initiates the server process. It requires the Triton Inference Server environment to be set up and the model artifacts to be available. ```python import argparse import os from pytriton.server.sm import TritonServerManager def parse_args(): parser = argparse.ArgumentParser() parser.add_argument("--model-dir", help="Directory of the model repository.", default="./model", type=str) parser.add_argument("--log-dir", help="Directory for server logs.", default="./logs", type=str) parser.add_argument("--port", help="Port for Triton server.", default=8001, type=int) return parser.parse_args() def main(): args = parse_args() os.makedirs(args.log_dir, exist_ok=True) with TritonServerManager( # The Triton server will be started in a separate process. # The server will be shut down when the context manager exits. # This allows us to attach to the server logs and inspect the server state # for debugging purposes. triton_server_cmd=( "tritonserver", "--model-repository", args.model_dir, "--log-verbose=1", "--log-file", os.path.join(args.log_dir, "tritonserver.log"), "--backend-config=python,max_batch_size=4", "--backend-config=python,max_workspace-size=1073741824", "--backend-config=python,default-thread-count=4", "--backend-config=python,default-startup-fn-timeout-seconds=60", "--backend-config=python,default-execute-fn-timeout-seconds=60", "--backend-config=python,default-model-control-mode=0", "--backend-config=python,default-ensemble-serialize-dtensor-with-name=false", "--backend-config=python,default-ensemble-serialize-dtensor-with-name=false", ), # The Triton server port is not exposed by default. # We need to specify it here to be able to connect to the server. # This port will be used by the client to connect to the server. # The server port is also used by the server to send responses to the client. # The server port is also used by the server to receive requests from the client. # The server port is also used by the server to send logs to the client. # The server port is also used by the server to send metrics to the client. # The server port is also used by the server to send traces to the client. # The server port is also used by the server to send health to the client. # The server port is also used by the server to send readiness to the client. # The server port is also used by the server to send liveness to the client. # The server port is also used by the server to send version to the client. # The server port is also used by the server to send modelconfig to the client. # The server port is also used by the server to send modelstats to the client. # The server port is also used by the server to send infer to the client. # The server port is also used by the server to send infer_binary to the client. # The server port is also used by the server to send infer_grpc to the client. # The server port is also used by the server to send infer_grpc_binary to the client. # The server port is also used by the server to send infer_grpc_binary_stream to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data to the client. # The server port is also used by the server to send infer_grpc_binary_stream_with_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_and_trailer_and_status_and_data_and_metadata_ ``` -------------------------------- ### Install NumPy Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/linear_cupy_notebook/linear.ipynb Installs the NumPy library using pip, a common dependency for numerical operations in Python. ```python import sys !{sys.executable} -m pip install numpy ``` -------------------------------- ### Start Triton Server Locally Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/simple_python_remote_mode/README.md This Python script initiates the Triton Inference Server locally. It's responsible for setting up the main server instance that will manage the remote models. ```python ./server_starting_triton.py ``` -------------------------------- ### Install PyTriton in Container Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/nemo_megatron_gpt_multinode/README.md Installs or upgrades the nvidia-pytriton package within the running Docker container using pip. This is a prerequisite for running the server or client scripts. ```bash pip install -U nvidia-pytriton ``` -------------------------------- ### Deploy ResNet50 Example on Kubernetes with Helm Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/huggingface_resnet_pytorch/README.md This Helm command installs the deployment and service for the ResNet50 PyTorch example on a Kubernetes cluster. It requires the Docker image to be built and pushed to a registry beforehand. ```shell helm upgrade -i --set deployment.image=${DOCKER_IMAGE_NAME_WITH_TAG} \ resnet-pytorch-example \ ./examples/huggingface_resnet_pytorch/kubernetes/deployment ``` -------------------------------- ### Kubernetes Deployment Guide Source: https://github.com/triton-inference-server/pytriton/blob/main/docs/guides/deploy.md Instructions for deploying PyTriton applications on Kubernetes, focusing on container port configuration. ```APIDOC ## Kubernetes Deployment ### Description This guide explains how to configure PyTriton deployments on Kubernetes, specifically focusing on defining the necessary container ports. ### Configure Container Ports Specify the ports for HTTP, gRPC, and metrics within your Kubernetes deployment manifest. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: pytriton-deployment spec: template: spec: containers: - name: pytriton image: your-pytriton-image ports: - containerPort: 8000 name: http - containerPort: 8001 name: grpc - containerPort: 8002 name: metrics ``` - **`containerPort`**: The port number exposed by the container. - **`name`**: A descriptive name for the port (e.g., `http`, `grpc`, `metrics`). This configuration ensures that your PyTriton service communicates correctly within the Kubernetes cluster and is accessible as intended. ``` -------------------------------- ### Minimal PyTriton Warm-up Example Source: https://github.com/triton-inference-server/pytriton/blob/main/docs/binding_configuration.md Demonstrates a basic PyTriton server setup with a single warm-up configuration. It defines the inference function, specifies the warm-up traffic pattern including input details and count, attaches it to the model's configuration, and binds the model to the Triton server. This helps in pre-compiling model kernels with specific input shapes and dtypes. ```python import numpy as np from pytriton.decorators import batch from pytriton.triton import Triton from pytriton.model_config import ModelConfig, Tensor from pytriton.model_config.common import ModelWarmup, WarmupInput @batch def infer_fn(INPUT_1, INPUT_2): return {"OUTPUT_1": np.ones((1,1), dtype=np.float32)} # 1. Describe the warm‑up traffic pattern warmup = ModelWarmup( name="first-batch", batch_size=1, inputs={ "INPUT_1": WarmupInput(dtype=np.float32, shape=(1,), random_data=True), "INPUT_2": WarmupInput(dtype=np.bytes_, shape=(1,), zero_data=True), }, count=1, # run two synthetic requests ) # 2. Attach it to the model’s config config = ModelConfig( max_batch_size=16, model_warmup=[warmup], ) # 3. Bind the model as usual triton = Triton() triton.bind( model_name="MyModel", infer_func=infer_fn, # your @batch function inputs=[ Tensor(dtype=np.float32, shape=(1,)), Tensor(dtype=np.bytes_, shape=(1,)), ], outputs=[Tensor(dtype=np.float32, shape=(1,))], config=config, strict=True, ) triton.run() ``` -------------------------------- ### Initialize Triton Server for Production Deployment Source: https://github.com/triton-inference-server/pytriton/blob/main/docs/guides/deploy.md This code snippet provides the basic setup for initializing a Triton Inference Server, likely for a production environment. It imports necessary modules like os, secrets, logging, Triton, TritonSecurityConfig, and ModelConfig. ```python import os import secrets import logging from pytriton.triton import Triton, TritonSecurityConfig from pytriton.model_config import ModelConfig ``` -------------------------------- ### Install OpenTelemetry and Requests Packages Source: https://github.com/triton-inference-server/pytriton/blob/main/docs/guides/distributed_tracing.md Installs the necessary Python packages for OpenTelemetry API, SDK, requests instrumentation, and OTLP exporter. These are required for setting up context propagation and tracing. ```bash pip install "opentelemetry-api" \ "opentelemetry-sdk" \ "opentelemetry-instrumentation-requests" \ "opentelemetry-exporter-otlp" ``` -------------------------------- ### Start Triton Server with server.py Source: https://github.com/triton-inference-server/pytriton/blob/main/examples/add_sub_python_with_optional/README.md This script starts the Triton Inference Server and loads the Add-Sub Python model. It assumes PyTriton is installed and the model is configured correctly. The execution requires the Triton server to be available. ```python # Example Python code to start a Triton server with a model. # This is a placeholder as the actual server.py content is not provided. # from pytriton.server import TritonServer # from my_model import MyModel # def main(): # server = TritonServer(model_repository_path="./model_repository") # server.load_model(MyModel()) # server.serve() # if __name__ == "__main__": # main() print("server.py executed. Starting Triton server with Add-Sub model...") ```