### Install Dependencies for Pipeline Optimization Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/quick_start.md Installs the necessary Python packages, including transformers, diffusers, and torch, required to run the Stable Diffusion pipeline optimization example. ```shell pip install transformers diffusers torch ``` -------------------------------- ### Install Environment for JAX Linear Model Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/jax/linear/README.md Executes the installation script to set up the necessary environment for the JAX linear model optimization example. This script is crucial for ensuring all dependencies are met before running the optimization. ```bash . ./install.sh ``` -------------------------------- ### Install Dependencies Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/torch/bert/README.md Executes the installation script to set up necessary dependencies for the example. ```bash ./install.sh ``` -------------------------------- ### Install Dependencies using install.sh (Bash) Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/tensorflow/efficientnet/README.md Installs necessary dependencies for the EfficientNet model optimization example. This script is typically run before the optimization process to set up the environment. ```bash . ./install.sh ``` -------------------------------- ### Install NeMo and Dependencies Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/08_optimize_pytorch_hifigan_qat_model/README.md Installs necessary system libraries (libsndfile1, ffmpeg) and the NeMo toolkit from GitHub. This is a prerequisite for running the HiFiGAN optimization example. ```bash sudo apt-get install -y libsndfile1 ffmpeg python3 -m pip install git+https://github.com/NVIDIA/NeMo.git@r1.7.2#egg=nemo_toolkit[all] ``` -------------------------------- ### Install Dependencies Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/jax/gpt2/README.md Installs necessary dependencies and sets up the environment for the Flax GPT-2 optimization example. This script typically handles package installations and environment configurations. ```bash . ./install.sh ``` -------------------------------- ### Install Torch Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/14_optimize_and_serve_onnx_and_tensorrt_ensemble_on_pytriton/README.md Installs the PyTorch package required for the example. Can be installed via pip or using an NVIDIA PyTorch container. ```shell pip install torch ``` -------------------------------- ### Install Python Dependencies Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/05_optimize_and_serve_model_on_pytriton/README.md Installs the PyTorch package required for the example. This can be done using pip or by running a specific NVIDIA PyTorch container. ```shell pip install torch ``` ```shell docker run -it --gpus 1 --shm-size 8gb -v ${PWD}:${PWD} -w ${PWD} nvcr.io/nvidia/pytorch:23.01-py3 bash ``` -------------------------------- ### Install Dependencies Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/torch/resnet50_nvidia_deep_learning_examples/README.md Executes a shell script to download the model repository and a dummy version of the ImageNet dataset. This script is essential for setting up the necessary files for model optimization. ```bash . ./install.sh ``` -------------------------------- ### Install PyTorch Dependency Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/06_optimize_and_serve_model_on_triton/README.md Installs the PyTorch package required for the example using pip. This is a prerequisite for running the optimization script. ```shell pip install torch ``` -------------------------------- ### Set Up Local Development Environment Source: https://github.com/triton-inference-server/model_navigator/blob/main/CONTRIBUTING.md Install your local copy of the Triton Model Navigator into a virtual environment and install development dependencies using make. ```shell mkvirtualenv model_navigator cd model_navigator/ make install-dev ``` -------------------------------- ### Navigate to Example Directory Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/14_optimize_and_serve_onnx_and_tensorrt_ensemble_on_pytriton/README.md Changes the current directory to the specific example folder for execution. ```shell cd examples/14_optimize_and_serve_onnx_and_tensorrt_ensemble_on_pytriton ``` -------------------------------- ### Start PyTriton Server Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/05_optimize_and_serve_model_on_pytriton/README.md Starts the PyTriton server using the package generated during the optimization step. ```shell ./serve.py ``` -------------------------------- ### Run JAX Linear Model Optimization Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/jax/linear/README.md Executes the Python script to perform the optimization of the JAX linear model. This is the main command to start the optimization process after the environment has been successfully configured. ```bash ./optimize.py ``` -------------------------------- ### Install Dependencies with Bash Script Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/tensorflow/bert/README.md Executes the installation script to set up all required dependencies for the BERT model optimization example. This script is typically run before the optimization process. ```bash ./install.sh ``` -------------------------------- ### Install Triton Model Navigator Dependencies Source: https://github.com/triton-inference-server/model_navigator/blob/main/README.md Installs necessary Python packages (transformers, diffusers, torch) required for optimizing models with Triton Model Navigator. This is a prerequisite for running the Stable Diffusion optimization example. ```shell pip install transformers diffusers torch ``` -------------------------------- ### Install NVIDIA PyTriton Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/14_optimize_and_serve_onnx_and_tensorrt_ensemble_on_pytriton/README.md Installs the NVIDIA PyTriton library, which is required for serving models. ```shell pip install nvidia-pytriton ``` -------------------------------- ### Example PyTriton Client Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/05_optimize_and_serve_model_on_pytriton/README.md Provides a client script to test the model deployment on the running PyTriton server. ```shell ./client.py ``` -------------------------------- ### Install Model Navigator Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/14_optimize_and_serve_onnx_and_tensorrt_ensemble_on_pytriton/README.md Installs the Triton Model Navigator package with Torch support. This command should be run from the main catalog inside the repository. ```shell pip install --extra-index-url https://pypi.ngc.nvidia.com .[torch] ``` -------------------------------- ### Install Triton Model Navigator Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/06_optimize_and_serve_model_on_triton/README.md Installs the Triton Model Navigator package with Torch support. This command should be run from the main repository catalog. ```shell pip install --extra-index-url https://pypi.ngc.nvidia.com .[torch] ``` -------------------------------- ### Navigate to Example Directory Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/13_optimize_tensorrt_linear_model/README.md Changes the current working directory to the specific example's directory. This is a prerequisite for executing the subsequent scripts correctly. ```bash cd examples/13_optimize_tensorrt_linear_model ``` -------------------------------- ### Run Triton Inference Server Example Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/06_optimize_and_serve_model_on_triton/README.md Executes the Python script that optimizes a linear model and prepares it for deployment. This script generates the model repository. ```shell python examples/triton/optimize.py ``` -------------------------------- ### Build Triton Model Navigator Wheel Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/installation.md Builds the Triton Model Navigator as a wheel package. This involves installing development dependencies and then running the distribution command. ```shell make install-dev ``` ```shell make dist ``` -------------------------------- ### Install Model Navigator for Torch Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/05_optimize_and_serve_model_on_pytriton/README.md Installs the Triton Model Navigator with Torch support. This command should be run from the main repository catalog. ```shell pip install --extra-index-url https://pypi.ngc.nvidia.com .[torch] ``` -------------------------------- ### Install Triton Model Navigator with PyTorch Extras Source: https://github.com/triton-inference-server/model_navigator/blob/main/README.md Installs the Triton Model Navigator package with dependencies required for PyTorch integration. This command fetches packages from NVIDIA's NGC PyPI index. ```shell pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[torch] ``` -------------------------------- ### Install Triton Model Navigator with TensorFlow Extras Source: https://github.com/triton-inference-server/model_navigator/blob/main/README.md Installs the Triton Model Navigator package with dependencies required for TensorFlow integration. This command fetches packages from NVIDIA's NGC PyPI index. ```shell pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[tensorflow] ``` -------------------------------- ### Run Model Optimization Script Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/models/onnx/identity/README.md Executes the `optimize.py` script to initiate the model optimization process. This command is intended to be run from the project's root directory. ```bash ./optimize.py ``` -------------------------------- ### Serve Model with PyTriton Source: https://github.com/triton-inference-server/model_navigator/blob/main/examples/14_optimize_and_serve_onnx_and_tensorrt_ensemble_on_pytriton/README.md Starts the PyTriton server using a Python script. The server will load and serve the optimized models generated in the previous steps. ```python python ./serve.py ``` -------------------------------- ### Generate Prediction with Optimized Pipeline Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/quick_start.md Demonstrates how to use the pipeline with its now-optimized models to generate predictions and save the output image. The optimized models are used transparently during inference. ```python pipe.to("cuda") images = pipe(["a photo of an astronaut riding a horse on mars"]) image = images[0][0] image.save("an_astronaut_riding_a_horse.png") ``` -------------------------------- ### Model Optimization Command Execution Source: https://github.com/triton-inference-server/model_navigator/blob/main/tests/assets/reports/optimize/scenario_inplace.txt Logs detailing the start and end of various optimization commands executed during the model optimization pipeline. This includes commands for correctness checks, performance benchmarking, and model verification across different configurations and runners. ```Log OptimizeEvent.COMMAND_STARTED {'command': 'Correctness', 'config_key': 'onnx', 'runner_name': 'OnnxTensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Correctness', 'config_key': 'trt-fp32', 'runner_name': 'TensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Correctness', 'config_key': 'trt-fp16', 'runner_name': 'TensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Performance', 'config_key': 'torchscript-script', 'runner_name': 'TorchScriptCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Performance', 'config_key': 'torchscript-trace', 'runner_name': 'TorchScriptCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Performance', 'config_key': 'onnx', 'runner_name': 'OnnxCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Performance', 'config_key': 'onnx', 'runner_name': 'OnnxTensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Performance', 'config_key': 'trt-fp32', 'runner_name': 'TensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Performance', 'config_key': 'trt-fp16', 'runner_name': 'TensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'Performance', 'config_key': 'torch', 'runner_name': 'TorchCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'VerifyModel', 'config_key': 'torch', 'runner_name': 'TorchCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'VerifyModel', 'config_key': 'torchscript-script', 'runner_name': 'TorchScriptCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'VerifyModel', 'config_key': 'torchscript-trace', 'runner_name': 'TorchScriptCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'VerifyModel', 'config_key': 'onnx', 'runner_name': 'OnnxCUDA'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'VerifyModel', 'config_key': 'onnx', 'runner_name': 'OnnxTensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'VerifyModel', 'config_key': 'trt-fp32', 'runner_name': 'TensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'VerifyModel', 'config_key': 'trt-fp16', 'runner_name': 'TensorRT'} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} ``` -------------------------------- ### Model Optimization Pipeline Stages Source: https://github.com/triton-inference-server/model_navigator/blob/main/tests/assets/reports/optimize/scenario_inplace.txt Events indicating the start and end of distinct phases within the overall model optimization process, such as 'Performance', 'Verify Models', 'Preprocessing', and 'PyTorch Export'. ```Log OptimizeEvent.PIPELINE_STARTED {'name': 'Performance'} OptimizeEvent.PIPELINE_FINISHED {} OptimizeEvent.PIPELINE_STARTED {'name': 'Verify Models'} OptimizeEvent.PIPELINE_FINISHED {} OptimizeEvent.PIPELINE_STARTED {'name': 'Preprocessing'} OptimizeEvent.COMMAND_STARTED {'command': 'InferInputMetadata', 'config_key': None, 'runner_name': None} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'FetchInputModelData', 'config_key': None, 'runner_name': None} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'InferOutputMetadata', 'config_key': None, 'runner_name': None} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'FetchOutputModelData', 'config_key': None, 'runner_name': None} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.PIPELINE_FINISHED {} OptimizeEvent.PIPELINE_STARTED {'name': 'PyTorch Export'} OptimizeEvent.COMMAND_STARTED {'command': 'ExportTorch2TorchScript', 'config_key': 'torchscript-script', 'runner_name': None} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'ExportTorch2TorchScript', 'config_key': 'torchscript-trace', 'runner_name': None} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} OptimizeEvent.COMMAND_STARTED {'command': 'ExportExportedProgram', 'config_key': 'torch-exportedprogram', 'runner_name': None} OptimizeEvent.COMMAND_FINISHED {'status': CommandStatus.OK} ``` -------------------------------- ### Install Triton Model Navigator with TensorFlow Extras Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/installation.md Installs the Triton Model Navigator with TensorFlow dependencies. This command utilizes an extra index URL for package retrieval. ```shell pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[tensorflow] ``` ```shell pip install nvidia-pyindex pip install -U triton-model-navigator[tensorflow] ``` -------------------------------- ### Deploy Model in PyTriton Server using Navigator Adapter Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/quick_start.md Shows how to deploy an optimized model using PyTriton. It utilizes `PyTritonAdapter` to retrieve the `runner` and other necessary information to bind the model for inference within a PyTriton server. ```python pytriton_adapter = nav.pytriton.PyTritonAdapter(package=package, strategy=nav.MaxThroughputStrategy()) runner = pytriton_adapter.runner runner.activate() @batch def infer_func(**inputs): return runner.infer(inputs) with Triton() as triton: triton.bind( model_name="resnet50", infer_func=infer_func, inputs=pytriton_adapter.inputs, outputs=pytriton_adapter.outputs, config=pytriton_adapter.config, ) triton.serve() ``` -------------------------------- ### Install Triton Model Navigator with JAX Extras Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/installation.md Installs the Triton Model Navigator with experimental JAX dependencies. This command uses an extra index URL for package acquisition. ```shell pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[jax] ``` ```shell pip install nvidia-pyindex pip install -U triton-model-navigator[jax] ``` -------------------------------- ### Install Triton Model Navigator with PyTorch Extras Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/installation.md Installs the Triton Model Navigator with PyTorch dependencies. This command uses an extra index URL to fetch the necessary packages. ```shell pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[torch] ``` ```shell pip install nvidia-pyindex pip install -U triton-model-navigator[torch] ``` -------------------------------- ### Install Triton Model Navigator Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/pypi.rst Installs the Triton Model Navigator package from the NVIDIA NGC PyPI index. Supports installing with optional extras for specific frameworks like PyTorch or TensorFlow. ```text pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[] ``` ```text pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[torch] ``` ```text pip install -U --extra-index-url https://pypi.ngc.nvidia.com triton-model-navigator[tensorflow] ``` -------------------------------- ### Add Triton Model Repository Entry from Navigator Package Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/quick_start.md Illustrates creating a Triton model repository entry for an optimized model package. This function selects the optimal model format based on a strategy and generates the necessary configuration files within a specified model repository path. ```python import pathlib nav.triton.model_repository.add_model_from_package( model_repository_path=pathlib.Path("model_repository"), model_name="resnet50", package=package, strategy=nav.MaxThroughputStrategy(), ) ``` -------------------------------- ### Prepare Dataloader for Pipeline Optimization Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/quick_start.md Provides a simple dataloader function that generates input data required for the model optimization process. The first element of the tuple specifies the batch size, and the second is the input prompt. ```python def get_dataloader(): # Please mind, the first element in tuple need to be a batch size return [(1, "a photo of an astronaut riding a horse on mars")] ``` -------------------------------- ### Load Optimized Pipeline Modules Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/quick_start.md Loads the most performant, optimized versions of the pipeline modules that were identified during the `nav.optimize` step. This allows subsequent inference to utilize the enhanced models directly. ```python nav.load_optimized() ``` -------------------------------- ### Initialize and Wrap Stable Diffusion Pipeline Modules Source: https://github.com/triton-inference-server/model_navigator/blob/main/docs/quick_start.md Initializes a Stable Diffusion pipeline and wraps its core components (text_encoder, unet, vae.decoder) using `nav.Module` for optimization. It includes a lambda function for mapping outputs to the correct HuggingFace class for the text encoder. ```python import model_navigator as nav from transformers.modeling_outputs import BaseModelOutputWithPooling from diffusers import DPMSolverMultistepScheduler, StableDiffusionPipeline def get_pipeline(): # Initialize Stable Diffusion pipeline and wrap modules for optimization pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1") pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config) pipe = pipe.to("cuda") pipe.text_encoder = nav.Module( pipe.text_encoder, name="clip", output_mapping=lambda output: BaseModelOutputWithPooling(**output), # Mapping to convert output data to HuggingFace class ) pipe.unet = nav.Module( pipe.unet, name="unet", ) pipe.vae.decoder = nav.Module( pipe.vae.decoder, name="vae", ) return pipe ``` -------------------------------- ### Install ONNX Runtime with CUDA 11 Support Source: https://github.com/triton-inference-server/model_navigator/blob/main/README.md Provides instructions for installing ONNX Runtime with CUDA 11 support, specifying an alternative extra index URL for compatibility with older CUDA versions. This is relevant for users needing CUDA 11 compatibility, as newer ONNX Runtime versions default to CUDA 12. ```shell .. --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/ .. ```