Model Optimizer

https://nvidia.github.io/Model-Optimizer
Admin
Model Optimizer is an NVIDIA tool for optimizing deep learning models through quantization, pruning,...
Tokens:296,772
Snippets:2,046
Trust Score:8.3
Update:10 months ago
Show doc for...
Tokens:
Raw
### QLoRA Checkpoint Export Example for vLLM Deployment

Source: https://nvidia.github.io/Model-Optimizer/reference/_changelog_for_Linux

An example demonstrating how to export a QLoRA checkpoint specifically for vLLM deployment is now available. This guides users on preparing their finetuned models for efficient inference with vLLM. Refer to `examples/llm_qat/README.md`.

```markdown
## Exporting QLoRA Checkpoint for vLLM Deployment

This example illustrates the process of exporting a QLoRA checkpoint finetuned with ModelOpt, optimized for deployment with vLLM. Consult `examples/llm_qat/README.md` for comprehensive details.
```

--------------------------------

### Installing NVIDIA Model Optimizer with ONNX/Deploy Dependencies

Source: https://nvidia.github.io/Model-Optimizer/getting_started/_installation_for_Linux

Installs the NVIDIA Model Optimizer with specific optional dependencies required for ONNX model optimization and deployment. This is useful if you only need ONNX-related functionalities.

```bash
pip install -U "nvidia-modelopt[onnx]"
```

--------------------------------

### Run Olive Optimization with Configuration (Shell)

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/windows/_installation_with_olive.rst

Executes the Olive optimization process using a specified configuration JSON file. The '--setup' flag can be used for initial setup or to apply configurations.

```shell
$ olive run --config <config json> --setup
$ olive run --config <config json>
```

--------------------------------

### Installing NVIDIA Model Optimizer with Hugging Face Dependencies

Source: https://nvidia.github.io/Model-Optimizer/getting_started/_installation_for_Linux

Installs the NVIDIA Model Optimizer with optional dependencies for integrating with Hugging Face libraries like 'transformers' and 'diffusers'. This is for users working with Hugging Face models.

```bash
pip install -U "nvidia-modelopt[hf]"
```

--------------------------------

### Installing Triton for Accelerated Quantization Kernels

Source: https://nvidia.github.io/Model-Optimizer/getting_started/_installation_for_Linux

Installs the Triton package, which is required to enable optimized quantization kernels in Model Optimizer. These kernels provide a significant speedup for AWQ and QAT workflows on compatible hardware.

```bash
pip install triton
```

--------------------------------

### Install Olive with NVIDIA Model Optimizer (Bash)

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/windows/_installation_with_olive.rst

Installs the Olive library along with the NVIDIA Model Optimizer (nvmo) package for Windows. This is the primary command for setting up the necessary tools.

```bash
pip install olive-ai[nvmo]
```

--------------------------------

### Install NVIDIA Model Optimizer with All Dependencies (Bash)

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/_installation_for_Linux.rst

Installs the NVIDIA Model Optimizer package ('nvidia-modelopt') along with all its optional dependencies using pip. This command upgrades the package to the latest version if it's already installed.

```bash
pip install -U "nvidia-modelopt[all]"
```

--------------------------------

### Environment Variable Setup for TensorRT Binaries (Shell)

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/_installation_for_Linux.rst

Sets environment variables required for TensorRT binaries, specifically adding paths for include and library files. This is typically used within a Docker container environment.

```shell
export PIP_CONSTRAINT=""
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/include:/usr/lib/x86_64-linux-gnu"
```

--------------------------------

### Install ONNX Runtime GPU Package

Source: https://nvidia.github.io/Model-Optimizer/getting_started/windows/_installation_standalone

This command installs the `onnxruntime-gpu` package, which provides CUDA Execution Provider support for ONNX Runtime. This requires that CUDA and cuDNN are correctly installed and configured on your system, and compatible versions are used.

```shell
pip install onnxruntime-gpu
```

--------------------------------

### Precompile ModelOptimizer PyTorch Quantization Kernels

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/_installation_for_Linux.rst

This command initiates the compilation of fast quantization kernels for ModelOptimizer's PyTorch APIs. It checks for CUDA availability and compiles the kernels, which can take a few minutes on the first run but speeds up subsequent operations. This is useful for verifying installation or pre-compiling for environments like Docker.

```bash
python -c "import modelopt.torch.quantization.extensions as ext; ext.precompile()"
```

--------------------------------

### Install ModelOpt-Windows with ONNX Module

Source: https://nvidia.github.io/Model-Optimizer/getting_started/windows/_installation_standalone

This command installs the ModelOpt-Windows toolkit along with its ONNX module and the `onnxruntime-directml` package. Ensure `pip` is installed and updated. This command installs the necessary components for ONNX model optimization and quantization.

```shell
pip install "nvidia-modelopt[onnx]"
```

--------------------------------

### Create and Activate Python Virtual Environment

Source: https://nvidia.github.io/Model-Optimizer/getting_started/windows/_installation_standalone

This snippet demonstrates how to create and activate a Python virtual environment using the `venv` module. This is recommended for managing project dependencies and avoiding conflicts. It requires Python to be installed and accessible in the system's PATH.

```shell
mkdir myEnv
python -m venv .\myEnv
.\myEnv\Scripts\activate
```

--------------------------------

### Deploy Model Optimizer Quantized Model with SGLang (Python)

Source: https://nvidia.github.io/Model-Optimizer/_sources/deployment/3_unified_hf.rst

This example demonstrates deploying a Model Optimizer quantized model from Hugging Face using SGLang. It requires the `sglang` library and a compatible SGLang installation. The output shows the generated text for each prompt.

```python
import sglang as sgl

def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = {"temperature": 0.8, "top_p": 0.95}
    llm = sgl.Engine(model_path="nvidia/Llama-3.1-8B-Instruct-FP8", quantization="modelopt")

    outputs = llm.generate(prompts, sampling_params)
    for prompt, output in zip(prompts, outputs):
        print("===============================")
        print(f"Prompt: {prompt}\nGenerated text: {output['text']}")

if __name__ == "__main__":
    main()

```

--------------------------------

### Install ONNX Runtime DirectML Prerequisites (Shell)

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/windows/_installation_with_olive.rst

Installs the required ONNX Runtime packages with DirectML support, which are necessary for Olive to work with the DirectML Execution Provider (EP).

```shell
$ pip install onnxruntime-genai-directml>=0.4.0
$ pip install onnxruntime-directml==1.20.0
```

--------------------------------

### Create and Activate Conda Virtual Environment (Shell)

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/_installation_for_Linux.rst

Creates a new conda virtual environment named 'modelopt' with Python version 3.12 and pip installed, then activates the environment. This is recommended for managing dependencies locally.

```shell
conda create -n modelopt python=3.12 pip
conda activate modelopt
```

--------------------------------

### PEFT Adapter Configuration Example (Python/JSON)

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.peft.config

Illustrates how to configure PEFT adapters using a dictionary where keys are module patterns and values are either PEFTAttributeConfig objects or dictionaries. This example shows overriding default settings for specific layers.

```python
"adapter_cfg": {
   "*": {
      "rank": 32,
      "scale": 1,
      "enable": True,
   },
   "*output_layer*": {"enable": False},
}
```

--------------------------------

### Example of layer-specific configuration (JSON)

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.prune.fastnas

This JSON example demonstrates how to apply configurations to specific submodules using glob patterns. It sets a default configuration for all modules and then overrides it for modules matching '*lm_head*' by setting their configuration to null (deactivating them).

```json
{
    "*": {...},
    "*lm_head*": null
}
```

--------------------------------

### Minitron Pruning Example for Megatron-LM

Source: https://nvidia.github.io/Model-Optimizer/reference/_changelog_for_Linux

An example demonstrating Minitron pruning integrated with the Megatron-LM framework is now available. This showcases how to apply Minitron's pruning techniques to large language models trained with Megatron-LM. See `Megatron-LM/examples/post_training/modelopt`.

```markdown
## Minitron Pruning Example for Megatron-LM

This example demonstrates the integration of Minitron pruning with the Megatron-LM framework. For detailed instructions and code, please refer to `Megatron-LM/examples/post_training/modelopt`.
```

--------------------------------

### Verify ONNX and ONNX Runtime Import in Python

Source: https://nvidia.github.io/Model-Optimizer/getting_started/windows/_installation_standalone

This Python command verifies that the `onnx` and `onnxruntime` libraries can be successfully imported. This check ensures that these core libraries are installed and accessible within the Python environment.

```python
python -c "import onnx; import onnxruntime"
```

--------------------------------

### Dynamic BatchNorm1d Configuration Example

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.nas.autonas

This example demonstrates how to configure dynamic behavior for nn.BatchNorm1d modules within the AutoNAS mode. It shows the default configuration if none is specified, how to deactivate dynamic behavior using None, and how to apply layer-specific configurations using glob patterns for submodule names.

```python
# Default configuration (if 'nn.BatchNorm1d' key is not specified in the main config)
{
  "*": {
    "features_ratio": [
      0.5,
      0.67,
      1.0
    ],
    "feature_divisor": 32
  }
}

# To deactivate dynamic nn.BatchNorm1d modules:
{
    "nn.BatchNorm1d": None
}

# To specify layer-specific configurations (e.g., exclude 'lm_head' submodule):
{
    "*": {...}, # General configuration for other BatchNorm1d layers
    "*lm_head*": None # Deactivate for layers within 'lm_head' submodule
}

```

--------------------------------

### Build QKV Configuration (Python)

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.export.layer_utils

Converts QKV (Query, Key, Value) modules into their configuration format. It accepts a list of QKV modules, model metadata configuration, optional external configuration, and tensor parallelism size.

```python
build_qkv(_qkv_modules_ , _model_metadata_config_ , _ext_config =None_, _tp_size =1_)
```

--------------------------------

### Create Advanced Custom INT4/INT8 Quantization Configuration in Python

Source: https://nvidia.github.io/Model-Optimizer/_sources/guides/_pytorch_quantization.rst

This example shows how to create a new, custom quantization configuration. It specifically sets up INT4 block-wise weight quantization and INT8 token-wise dynamic activation quantization, demonstrating advanced recipe exploration.

```python
# Custom configuration for INT4 block-wise weights and INT8 dynamic activations
MY_CUSTOM_CONFIG = {
    "quant_cfg": {
        # Configure weight quantizers with 4-bit precision and 128-element blocks
        "*weight_quantizer": {"num_bits": 4, "block_sizes": {-1: 128}, "enable": True},

        # Configure input quantizers with 8-bit dynamic quantization
        "*input_quantizer": {"num_bits": 8, "type": "dynamic", "block_sizes": {-1: None}},

        # Include default disabled quantizer configurations
        **_default_disabled_quantizer_cfg,
    },
    "algorithm": "max",
}
```

--------------------------------

### Apply Post Training Quantization (PTQ) in PyTorch

Source: https://nvidia.github.io/Model-Optimizer/_sources/guides/_pytorch_quantization.rst

Demonstrates applying PTQ using ModelOpt's `quantize` function. This involves setting up the model, selecting a quantization configuration, preparing a data loader for calibration, defining a forward loop, and then calling `quantize`. The `print_quant_summary` function is used to verify the quantization.

```python
import modelopt.torch.quantization as mtq

# Setup the model
model = get_model()

# Select quantization config
config = mtq.INT8_SMOOTHQUANT_CFG

# Quantization need calibration data. Setup calibration data loader
# An example of creating a calibration data loader looks like the following:
data_loader = get_dataloader(num_samples=calib_size)


# Define forward_loop. Please wrap the data loader in the forward_loop
def forward_loop(model):
    for batch in data_loader:
        model(batch)


# Quantize the model and perform calibration (PTQ)
model = mtq.quantize(model, config, forward_loop)

# Print quantization summary after successfully quantizing the model with mtq.quantize
# This will show the quantizers inserted in the model and their configurations
mtq.print_quant_summary(model)

# Export the model to ONNX
torch.onnx.export(model, sample_input, onnx_file)
```

--------------------------------

### build_qkv

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.export.layer_utils

Converts the Query, Key, Value (QKV) modules into their configuration representation.

```APIDOC
## build_qkv

### Description
Converts the qkv modules to the config.

### Method
N/A (Function Call)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```json
{
  "_qkv_modules_": "<QKV Modules object>"
}
```

### Response
#### Success Response
- **QKVConfig** (_object_) - The configuration of the QKV modules.

#### Response Example
```json
{
  "qkv_config": "<QKVConfig object>"
}
```
```

--------------------------------

### Perform Quantization-Aware Training (QAT) in PyTorch

Source: https://nvidia.github.io/Model-Optimizer/_sources/guides/_pytorch_quantization.rst

Illustrates the process of Quantization-Aware Training (QAT) using ModelOpt. This involves quantizing a model using `mtq.quantize` with a specified configuration and forward loop, then fine-tuning the quantized model using a standard training pipeline.

```python
import modelopt.torch.quantization as mtq

# Select quantization config
config = mtq.INT8_DEFAULT_CFG


# Define forward loop for calibration
def forward_loop(model):
    for data in calib_set:
        model(data)


# QAT after replacement of regular modules to quantized modules
model = mtq.quantize(model, config, forward_loop)

# Fine-tune with original training pipeline
# Adjust learning rate and training duration
train(model, train_loader, optimizer, scheduler, ...)
```

--------------------------------

### Minitron Pruning and Distillation Guidelines

Source: https://nvidia.github.io/Model-Optimizer/reference/_changelog_for_Linux

General guidelines for performing pruning and distillation using Minitron are now available. Refer to `examples/pruning/README.md` for detailed instructions and best practices.

```markdown
## Minitron Pruning and Distillation Guidelines

This section provides an overview of how to use Minitron for model pruning and distillation. Please refer to `examples/pruning/README.md` for detailed examples and setup instructions.
```

--------------------------------

### Deploy Quantized Model with vLLM (Python)

Source: https://nvidia.github.io/Model-Optimizer/_sources/deployment/3_unified_hf.rst

This script demonstrates loading and running a quantized model from Huggingface using vLLM. It requires the vLLM library and a compatible model. The example shows setting up sampling parameters and generating text from prompts.

```python
from vllm import LLM, SamplingParams

def main():

    model_id = "nvidia/Llama-3.1-8B-Instruct-FP8"
    sampling_params = SamplingParams(temperature=0.8, top_p=0.9)

    prompts = [
        "Hello, my name is",

```

--------------------------------

### Define Custom Quantized Module in PyTorch with ParallelState

Source: https://nvidia.github.io/Model-Optimizer/guides/_pytorch_quantization

Illustrates the initial setup for defining a custom quantized linear module in PyTorch using Model Optimizer. It imports necessary classes for tensor quantization and distributed parallel state management. The example highlights the importance of `ParallelState` for synchronizing quantization parameters in distributed training.

```python
from modelopt.torch.quantization.nn import TensorQuantizer
from modelopt.torch.utils.distributed import ParallelState

```

--------------------------------

### Initialize HF Models with Compressed Weights

Source: https://nvidia.github.io/Model-Optimizer/_sources/guides/_compress_quantized_models.rst

Initializes Hugging Face models with compressed weights using ModelOpt for reduced memory usage, especially effective for NVFP4 quantized formats. This workflow requires the max calibration algorithm and is demonstrated with `AutoModelForCausalLM.from_pretrained`.

```python
import modelopt.torch.quantization as mtq
from modelopt.torch.quantization.plugins import init_quantized_weights
from transformers import AutoModelForCausalLM, AutoConfig

# Placeholder for calibration loop implementation
def calibrate_loop(model, dataloader):
    pass

ckpt_path = "path/to/your/model/checkpoint"

# Step 1: Initialize the model with compressed weights
with init_quantized_weights(mtq.NVFP4_DEFAULT_CFG):
    model = AutoModelForCausalLM.from_pretrained(ckpt_path)

# Step 2: Calibrate the model
mtq.calibrate(model, algorithm="max", forward_loop=calibrate_loop)

# downstream tasks, e.g., export quantized model/accuracy evaluation
...
```

--------------------------------

### Support PTQ and FakeQuant in vLLM for Fast Evaluation

Source: https://nvidia.github.io/Model-Optimizer/reference/_changelog_for_Linux

Enables Post-Training Quantization (PTQ) and fake quantization within vLLM for accelerated evaluation of various quantization formats. Detailed examples and usage instructions can be found in the `examples/vllm_serve` directory.

```python
# Example usage within vLLM (conceptual)
# See examples/vllm_serve for actual implementation details
```

--------------------------------

### Using MFTDistillationLoss for Minifinetuning (Python)

Source: https://nvidia.github.io/Model-Optimizer/_sources/guides/4_distillation.rst

This example shows how to replace the standard `LogitsDistillationLoss` with `MFTDistillationLoss` for minifinetuning. This technique helps train models on small datasets without losing original knowledge by adjusting the teacher's distribution.

```python
import modelopt.torch.distill.losses as mtd_losses

# Assuming student_model and teacher_model are defined
distillation_config = {
    "teacher_model": teacher_model,
    # Using MFTDistillationLoss for minifinetuning
    "criterion": {("output", "output"): mtd_losses.MFTDistillationLoss()}
}

# The rest of the conversion process would follow using atd.convert
```

--------------------------------

### ONNX Model Conversion with Verbose Logging and Custom Output

Source: https://nvidia.github.io/Model-Optimizer/_sources/guides/8_autocast.rst

This example shows how to convert an ONNX model with detailed logging (DEBUG level) and specify a custom output path for the converted model. This is useful for debugging and managing output files.

```bash
python -m modelopt.onnx.autocast --onnx_path model.onnx --output_path custom_path.onnx --log_level DEBUG
```

--------------------------------

### Perform Pruning with FastNAS using ModelOpt

Source: https://nvidia.github.io/Model-Optimizer/guides/3_pruning

Demonstrates how to prune a ResNet50 model using the FastNAS method from NVIDIA Model Optimizer. This involves defining a scoring function, specifying pruning constraints (e.g., FLOPs reduction), and providing a dummy input for constraint measurement. The example assumes the availability of `modelopt.torch.prune`, `torchvision.models`, a pretrained model, validation data (`val_loader`), and a `validate` function.

```python
import modelopt.torch.prune as mtp
from torchvision.models import resnet50
import torch

# User-defined model
model = resnet50()

# Load pretrained weights here
# pretrained_weights = ... # Load your actual weights
# model.load_state_dict(pretrained_weights)


# Wrap your original validation function to only take the model as input.
# This function acts as the score function to rank models.
# def score_func(model):
#     return validate(model, val_loader, ...)


# Define a dummy input with similar shape as that of your input data
dummy_input = torch.randn(1, 3, 224, 244)

# Prune to a model with less than or equal to 60% of original FLOPs
prune_constraints = {"flops": "60%"}

# Example of calling prune (actual call requires more parameters like mode, dataloader etc.)
# pruned_model = mtp.prune(model, 
#                         mode="fastnas", 
#                         constraints=prune_constraints, 
#                         dummy_input=dummy_input,
#                         score_func=score_func, 
#                         ... # other necessary parameters
#                        )

```

--------------------------------

### JSON Configuration Example

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.sparsity.weight_sparsity.config

An example of a JSON configuration structure for the NVIDIA model optimizer. This format is used to define settings for various submodules, allowing for specific overrides.

```json
{...}
```

--------------------------------

### Apply INT4 AWQ Quantization to LLM ONNX Model (Python)

Source: https://nvidia.github.io/Model-Optimizer/_sources/guides/windows_guides/_ONNX_PTQ_guide.rst

Demonstrates how to apply INT4 AWQ quantization to an LLM ONNX model using the modelopt library. It requires calibration data and specifies the calibration method and error thresholds.

```python
from modelopt.onnx.quantization.int4 import quantize as quantize_int4
# import other packages as needed
calib_inputs = get_calib_inputs(dataset, model_name, cache_dir, calib_size, batch_size,...)
quantized_onnx_model = quantize_int4(
    onnx_path,
    calibration_method="awq_lite",
    calibration_data_reader=None if use_random_calib else calib_inputs,
    calibration_eps=["dml", "cpu"]
)
```

--------------------------------

### Verify ModelOpt-Windows Import in Python

Source: https://nvidia.github.io/Model-Optimizer/getting_started/windows/_installation_standalone

This Python command verifies that the ModelOpt-Windows quantization module can be successfully imported. This confirms that the ModelOpt-Windows toolkit has been installed correctly and its core functionality is accessible.

```python
python -c "import modelopt.onnx.quantization"
```

--------------------------------

### Python: Install nvidia-modelopt for ONNX Quantization

Source: https://nvidia.github.io/Model-Optimizer/_sources/support/2_faqs.rst

Installs the necessary components for ONNX quantization using ModelOpt. This command ensures that the 'onnx' extra is included, providing the required dependencies.

```python
pip install nvidia-modelopt[onnx]
```

--------------------------------

### Start and Stop GPUMemoryMonitor Thread

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.utils.memory_monitor

Demonstrates starting the GPU memory monitoring in a background thread and then stopping it to retrieve peak memory usage. This functionality is crucial for real-time performance analysis.

```python
from model_optimizer.utils.memory_monitor import GPUMemoryMonitor

# Create a monitor instance
monitor = GPUMemoryMonitor(monitor_interval=0.5)

# Start monitoring in a separate thread
monitor.start()

# --- Run your GPU operations ---
import time
time.sleep(3)

# Stop monitoring and display results
monitor.stop()

```

--------------------------------

### Example Forward Loop for Quantization Calibration

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.quantization.model_quant

Provides example implementations of a `forward_loop` function used for gathering calibration statistics during model quantization. The function should iterate over a data loader and forward data through the model.

```python
def forward_loop(model) -> None:
    # iterate over the data loader and forward data through the model
    for batch in data_loader:
        model(batch)
```

```python
def forward_loop(model) -> float:
    # evaluate the model on the task
    return evaluate(model, task, ....)
```

```python
def forward_loop(model) -> None:
    # run evaluation pipeline
    evaluator.model = model
    evaluator.evaluate()
```

--------------------------------

### Run Olive Optimization with Configuration (Python)

Source: https://nvidia.github.io/Model-Optimizer/_sources/getting_started/windows/_installation_with_olive.rst

Initiates the Olive optimization process programmatically using Python. This method requires importing the 'run' function from Olive's workflows and providing the configuration file path.

```python
from olive.workflows import run as olive_run

olive_run("config.json")
```

--------------------------------

### Layer-Specific BatchNorm2d Configuration Example

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.nas.autonas

Demonstrates how to apply layer-specific configurations for nn.BatchNorm2d modules using glob patterns. This example excludes layers within the 'lm_head' submodule from dynamic conversion by setting their configuration to None.

```JSON
{
    "*": {...},
    "*lm_head*": None
}
```

--------------------------------

### Export QLoRA Checkpoint Finetuned with ModelOpt

Source: https://nvidia.github.io/Model-Optimizer/reference/_changelog_for_Linux

Provides functionality to export QLoRA checkpoints that have been finetuned using ModelOpt. This ensures that the quantized weights and configurations are correctly saved for deployment.

```python
from modelopt.torch.peft import export_lora_checkpoint

export_lora_checkpoint(model, output_dir='path/to/save/checkpoint')
```

--------------------------------

### is_tensorrt_llm_0_8_or_9

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.export.tensorrt_llm_utils

Checks if the installed TensorRT-LLM version is 0.8 or 0.9.

```APIDOC
## is_tensorrt_llm_0_8_or_9

### Description
Returns true if tensorrt_llm version is 0.8 or 0.9.

### Method
GET

### Endpoint
/websites/nvidia_github_io_model-optimizer/tensorrt_llm_utils/is_tensorrt_llm_0_8_or_9

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **is_version** (boolean) - True if the version is 0.8 or 0.9, false otherwise.

#### Response Example
```json
{
  "is_version": true
}
```
```

--------------------------------

### Build Stacked Experts Configuration (Python)

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.export.layer_utils

Builds configurations for experts' weights (weight_1 and weight_2) for Mixture of Experts (MoE) models. It requires the experts module, a list of linear layer names, the number of experts, and an expert getter function.

```python
build_stacked_experts(_experts_ , _linear_names_ , _num_experts_ , _expert_getter_)
```

--------------------------------

### Deploy with vLLM

Source: https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf

This code demonstrates deploying a modelopt quantized model with vLLM. It requires vLLM v0.6.5 or later and supports fp8 quantized models (without fp8 kv cache). The script takes prompts and sampling parameters to generate model outputs.

```python
from vllm import LLM, SamplingParams

def main():

    model_id = "nvidia/Llama-3.1-8B-Instruct-FP8"
    sampling_params = SamplingParams(temperature=0.8, top_p=0.9)

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]

    llm = LLM(model=model_id, quantization="modelopt")
    outputs = llm.generate(prompts, sampling_params)

    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

if __name__ == "__main__":
    main()

```

--------------------------------

### ONNX Model Conversion with Verbose Logging and Custom Output

Source: https://nvidia.github.io/Model-Optimizer/guides/8_autocast

This example shows how to convert an ONNX model to FP16 while enabling DEBUG level logging for detailed information and specifying a custom output path for the converted model. This is useful for troubleshooting and managing output files.

```python
python -m modelopt.onnx.autocast --onnx_path model.onnx --output_path custom_path.onnx --log_level DEBUG

```

--------------------------------

### get_encoder_config

Source: https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.export.layer_utils

Retrieves encoder information for decoder models within an encoder-decoder setup.

```APIDOC
## get_encoder_config

### Description
Get the encoder information for decoder model in enc-dec model.

### Method
N/A (Function Call)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```json
{
  "_decoder_model_": "<Decoder Model object>"
}
```

### Response
#### Success Response
- **dict** (_dict_) - A dictionary containing encoder configuration details.

#### Response Example
```json
{
  "encoder_config": {
    "hidden_size": 768,
    "num_layers": 12
  }
}
```
```
Tokens:
Raw