MMDeploy

https://mmdeploy.readthedocs.io/en/latest/
Admin
MMDeploy is a model deployment framework that converts and optimizes deep learning models from...
Tokens:138,493
Snippets:691
Trust Score:8.3
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install MM Engine and MMCV

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs the necessary prerequisites for MMDeploy: MMEngine and MMCV. It uses the 'openmim' tool for installation, recommending specific versions for compatibility.

```bash
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0rc2"
```

--------------------------------

### Install MMDeploy Model Converter

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs the MMDeploy model converter package. This is a core component for converting training models into backend-specific formats.

```bash
pip install mmdeploy==1.3.1
```

--------------------------------

### Install and Configure ONNX Runtime GPU

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs a specific version of onnxruntime-gpu, downloads the corresponding TGZ package, extracts it, and sets environment variables for ONNX Runtime GPU.

```bash
pip install onnxruntime-gpu==1.8.1
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-gpu-1.8.1.tgz
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-gpu-1.8.1
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
```

--------------------------------

### Install MMDeploy Runtime for ONNX Runtime

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs the MMDeploy runtime package that supports ONNX Runtime. This is used for inference with models converted to ONNX format.

```bash
pip install mmdeploy-runtime==1.3.1
```

--------------------------------

### Install ONNX Runtime

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs the ONNX Runtime inference engine for Linux. It downloads a precompiled TGZ archive from GitHub releases and sets the ONNXRUNTIME_DIR environment variable to include its libraries in the path.

```bash
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-1.8.1
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
```

--------------------------------

### Build MMDeploy SDK and Demo for CPU + ONNXRuntime

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds the MMDeploy SDK and demo applications for CPU with ONNXRuntime as the inference engine. Enables Python API and examples.

```bash
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake .. \
    -DCMAKE_CXX_COMPILER=g++-7 \
    -DMMDEPLOY_BUILD_SDK=ON \
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
    -DMMDEPLOY_BUILD_EXAMPLES=ON \
    -DMMDEPLOY_TARGET_DEVICES=cpu \
    -DMMDEPLOY_TARGET_BACKENDS=ort \
    -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR}

make -j$(nproc) && make install
```

--------------------------------

### Install PyTorch for CPU

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs PyTorch and Torchvision on a CPU platform using conda. This command is used when GPU acceleration is not available or required.

```bash
conda install pytorch=={pytorch_version} torchvision=={torchvision_version} cpuonly -c pytorch
```

--------------------------------

### Install PPL.NN

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs PPL.NN by first building it and installing pyppl, then exporting its root path to an environment variable.

```shell
cd ppl.nn
export PPLNN_DIR=$(pwd)
```

--------------------------------

### Install MMDeploy Runtime for ONNX Runtime GPU

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs the MMDeploy runtime package optimized for GPU inference using ONNX Runtime. This package is required if you need to perform inference on NVIDIA GPUs with ONNX.

```bash
pip install mmdeploy-runtime-gpu==1.3.1
```

--------------------------------

### Convert Faster R-CNN Model to TensorRT with MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Clones MMDeploy and MMDetection repositories, installs MMDetection, downloads a Faster R-CNN checkpoint, and runs the MMDeploy conversion script to convert the PyTorch model to TensorRT format.

```bash
# clone mmdeploy to get the deployment config. --recursive is not necessary
git clone -b main https://github.com/open-mmlab/mmdeploy.git

# clone mmdetection repo. We have to use the config file to build PyTorch nn module
git clone -b 3.x https://github.com/open-mmlab/mmdetection.git
cd mmdetection
mim install -v -e .
cd ..

# download Faster R-CNN checkpoint
wget -P checkpoints https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth

# run the command to start model conversion
python mmdeploy/tools/deploy.py \
    mmdeploy/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    mmdetection/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
    checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    mmdetection/demo/demo.jpg \
    --work-dir mmdeploy_model/faster-rcnn \
    --device cuda \
    --dump-info
```

--------------------------------

### Install TensorRT with MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

This snippet covers the installation of TensorRT and its Python bindings for MMDeploy, along with setting environment variables for TensorRT and cuDNN. It assumes manual download and extraction of TensorRT and cuDNN tar packages.

```bash
pip install TensorRT-8.2.3.0/python/tensorrt-8.2.3.0-cp38-none-linux_x86_64.whl
pip install pycuda
export TENSORRT_DIR=$(pwd)/TensorRT-8.2.3.0
export LD_LIBRARY_PATH=${TENSORRT_DIR}/lib:$LD_LIBRARY_PATH
export CUDNN_DIR=$(pwd)/cuda
export LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
```

--------------------------------

### Run MMDeploy Inference Demo (C++/CLI)

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Downloads and extracts MMDeploy C++ demo, then runs the object detection demo using a converted model and an image on the CUDA device.

```bash
wget https://github.com/open-mmlab/mmdeploy/releases/download/v1.3.1/mmdeploy-1.3.1-linux-x86_64-cuda11.8.tar.gz
tar xf mmdeploy-1.3.1-linux-x86_64-cuda11.8
cd mmdeploy-1.3.1-linux-x86_64-cuda11.8
# run python demo
python example/python/object_detection.py cuda ../mmdeploy_model/faster-rcnn ../mmdetection/demo/demo.jpg
# run C/C++ demo
# build the demo according to the README.md in the folder.
./bin/object_detection cuda ../mmdeploy_model/faster-rcnn ../mmdetection/demo/demo.jpg
```

--------------------------------

### Install OpenVINO

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs the OpenVINO development package using pip.

```shell
pip install openvino-dev
```

--------------------------------

### Install TorchScript (Libtorch)

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Downloads and unpacks Libtorch, then sets environment variables for its directory and shared libraries. This example uses Libtorch 1.8.1 with CUDA 11.1.

```shell
wget https://download.pytorch.org/libtorch/cu111/libtorch-shared-with-deps-1.8.1%2Bcu111.zip
unzip libtorch-shared-with-deps-1.8.1+cu111.zip
cd libtorch
export Torch_DIR=$(pwd)
export LD_LIBRARY_PATH=$Torch_DIR/lib:$LD_LIBRARY_PATH
```

--------------------------------

### Install ONNXRuntime

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs the ONNXRuntime Python package and configures environment variables for using the prebuilt binary package.

```shell
pip install onnxruntime==1.8.1
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
cd onnxruntime-linux-x64-1.8.1
export ONNXRUNTIME_DIR=$(pwd)
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
```

--------------------------------

### CMake Integration for MMDeploy C++ Projects

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Shows how to link the MMDeploy library and OpenCV in a CMake project. Assumes MMDeploy and OpenCV are installed and discoverable by CMake.

```cmake
find_package(MMDeploy REQUIRED)
target_link_libraries(${name} PRIVATE mmdeploy ${OpenCV_LIBS})

```

--------------------------------

### Install MM Engine, MMCV, and MMDeploy Dependencies

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs essential MMlab libraries including MMENGINE, MMCV (version >= 2.0.0rc2), and sets up OpenMIM. These are core components for MMDeploy functionality.

```bash
export cu_version=cu111 # cuda 11.1
export torch_version=torch1.8
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0rc2"

```

--------------------------------

### Install PyTorch for GPU

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Installs PyTorch and Torchvision on a GPU platform using conda. It requires specifying the PyTorch version, Torchvision version, and the CUDA toolkit version, ensuring compatibility with the host CUDA toolkit version.

```bash
conda install pytorch=={pytorch_version} torchvision=={torchvision_version} cudatoolkit={cudatoolkit_version} -c pytorch -c conda-forge
```

--------------------------------

### Build MMDeploy SDK and Demo for PPLNN

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds the MMDeploy SDK and demo applications with PPLNN as the inference engine. Supports CUDA and CPU devices, and requires pplcv and pplnn installations.

```bash
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake .. \
    -DCMAKE_CXX_COMPILER=g++-7 \
    -DMMDEPLOY_BUILD_SDK=ON \
    -DMMDEPLOY_BUILD_EXAMPLES=ON \
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
    -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
    -DMMDEPLOY_TARGET_BACKENDS=pplnn \
    -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/cuda-build/install/lib/cmake/ppl \
    -Dpplnn_DIR=${PPLNN_DIR}/pplnn-build/install/lib/cmake/ppl

make -j$(nproc) && make install
```

--------------------------------

### Evaluate MMDeploy Model Performance

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Command-line script to test the performance of a deployed MMDeploy model. Requires MMDeploy and MMDetection directories to be set as environment variables. Specifies model configuration, metrics, and device.

```bash
python ${MMDEPLOY_DIR}/tools/test.py \
    ${MMDEPLOY_DIR}/configs/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    ${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
    --model ${BACKEND_MODEL_FILES} \
    --metrics ${METRICS} \
    --device cuda:0

```

--------------------------------

### Setup Ascend Environment

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Sets the ASCEND_TOOLKIT_HOME environment variable for Ascend integration.

```shell
export ASCEND_TOOLKIT_HOME="/usr/local/Ascend/ascend-toolkit/latest"
```

--------------------------------

### Pull MMDeploy Docker Image

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/build_from_docker

Pulls a prebuilt MMDeploy Docker image from Docker Hub. Users can specify a tag to obtain a specific version or the latest release. This is the recommended method for quick setup.

```bash
export TAG=openmmlab/mmdeploy:ubuntu20.04-cuda11.8-mmdeploy
docker pull $TAG
```

--------------------------------

### Setup TVM Environment

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Sets the LD_LIBRARY_PATH and PYTHONPATH environment variables for TVM integration.

```shell
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${TVM_HOME}/build
export PYTHONPATH=${TVM_HOME}/python:${PYTHONPATH}
```

--------------------------------

### Build MMDeploy SDK and Demo with Multiple Backends

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds the MMDeploy SDK and demo applications with support for multiple inference engines including TensorRT, ONNXRuntime, ncnn, and OpenVINO on CUDA and CPU devices. Requires specific build directories for dependencies and protobuf.

```bash
export PROTO_DIR=/path/to/mmdeploy-dep/pbinstall
cmake .. \
    -DCMAKE_CXX_COMPILER=g++-7 \
    -DMMDEPLOY_BUILD_SDK=ON \
    -DMMDEPLOY_BUILD_EXAMPLES=ON \
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
    -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
    -DMMDEPLOY_TARGET_BACKENDS="trt;ort;ncnn;openvino" \
    -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/install/lib/cmake/ppl \
    -DTENSORRT_DIR=${TENSORRT_DIR} \
    -DCUDNN_DIR=${CUDNN_DIR} \
    -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} \
    -DInferenceEngine_DIR=${OPENVINO_DIR}/runtime/cmake \
    -Dncnn_DIR=${NCNN_DIR}/build/install/lib/cmake/ncnn \
    -DProtobuf_LIBRARIES=${PROTO_DIR}/lib/libprotobuf.so \
    -DProtobuf_PROTOC_EXECUTABLE=${PROTO_DIR}/bin/protoc \
    -DProtobuf_INCLUDE_DIR=${PROTO_DIR}/pbinstall/include
```

--------------------------------

### MMDeploy RKNN Backend Configuration Example

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/rockchip

An example of a `backend_config` dictionary for MMDeploy when using the RKNN backend. It includes common configurations like mean/std values, target platform, optimization level, and input size list.

```python
backend_config = dict(
    type='rknn',
    common_config=dict(
        mean_values=None,
        std_values=None,
        target_platform='rk3588',
        optimization_level=3),
    quantization_config=dict(do_quantization=False, dataset=None),
    input_size_list=[[3, 224, 224]])
```

--------------------------------

### Build ONNXRuntime Custom Ops for MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds custom operations for ONNXRuntime with MMDeploy. Requires ONNXRuntime to be installed. Outputs are typically installed to a specified directory.

```bash
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
make -j$(nproc) && make install
```

--------------------------------

### Install TensorRT

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs TensorRT by downloading a specific version, extracting it, and setting environment variables. It also includes installing the PyCUDA package.

```shell
cd /the/path/of/tensorrt/tar/gz/file
tar -zxvf TensorRT-8.2.3.0.Linux.x86_64-gnu.cuda-11.4.cudnn8.2.tar.gz
pip install TensorRT-8.2.3.0/python/tensorrt-8.2.3.0-cp37-none-linux_x86_64.whl
export TENSORRT_DIR=$(pwd)/TensorRT-8.2.3.0
export LD_LIBRARY_PATH=$TENSORRT_DIR/lib:$LD_LIBRARY_PATH
pip install pycuda
```

--------------------------------

### Install PPQ for Quantization

Source: https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/quantize_model

Installs the PPQ (Post-training Quantization) library from its GitHub repository, which is a dependency for enabling quantization within MMDeploy. This involves cloning the repository, installing requirements, and setting up the package.

```shell
git clone https://github.com/openppl-public/ppq.git
cd ppq
pip install -r requirements.txt
python3 setup.py install
```

--------------------------------

### Install CMake 3.20.0

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs a specific version of CMake (3.20.0) from a tarball and links its binaries to the system's PATH. This ensures a compatible CMake version for building.

```bash
wget https://github.com/Kitware/CMake/releases/download/v3.20.0/cmake-3.20.0-linux-x86_64.tar.gz
tar -xzvf cmake-3.20.0-linux-x86_64.tar.gz
sudo ln -sf $(pwd)/cmake-3.20.0-linux-x86_64/bin/* /usr/bin/

```

--------------------------------

### Configure ncnn

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds ncnn with Python support enabled, exports its root path, and installs the pyncnn package.

```shell
cd ncnn
export NCNN_DIR=$(pwd)
export LD_LIBRARY_PATH=${NCNN_DIR}/build/install/lib/:$LD_LIBRARY_PATH
cd ${NCNN_DIR}/python
pip install -e .
```

--------------------------------

### Build MMDeploy SDK and Demo for CUDA + TensorRT

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds the MMDeploy SDK and demo applications for CUDA devices with TensorRT as the inference engine. Includes support for pplcv, TensorRT, and cuDNN.

```bash
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake .. \
    -DCMAKE_CXX_COMPILER=g++-7 \
    -DMMDEPLOY_BUILD_SDK=ON \
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
    -DMMDEPLOY_BUILD_EXAMPLES=ON \
    -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
    -DMMDEPLOY_TARGET_BACKENDS=trt \
    -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/install/lib/cmake/ppl \
    -DTENSORRT_DIR=${TENSORRT_DIR} \
    -DCUDNN_DIR=${CUDNN_DIR}

make -j$(nproc) && make install
```

--------------------------------

### Perform Object Detection with MMDeploy Python SDK

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

Demonstrates how to use the MMDeploy Python SDK's `Detector` class to perform object detection. It loads a model, runs inference on an image, and draws bounding boxes on the detected objects.

```python
from mmdeploy_runtime import Detector
import cv2

img = cv2.imread('mmdetection/demo/demo.jpg')

# create a detector
detector = Detector(model_path='mmdeploy_models/faster-rcnn', device_name='cuda', device_id=0)
# run the inference
bboxes, labels, _ = detector(img)
# Filter the result according to threshold
indices = [i for i in range(len(bboxes))]
for index, bbox, label_id in zip(indices, bboxes, labels):
  [left, top, right, bottom], score = bbox[0:4].astype(int),  bbox[4]
  if score < 0.3:
      continue
  cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))

cv2.imwrite('output_detection.png', img)
```

--------------------------------

### MMDeploy RKNN Model Deployment Example

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/rockchip

This command demonstrates how to deploy a classification model using MMDeploy with the RKNN backend. It specifies configuration files, model weights, an input image, and output directory.

```bash
python tools/deploy.py \
    configs/mmpretrain/classification_rknn-fp16_static-224x224.py \
    /mmpretrain_dir/configs/resnet/resnet50_8xb32_in1k.py \
    https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_batch256_imagenet_20200708-cfb998bf.pth \
    /mmpretrain_dir/demo/demo.JPEG \
    --work-dir ../resnet50 \
    --device cpu
```

--------------------------------

### Build ppl.cv with CUDA Support

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Clones the ppl.cv repository, checks out a specific version (v0.7.0), sets the PPLCV_DIR environment variable, and builds the library with CUDA support. This is an optional dependency for specific MMDeploy features.

```bash
git clone https://github.com/openppl-public/ppl.cv.git
cd ppl.cv
export PPLCV_DIR=$(pwd)
git checkout tags/v0.7.0 -b v0.7.0
./build.sh cuda

```

--------------------------------

### Install spdlog

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/jetsons

Installs spdlog, a fast C++ logging library required for the C/C++ Inference SDK.

```bash
sudo apt-get install -y libspdlog-dev
```

--------------------------------

### Create and Activate Conda Environment

Source: https://mmdeploy.readthedocs.io/en/latest/get_started

This snippet demonstrates how to create a new conda environment named 'mmdeploy' with Python 3.8 and then activate it. This is a common first step for managing project dependencies.

```bash
conda create --name mmdeploy python=3.8 -y
conda activate mmdeploy
```

--------------------------------

### Install OpenVINO with pip

Source: https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/openvino

Installs the OpenVINO development package including ONNX support using pip. Ensure you have pip installed and a compatible Python version.

```bash
pip install openvino-dev[onnx]==2022.3.0
```

--------------------------------

### Install MMDeploy for SNPE - Bash

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/snpe

Steps to set up the environment and install MMDeploy after downloading the SNPE package. This involves exporting environment variables for Python path, library path, and executable path, followed by a pip installation.

```bash
$ unzip snpe-1.59.0.zip
$ export SNPE_ROOT=${PWD}/snpe-1.59.0.3230
$ cd /path/to/mmdeploy
$ export PYTHONPATH=${PWD}/service/snpe/client:${SNPE_ROOT}/lib/python:${PYTHONPATH}
$ export LD_LIBRARY_PATH=${SNPE_ROOT}/lib/x86_64-linux-clang:${LD_LIBRARY_PATH}
$ export PATH=${SNPE_ROOT}/bin/x86_64-linux-clang:${PATH}
$ python3 -m pip install -e .
```

--------------------------------

### Install Protobuf Dependencies

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/jetsons

Installs protobuf-compiler and libprotoc-dev, which are required if the ONNX installation fails with a 'Protobuf compiler not found' error.

```bash
sudo apt-get install protobuf-compiler libprotoc-dev
```

--------------------------------

### Download and Install OpenVINO Runtime

Source: https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/openvino

Downloads the OpenVINO runtime for SDK using wget, extracts the tarball, navigates into the directory, sets the InferenceEngine_DIR environment variable, and runs a script to install OpenVINO dependencies. This is optional and for SDK usage.

```bash
wget https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3/linux/l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64.tgz
tar xzf ./l_openvino_toolkit*.tgz
cd l_openvino*
export InferenceEngine_DIR=$pwd/runtime/cmake
bash ./install_dependencies/install_openvino_dependencies.sh
```

--------------------------------

### Make Environment Variables Permanent

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Appends environment variable settings to ~/.bashrc to make them permanent, using ONNXRuntime as an example.

```shell
echo '# set env for onnxruntime' >> ~/.bashrc
echo "export ONNXRUNTIME_DIR=${ONNXRUNTIME_DIR}" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH" >> ~/.bashrc
source ~/.bashrc
```

--------------------------------

### Example Deployment Config Filename

Source: https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/write_config

Provides an example of a deployment configuration filename adhering to the specified naming convention, including task name, backend with quantization type, and dynamic input shapes.

```text
detection_tensorrt-int8_dynamic-320x320-1344x1344.py
```

--------------------------------

### Load Model from Disk (C++)

Source: https://mmdeploy.readthedocs.io/en/latest/sdk_usage/quick_start

Demonstrates how to load an MMDeploy model from disk using its path. The SDK can read a directory or a zip archive (if built with `-DMMDEPLOY_ZIP_MODEL=ON`). It then creates a classifier instance.

```cpp
#include <fstream>
#include <string>
#include <opencv2/core.hpp>
#include <mmdeploy/classifier.h>

// ...

std::string model_path = "/data/resnet"; // or "/data/resnet.zip" if build with `-DMMDEPLOY_ZIP_MODEL=ON`
mmdeploy_model_t model;
mmdeploy_model_create_by_path(model_path.c_str(), &model);

mmdeploy_classifier_t classifier{};
mmdeploy_classifier_create(model, "cpu", 0, &classifier);

```

--------------------------------

### Install MMDeploy Model Converter

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs the MMDeploy model converter using the 'mim install' command. This command installs the package in editable mode.

```bash
cd ${MMDEPLOY_DIR}
mim install -e .
```

--------------------------------

### Build MMDeploy SDK with OpenVINO

Source: https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/openvino

Builds the MMDeploy SDK with OpenVINO support. It requires navigating to the MMDeploy root directory, creating a build directory, configuring the build with specific OpenVINO parameters using cmake, and then compiling and installing the SDK.

```bash
cd ${MMDEPLOY_DIR} # To MMDeploy root directory
mkdir -p build && cd build
cmake -DMMDEPLOY_TARGET_DEVICES='cpu' -DMMDEPLOY_TARGET_BACKENDS=openvino -DInferenceEngine_DIR=${InferenceEngine_DIR} ..
make -j$(nproc) && make install
```

--------------------------------

### MMDeploy Installation

Source: https://mmdeploy.readthedocs.io/en/latest/04-supported-codebases/mmpretrain

Provides instructions for installing MMDeploy using precompiled packages or by building from source scripts or directly from source.

```APIDOC
## MMDeploy Installation

### Method I: Install precompiled package
Refer to the get_started guide for installation.

### Method II: Build using scripts
For Ubuntu 18.04 or later:
```bash
git clone --recursive -b main https://github.com/open-mmlab/mmdeploy.git
cd mmdeploy
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
```

### Method III: Build from source
Use this method if the above options do not meet your requirements.
```

--------------------------------

### Load Model from Memory (C++)

Source: https://mmdeploy.readthedocs.io/en/latest/sdk_usage/quick_start

Illustrates loading an MMDeploy model directly from memory, typically from a zip archive. The code reads the zip file's content into a string and then uses it to create the model.

```cpp
#include <fstream>
#include <string>
#include <vector>
#include <opencv2/core.hpp>
#include <mmdeploy/classifier.h>

// ...

std::string model_path = "/data/resnet.zip"
std::ifstream ifs(model_path, std::ios::binary); // /path/to/zipmodel
ifs.seekg(0, std::ios::end);
auto size = ifs.tellg();
ifs.seekg(0, std::ios::beg);
std::string str(size, '\0'); // binary data, should decrypt if it's encrypted
ifs.read(str.data(), size);

mmdeploy_model_t model;
mmdeploy_model_create(str.data(), size, &model);

mmdeploy_classifier_t classifier{};
mmdeploy_classifier_create(model, "cpu", 0, &classifier);

```

--------------------------------

### Build TorchScript Custom Ops for MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds custom operations for TorchScript with MMDeploy. Requires PyTorch installation. The CMake configuration specifies the Torch installation directory.

```bash
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=torchscript -DTorch_DIR=${Torch_DIR} ..
make -j$(nproc) && make install
```

--------------------------------

### Base Backend Manager `to_backend` Example — MMDeploy 0.11.0 documentation

Source: https://mmdeploy.readthedocs.io/en/latest/07-developer-guide/support_new_backend

This example demonstrates the implementation of the `to_backend` static method for a backend manager class, which derives from `BaseBackendManager`. It takes intermediate representation (IR) files, a deploy configuration, working directory, log level, and device as input, and returns the IR files.

```python
@classmethod
def to_backend(cls,
             ir_files: Sequence[str],
             deploy_cfg: Any,
             work_dir: str,
             log_level: int = logging.INFO,
             device: str = 'cpu',
             **kwargs) -> Sequence[str]:
     return ir_files
```

--------------------------------

### Build TensorRT Custom Ops for MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds custom operations for TensorRT with MMDeploy. Requires TensorRT and cuDNN installations. Outputs are typically installed to a specified directory.

```bash
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=trt -DTENSORRT_DIR=${TENSORRT_DIR} -DCUDNN_DIR=${CUDNN_DIR} ..
make -j$(nproc) && make install
```

--------------------------------

### Install MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/04-supported-codebases/mmpose

Provides instructions for installing MMDeploy, including options for precompiled packages, building with scripts for specific platforms, and building from source.

```APIDOC
## Install MMDeploy

There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.

### Method I: Install precompiled package

You can refer to [get_started](https://mmdeploy.readthedocs.io/en/latest/get_started.html).

### Method II: Build using scripts

If your target platform is **Ubuntu 18.04 or later version** , we encourage you to run scripts. For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.

```bash
git clone --recursive -b main https://github.com/open-mmlab/mmdeploy.git
cd mmdeploy
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
```

### Method III: Build from source

If neither **I** nor **II** meets your requirements, building mmdeploy from source is the last option.
```

--------------------------------

### Download and Setup ONNX Runtime Library (CPU)

Source: https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/onnxruntime

Downloads the ONNX Runtime library for Linux x64 CPU and sets environment variables for its usage. This is needed for building custom ops.

```bash
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
cd onnxruntime-linux-x64-1.8.1
export ONNXRUNTIME_DIR=$(pwd)
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH

```

--------------------------------

### Build ncnn Custom Ops for MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Builds custom operations for ncnn with MMDeploy. Requires ncnn to be built and installed. The CMake configuration points to the ncnn installation directory.

```bash
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=ncnn -Dncnn_DIR=${NCNN_DIR}/build/install/lib/cmake/ncnn ..
make -j$(nproc) && make install
```

--------------------------------

### Install OpenCV Development Files on Ubuntu

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs the OpenCV development header files and libraries on Ubuntu systems (version >= 18.04). These are required for applications that use OpenCV functionalities.

```bash
sudo apt-get install libopencv-dev

```

--------------------------------

### Run Image Classification Demo with Python SDK (ONNX Runtime)

Source: https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/prebuilt_package_windows

Executes an image classification demo using the MMDeploy Python SDK with an ONNX Runtime backend. It requires specifying the device, path to the converted model, and the input image.

```bash
python .\mmdeploy\demo\python\image_classification.py cpu .\work_dir\onnx\resnet\ .\mmpretrain\demo\demo.JPEG
```

--------------------------------

### Download and Setup ONNX Runtime Library (GPU x64)

Source: https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/onnxruntime

Downloads the ONNX Runtime library for Linux x64 GPU and sets environment variables. This is required for building custom ops with GPU support.

```bash
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-gpu-1.8.1.tgz
cd onnxruntime-linux-x64-gpu-1.8.1
export ONNXRUNTIME_DIR=$(pwd)
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH

```

--------------------------------

### Install NVIDIA Container Toolkit

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/build_from_docker

Installs the NVIDIA Container Toolkit on Ubuntu-based systems to enable GPU support in Docker containers. This is a prerequisite for running GPU-accelerated Docker applications.

```bash
# Add the package repositories
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker
```

--------------------------------

### Deploy Model using MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/openvino

Example of deploying a model using MMDeploy's deploy.py script. It specifies the MMDeploy configuration, the original model's configuration, the model's checkpoint path, an input image, and deployment options like work directory, device, and log level.

```python
python tools/deploy.py \
    configs/mmdet/detection/detection_openvino_static-300x300.py \
    /mmdetection_dir/mmdetection/configs/ssd/ssd300_coco.py \
    /tmp/snapshots/ssd300_coco_20210803_015428-d231a06e.pth \
    tests/data/tiger.jpeg \
    --work-dir ../deploy_result \
    --device cpu \
    --log-level INFO
```

--------------------------------

### Install PyTorch 1.8.0 with CUDA 11.1

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs PyTorch version 1.8.0 and Torchvision version 0.9.0, specifically compiled with CUDA 11.1 support. This command ensures compatibility with CUDA-enabled GPU acceleration.

```bash
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1 -c pytorch -c conda-forge

```

--------------------------------

### Install GCC 7 and G++ 7

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Installs GCC and G++ version 7 on Ubuntu systems, which is required for C++17 support needed by MMDeploy. It adds a PPA for older Ubuntu versions to access the required compiler versions.

```bash
# Add repository if ubuntu < 18.04
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-7
sudo apt-get install g++-7

```

--------------------------------

### Get PyTorch Installation Path

Source: https://mmdeploy.readthedocs.io/en/latest/faq

A Python snippet to help users find their PyTorch installation directory, which can be useful for troubleshooting DLL-related issues.

```python
import torch
print(torch.__file__)

```

--------------------------------

### Constructing Deploy Config - Python

Source: https://mmdeploy.readthedocs.io/en/latest/07-developer-guide/support_new_backend

Demonstrates how to create a complete deploy configuration by inheriting from a base backend configuration, specifying codebase details, and ONNX input shapes.

```python
_base_ = ['../_base_/backends/onnxruntime.py']

codebase_config = dict(type='mmpretrain', task='Classification')

```

--------------------------------

### Load Input Image (C++)

Source: https://mmdeploy.readthedocs.io/en/latest/sdk_usage/quick_start

Shows how to load an image file using OpenCV for model inference. The loaded image is typically a `cv::Mat` object.

```cpp
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>

// ...

cv::Mat img = cv::imread(image_path);

```

--------------------------------

### Install RKNN Python Package

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/rockchip

Installs the RKNN Python package from a local wheel file. The `--no-deps` flag is recommended to prevent potential dependency conflicts with MMDeploy.

```bash
pip install packages/rknn_toolkit2-1.4.0_22dcfef4-cp36-cp36m-linux_x86_64.whl --no-deps
```

--------------------------------

### ONNX Runtime Inference Example (C++)

Source: https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/prebuilt_package_windows

This snippet demonstrates how to run an image classification inference using ONNX Runtime backend with MMDeploy. It requires the `image_classification.exe` executable and specifies the backend ('cpu'), model directory, and input image path.

```bash
.\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmpretrain\demo\demo.JPEG
```

--------------------------------

### Install PyTorch and Torchvision with Conda

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/rockchip

Installs specific versions of PyTorch and Torchvision using Conda, including the CUDA toolkit. This is a common method for managing deep learning environment dependencies.

```bash
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1 -c pytorch -c conda-forge
```

--------------------------------

### Build MMDeploy

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Sets the MMDEPLOY_DIR environment variable to the root path of the MMDeploy project.

```shell
cd /the/root/path/of/MMDeploy
export MMDEPLOY_DIR=$(pwd)
```

--------------------------------

### Configure cuDNN

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Extracts the cuDNN compressed file and sets environment variables for its directory and library path.

```shell
cd /the/path/of/cudnn/tgz/file
tar -zxvf cudnn-11.3-linux-x64-v8.2.1.32.tgz
export CUDNN_DIR=$(pwd)/cuda
export LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
```

--------------------------------

### Compile MMDeploy SDK for RISC-V with ncnn Backend (CMake)

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/riscv

This CMake configuration builds the MMDeploy SDK and examples for RISC-V, targeting the ncnn backend and CPU device. It requires specifying the paths to the MMDeploy toolchain, ncnn installation, and OpenCV installation.

```bash
cd /path/to/mmdeploy
mkdir build_riscv && cd build_riscv
cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/riscv64-unknown-linux-gnu.cmake \
  -DMMDEPLOY_BUILD_SDK=ON \
  -DMMDEPLOY_SHARED_LIBS=OFF \
  -DMMDEPLOY_BUILD_EXAMPLES=ON \
  -DMMDEPLOY_TARGET_DEVICES="cpu" \
  -DMMDEPLOY_TARGET_BACKENDS="ncnn" \
  -Dncnn_DIR=${ncnn_DIR}/build-c906/install/lib/cmake/ncnn/ \
  -DMMDEPLOY_CODEBASES=all \
  -DOpenCV_DIR=${OpenCV_DIR}/build_riscv/install/lib/cmake/opencv4

make -j$(nproc) && make install

```

--------------------------------

### Download and Build ncnn

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/windows

Clones the ncnn repository, creates a build directory, and configures the CMake project to build ncnn. It specifies the installation prefix, disables Vulkan support, and points to the previously compiled protobuf installation.

```bash
git clone --recursive https://github.com/Tencent/ncnn.git
cd <ncnn-dir>
mkdir -p ncnn_build
cd ncnn_build
cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_INSTALL_PREFIX=%cd%/install -Dprotobuf_DIR=/build/install/cmake -DNCNN_VULKAN=OFF ..
cmake --build . --config Release -j 2
cmake --build . --config Release --target install
```

--------------------------------

### Create and Activate Conda Environment

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/linux-x86_64

Creates a new Conda virtual environment named 'mmdeploy' with Python 3.7 and activates it. This isolates project dependencies.

```bash
conda create -n mmdeploy python=3.7 -y
conda activate mmdeploy

```

--------------------------------

### Install MMDeploy with NCNN Backend (Ubuntu x64)

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/build_from_script

This script installs MMDeploy along with the NCNN backend on Ubuntu x64 systems. It requires Python3, nvcc, and optionally PyTorch. The script automates dependency detection, compilation of necessary libraries like pyncnn and protobuf, and installation of MMDeploy SDK and demos.

```shell
cd /path/to/mmdeploy
python3 tools/scripts/build_ubuntu_x64_ncnn.py
```

--------------------------------

### Clone RKNN-Toolkit2 Repository

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/rockchip

This command clones the RKNN-Toolkit2 repository from GitHub, which is necessary for building MMDeploy for Rockchip NPUs. Ensure you have Git installed.

```bash
git clone git@github.com:rockchip-linux/rknn-toolkit2.git
```

--------------------------------

### Run MMDeploy Docker Container

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/build_from_docker

Launches a Docker container for MMDeploy with GPU acceleration enabled. This command assumes a Docker image has been pulled or built.

```bash
export TAG=openmmlab/mmdeploy:ubuntu20.04-cuda11.8-mmdeploy
docker run --gpus=all -it --rm $TAG
```

--------------------------------

### Run Object Detection Demo

Source: https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/jetsons

Executes the object detection demo using the compiled SDK. It requires specifying the target device (e.g., 'cuda'), the directory containing the converted models, and the path to an image for inference.

```bash
./object_detection cuda ${directory/to/the/converted/models} ${path/to/an/image}
```
Tokens:
Raw