### Install packages for Backend.AI

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/llama3-8B

Installs required packages for Backend.AI including rebel-compiler, optimum-rbln, and vllm-rbln from a custom PyPI repository.

```bash
$ pip3 install --extra-index-url https://pypi.rbln.ai/simple/ "rebel-compiler>=0.9.2.post1" "optimum-rbln>=0.9.2" "vllm-rbln>=0.9.2.post1"
```

--------------------------------

### Install RBLN SDK

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/resnet50

Installs the RBLN SDK using pip. This is a prerequisite for running the inference server and making requests. Ensure you have Python and pip installed.

```bash
pip3 install -i https://pypi.rbln.ai/simple/ rebel-compiler
```

--------------------------------

### Install Optimum RBLN and Rebel Compiler

Source: https://docs.rbln.ai/latest/ko/software/optimum/tutorial/sdxl_turbo

Installs the necessary libraries for using SDXL-Turbo with Optimum RBLN. Requires pip and the RBLN portal account for rebel-compiler.

```shell
pip install optimum-rbln
pip install --extra-index-url https://pypi.rbln.ai/simple/ rebel-compiler>=0.9.2.post1
```

--------------------------------

### Start TorchServe server in Bash

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/llama3-8B

Starts the TorchServe server with specified model store and model for serving. Depends on TorchServe installation and the archived model. Inputs are server options like model store path and model name; output is a running server process on default ports. Limitations include potential port conflicts and requirement for proper model configuration.

```bash
$ torchserve --start --ncs --model-store model_store --models llama3-8b --disable-token-auth
```

--------------------------------

### Starting and Stopping TorchServe

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/resnet50

Instructions on how to start the TorchServe process with specified configurations and how to stop it. It also covers default port usage and checking the status of deployed models.

```APIDOC
## Starting and Stopping TorchServe

### Description
Learn how to launch TorchServe with your archived models and how to gracefully stop the server. This section also covers checking server status and understanding port configurations.

### Start TorchServe:

Use the `torchserve` command with relevant options to start the model serving process. Token authentication can be disabled for simpler testing.

```bash
torchserve --start --ncs \
        --ts-config ./config.properties \
        --model-store ./model_store \
        --models resnet50=resnet50.mar \
        --disable-token-auth
```

**Options:**

*   `--start`: Initiates the model serving.
*   `--ncs`: Disables configuration snapshots (`--no-config-snapshots`).
*   `--ts-config`: Specifies the configuration file for TorchServe (e.g., `./config.properties`).
*   `--model-store`: Sets the path for loading models (e.g., `./model_store`).
*   `--models`: Defines the models to be served. Use `all` to serve all models in the store. Format: `modelName=model.mar`.
*   `--disable-token-auth`: Disables token-based authorization.

### Stop TorchServe:

To stop the running TorchServe process:

```bash
torchserve --stop
```

### Port Configuration:

*   **Management API**: Default port `8081`.
*   **Inference API**: Default port `8080`.

### Check Model Status:

Query the Management API to list deployed models:

```bash
curl -X GET "http://localhost:8081/models"
```

**Expected Success Response:**

```json
{
  "models": [
    {
      "modelName": "resnet50",
      "modelUrl": "resnet50.mar"
    }
  ]
}
```
```

--------------------------------

### Start OpenAI API Server

Source: https://docs.rbln.ai/latest/ko/software/model_serving/vllm_support/tutorial/openai_api_server

This section explains how to start the OpenAI API server using the vllm module. You need to specify the model to be used.

```APIDOC
## POST /websites/rbln_ai_ko

### Description
Starts an OpenAI-compatible API server using vLLM with a specified model.

### Method
GET

### Endpoint
/websites/rbln_ai_ko

### Parameters
#### Query Parameters
- **model** (string) - Required - The model to be loaded and served by the API.

### Request Example
```bash
python3 -m vllm.entrypoints.openai.api_server --model rbln-Llama-3-8B-Instruct
```

### Response
#### Success Response (200)
- The API server starts and listens for requests on the specified host and port.

#### Response Example
(No specific response body for server startup, typically logs to the console.)
```

--------------------------------

### Start Triton Inference Server with Backend.AI

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/resnet50

Starts the Triton inference server using a Docker container provided by Backend.AI. It specifies the model repository path. This command assumes Backend.AI is already set up.

```bash
tritonserver --model-repository /opt/tritonserver/python_backend/examples/rbln
```

--------------------------------

### Install Triton Client and Dependencies

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/resnet50

Installs the Triton client library along with `gevent` and `geventhttpclient` for asynchronous operations. This is required to make HTTP API requests to the Triton inference server.

```bash
pip3 install tritonclient==2.41.1 gevent geventhttpclient fire
```

--------------------------------

### Start Triton server with Backend.AI

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/llama3-8B

Starts the Triton inference server with Backend.AI, specifying the model repository path and environment variable for VLLM.

```bash
$ VLLM_USE_V1=0 tritonserver --model-repository PATH/TO/YOUR/vllm_backend/samples/model_repository
```

--------------------------------

### Install torch-rbln package via pip

Source: https://docs.rbln.ai/latest/ko/software/rbln_pytorch/tutorial_running_n_debugging

Installs the torch-rbln library from the RBLN PyPI index using pip. Requires internet access and a valid RBLN Portal account. output is captured here.

```bash
pip installi https://pypi.rbln.ai/simple/ torch-rbln
```

--------------------------------

### Download Sample Image

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/resnet50

Downloads a sample image file (`tabby.jpg`) from a public S3 bucket. This image can be used as input for inference requests to test the setup.

```bash
wget https://rbln-public.s3.ap-northeast-2.amazonaws.com/images/tabby.jpg
```

--------------------------------

### Install RBLN PyTorch via pip

Source: https://docs.rbln.ai/latest/ko/software/rbln_pytorch/tutorial_llama

Installs the RBLN PyTorch package from the official RBLN PyPI repository. Requires RBLN Portal account for authentication. Essential first step before running transformers models with RBLN acceleration.

```bash
pip install -i https://pypi.rbln.ai/simple/ torch-rbln
```

--------------------------------

### TorchServe Configuration Properties

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/resnet50

Configuration file (`config.properties`) for TorchServe, defining model serving parameters. This example sets `default_workers_per_model` to 1 and specifies the `resnet50` model with version `1.0`, pointing to its `.mar` file and setting a response timeout.

```properties
default_workers_per_model:1

models={
    "resnet50":{
        "1.0":{
            "marName": "resnet50.mar",
            "responseTimeout": 120
        }
    }
}

```

--------------------------------

### GET /models/{model-name}

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/resnet50

Retrieves the status and configuration of a specific model. This endpoint is useful for verifying that batching configurations like batchSize and maxBatchDelay have been applied correctly.

```APIDOC
## GET /models/{model-name}

### Description
Retrieves the status and configuration of a specific model. This endpoint is useful for verifying that batching configurations like `batchSize` and `maxBatchDelay` have been applied correctly.

### Method
GET

### Endpoint
/models/{model-name}

### Parameters
#### Path Parameters
- **model-name** (string) - Required - The name of the model to retrieve information for.

### Request Example
```bash
curl -X GET "http://localhost:8081/models/resnet50"
```

### Response
#### Success Response (200)
- **modelName** (string) - The name of the model.
- **modelVersion** (string) - The version of the model.
- **modelUrl** (string) - The path to the model archive.
- **runtime** (string) - The runtime environment for the model.
- **minWorkers** (integer) - The minimum number of workers for the model.
- **maxWorkers** (integer) - The maximum number of workers for the model.
- **batchSize** (integer) - The batch size configured for the model.
- **maxBatchDelay** (integer) - The maximum delay for batching in milliseconds.
- **workers** (array) - A list of worker details.

#### Response Example
```json
[
  {
    "modelName": "resnet50",
    "modelVersion": "1.0",
    "modelUrl": "resnet50.mar",
    "runtime": "python",
    "minWorkers": 1,
    "maxWorkers": 1,
    "batchSize": 4,
    "maxBatchDelay": 100,
    "workers": [
      {
        
      }
    ]
  }
]
```
```

--------------------------------

### Install TorchServe Dependencies (Bash)

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/torchserve

This snippet demonstrates how to install TorchServe dependencies using a Git clone and installation script. It is used to setup the necessary packages for TorchServe, including torchserve, torch-model-archiver and torch-workflow-archiver.

```bash
git clone https://github.com/pytorch/serve.git
cd serve
python3 ./ts_scripts/install_dependencies.py
pip3 install torchserve torch-model-archiver torch-workflow-archiver
```

--------------------------------

### Building RBLN C++ Example with CMake

Source: https://docs.rbln.ai/latest/ko/software/api/language_binding/c/tutorial/image_classification

Compiles the RBLN image classification example using CMake and Make. Assumes source in ${SAMPLE_PATH}; creates build directory and generates executables. Requires CMake and Make installed; no additional inputs needed.

```bash
$ mkdir ${SAMPLE_PATH}/build
$ cd ${SAMPLE_PATH}/build
$ cmake ..
$ make
```

--------------------------------

### Install RBLN Python packages with pip3

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/llama3

Installs required RBLN packages (rebel-compiler, optimum-rbln, vllm-rbln) from the custom PyPI repository. This command is needed for both Backend.AI and Docker container setups. Uses pip3 with an extra index URL pointing to the RBLN package repository.

```shell
$ pip3 install --extra-index-url https://pypi.rbln.ai/simple/ "rebel-compiler>=0.9.2.post1" "optimum-rbln>=0.9.2" "vllm-rbln>=0.9.2.post1"
```

```shell
$ pip3 install --extra-index https://pypi.rbln.ai/simple/ "rebel-compiler>=0.9.2.post1" "optimum-rbln>=0.9.2" "vllm-rbln>=0.9.2.post1"
```

--------------------------------

### RBLN 드라이버 설치 확인

Source: https://docs.rbln.ai/latest/ko/getting_started/quick_start

시스템에서 RBLN 드라이버가 설치되었는지, RBLN NPU가 존재하는지 확인하기 위해 `rbln-stat` 명령어를 실행합니다. 이 명령어는 NPU 장치 정보 및 상태를 표시합니다.

```shell
rbln-stat
```

--------------------------------

### Prepare ResNet50 Model Directory

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/resnet50

컴파일된 `resnet50.rbln` 모델 파일을 Triton `python_backend`의 예제 디렉토리 구조에 맞게 복사합니다. 이는 Triton 서버가 모델을 올바르게 로드하고 접근할 수 있도록 하기 위함입니다.

```bash
mkdir -p python_backend/examples/rbln/resnet50/1
mv resnet50.rbln python_backend/examples/rbln/resnet50/1/
```

--------------------------------

### Install RBLN Llama3-8B Dependencies

Source: https://docs.rbln.ai/latest/ko/software/profiler/examples/text_generation

This bash script installs the required Python packages for the RBLN Llama3-8B model. It changes to the model directory and runs pip install with requirements.txt. Dependencies include RBLN-specific libraries; inputs are none beyond the requirements file; outputs are installed packages. Limitation: Assumes Python 3 and pip are available.

```bash
$ cd rbln-model-zoo/huggingface/transformers/text2text-generation/llama/llama3-8b
$ pip3 install -r requirements.txt
```

--------------------------------

### Install RBLN Compiler and TensorFlow

Source: https://docs.rbln.ai/latest/ko/software/api/python/tutorial/basic/tensorflow_efficientnetb0

Installs the necessary RBLN compiler and TensorFlow packages. Ensure you have Python 3.9-3.12 and the RBLN driver installed. Requires an RBLN Portal account for the rebel-compiler.

```bash
pip install tensorflow numpy
pip install --extra-index-url https://pypi.rbln.ai/simple/ rebel-compiler>=0.9.2.post1
```

--------------------------------

### ResNet50 모델 서빙 핸들러 - Python

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/resnet50

TorchServe에서 ResNet50 모델의 요청을 처리하기 위한 사용자 정의 핸들러입니다. BaseHandler를 상속받아 initialize, inference, postprocess, handle 메서드를 구현합니다. 모델 로드 시 initialize가 호출되고, 예측 요청 시 handle이 호출됩니다.

```python
from abc import ABC, abstractmethod
from io import BytesIO
from typing import Any, Dict, List, Optional, Tuple

import numpy as np

import rbln_infer


def _to_image(data: bytes) -> np.ndarray:
    from PIL import Image

    return np.array(Image.open(BytesIO(data)))


def _to_tensor(data: bytes) -> np.ndarray:
    return np.frombuffer(data, dtype=np.float32)


class BaseHandler(ABC):
    def __init__(self) -> None:
        self.initialized = False
        self.model = None

    def initialize(self, context: Any) -> None:
        properties = context.system_properties
        self.manifest = context.manifest
        model_dir = properties.get("model_dir")

        if "model_signature" in self.manifest:
            signature = self.manifest["model_signature"]
        else:
            signature = None

        self.model = rbln_infer.RBLNModel(
            model_dir=model_dir,
            signature=signature,
            compiler_kwargs={ "log_level": "ERROR" })

        self.initialized = True

    def preprocess(self, request: List[Dict[str, Any]]) -> List[Any]:
        input_datas = []
        for data in request:
            input_data = data.get("body")
            if input_data is None:
                raise ValueError("Request body is missing.")

            content_type = data.get("content_type")
            if content_type == "application/octet-stream":
                input_datas.append(_to_image(input_data))
            elif content_type == "application/json":
                input_datas.append(_to_tensor(input_data))
            else:
                raise ValueError(f"Unsupported content type: {content_type}")

        return input_datas

    def inference(self, data: List[Any]) -> List[Any]:
        if not self.initialized:
            raise RuntimeError("Handler not initialized.")
        return self.model.run(data)

    def postprocess(self, data: List[Any]) -> List[Any]:
        return data

    def handle(self, request: List[Dict[str, Any]]) -> List[Any]:
        if not self.initialized:
            self.initialize(None)

        input_datas = self.preprocess(request)
        inference_result = self.inference(input_datas)
        return self.postprocess(inference_result)


class ResNet50Handler(BaseHandler):
    def __init__(self) -> None:
        super().__init__()

    def postprocess(self, data: List[Any]) -> List[Any]:
        # ResNet50 모델의 출력은 softmax 적용 후의 확률값입니다.
        # 따라서, 후처리 과정에서는 별도의 작업 없이 그대로 반환합니다.
        return data


_service = ResNet50Handler()

def handle(context):
    return _service.handle(context)
```

--------------------------------

### Install packages in Docker container

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/llama3-8B

Installs required packages inside the Docker container from a custom PyPI repository.

```bash
$ pip3 install --extra-index https://pypi.rbln.ai/simple/ "rebel-compiler>=0.9.2.post1" "optimum-rbln>=0.9.2" "vllm-rbln>=0.9.2.post1"
```

--------------------------------

### Install Dependencies for RBLN SDK (Bash)

Source: https://docs.rbln.ai/latest/ko/software/api/language_binding/c/tutorial/object_detection

Installs required Python packages including Torch, Ultralytics, CMake, and RBLN compiler for model preparation and compilation. Requires Python 3.9-3.12 and RBLN driver. Inputs are pip commands; outputs installed packages. Limitation: Needs RBLN Portal account for rebel-compiler.

```bash
pip install torch ultralytics cmake
pip install --extra-index-url https://pypi.rbln.ai/simple/ rebel-compiler>=0.9.2.post1
```

--------------------------------

### 버케팅으로 ResNet50 모델 컴파일 (Python)

Source: https://docs.rbln.ai/latest/ko/software/api/python/tutorial/advanced/bucketing

정의된 다중 입력 형태(버켓)를 사용하여 PyTorch 모델을 RBLN SDK의 rebel.compile_from_torch 함수로 컴파일합니다. 컴파일된 모델은 'resnet50_bucketed.rbln' 파일로 저장될 수 있습니다. 이 과정은 각 버켓에 대한 최적화된 커널을 생성합니다.

```python
# 다중 입력 형태로 모델 컴파일
compiled_model = rebel.compile_from_torch(
    model,
    input_info=input_infos  # 버케팅을 위한 input_info 리스트 전달
)

# 컴파일된 모델 저장 (선택사항)
compiled_model.save("resnet50_bucketed.rbln")
```

--------------------------------

### Install RBLN Kubernetes Device Plugin

Source: https://docs.rbln.ai/latest/ko/software/system_management/kubernetes/device_plugin

Commands to install the RBLN Kubernetes Device Plugin using kubectl. This deploys the necessary ConfigMap and DaemonSet for the plugin.

```bash
$ kubectl apply -f https://raw.githubusercontent.com/rebellions-sw/rbln-k8s-device-plugin/master/deployments/rbln/configmap.yaml
$ kubectl apply -f https://raw.githubusercontent.com/rebellions-sw/rbln-k8s-device-plugin/master/deployments/rbln/daemonset.yaml
```

--------------------------------

### TorchServe 모델 서빙 시작 및 중지

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/yolov8

torchserve 명령어를 사용하여 YOLOv8 모델 서빙을 시작하고 중지하는 방법을 안내합니다. 서빙 시작 시 모델 스토어 경로, 설정 파일, 서빙할 모델 등을 지정할 수 있으며, 토큰 인증을 비활성화하는 옵션도 제공됩니다. 서빙 중지 명령은 간단히 TorchServe 프로세스를 종료합니다.

```bash
# TorchServe 시작
torchserve --start --ncs \
        --ts-config ./config.properties \
        --model-store ./model_store \
        --models yolov8l=yolov8l.mar \
        --disable-token-auth

# TorchServe 중지
torchserve --stop
```

--------------------------------

### torchserve 모델 서버 시작/중지

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/llama3

torchserve를 시작하고 중지하는 명령어를 포함하며, 모델 로딩 및 서버 관리를 위한 설정 파라미터가 포함된 스크립트

```bash
# Start torchserve server
$ torchserve --start --ncs --model-store model_store --models llama3.1-8b --disable-token-auth

# Stop torchserve server
$ torchserve --stop
```

--------------------------------

### 가변 이미지 크기 버켓팅 예제 (Python)

Source: https://docs.rbln.ai/latest/ko/software/api/python/tutorial/advanced/bucketing

다양한 이미지 해상도와 배치 크기를 지원하는 EfficientNet-B0 모델에 대한 버케팅 예제입니다. 여러 'configuration'을 정의하고 이를 기반으로 모델을 컴파일하여 다양한 입력 크기에 대한 추론을 테스트합니다. 이는 컴퓨터 비전 작업에서 유용합니다.

```python
import torch
import torchvision.models as models
import rebel

# 모델 로드
model = models.efficientnet_b0(pretrained=True).eval()

# 다양한 이미지 크기와 배치 크기 정의
configurations = [
    (1, 3, 224, 224),  # 표준 ImageNet 크기
    (1, 3, 256, 256),  # 약간 큰 크기
    (1, 3, 288, 288),  # 더 큰 크기
    (2, 3, 224, 224),  # 표준 크기의 배치 2
    (4, 3, 224, 224),  # 표준 크기의 배치 4
]

input_infos = []
for batch, channels, height, width in configurations:
    input_info = [("input", [batch, channels, height, width], "float32")]
    input_infos.append(input_info)

# 모든 구성으로 컴파일
compiled_model = rebel.compile_from_torch(
    model,
    input_info=input_infos
)

runtime = rebel.Runtime(compiled_model, tensor_type="pt")

# 다양한 입력 크기로 테스트
def test_inference(batch_size, height, width):
    # 랜덤 입력 생성
    test_input = torch.randn(batch_size, 3, height, width)

    # 추론 실행
    output = runtime(test_input)
    print(f"입력: {test_input.shape} -> 출력: {output.shape}")

# 다양한 구성 테스트
test_inference(1, 224, 224)
test_inference(1, 256, 256)
test_inference(1, 288, 288)
test_inference(2, 224, 224)
test_inference(4, 224, 224)
```

--------------------------------

### 컴파일된 Llama3.1-8B 모델 복사 (Bash)

Source: https://docs.rbln.ai/latest/ko/software/model_serving/nvidia_triton_inference_server/tutorial/llama3

미리 컴파일된 Llama3.1-8B 모델 파일을 Triton 모델 저장소의 해당 버전 디렉토리로 복사합니다. 이 단계는 Triton 서버가 모델을 로드하고 서빙할 수 있도록 모델 파일을 올바른 위치에 배치하는 데 필요합니다.

```bash
$ cp -R /PATH/TO/YOUR/Llama-3.1-8B-Instruct \
        /PATH/TO/YOUR/CLONED/vllm_backend/samples/model_repository/vllm_model/1/
```

--------------------------------

### Install required packages for BERT-base with RBLN

Source: https://docs.rbln.ai/latest/ko/software/api/python/tutorial/basic/pytorch_bert

Installation commands for torch, transformers, numpy, and the RBLN compiler. Requires a RBLN Portal account for the rebel-compiler.

```shell
pip install torch transformers numpy
pip install --extra-index-url https://pypi.rbln.ai/simple/ rebel-compiler>=0.9.2.post1
```

--------------------------------

### Create model store directory

Source: https://docs.rbln.ai/latest/ko/software/model_serving/torchserve/tutorial/llama3

모델 아카이브가 생성될 model_store 디랙토리를 만들고, torch-model-archiver를 사용하여 Llama3.1-8B 모델을 아카이브하는 스크립트

```bash
# Create model store directory
$ mkdir model_store

# Create model archive using torch-model-archiver
$ torch-model-archiver \
        --model-name llama3.1-8b \
        --version 1.0 \
        --handler ./rbln_vllm_handler.py \
        --config-file ./model_config.yaml \
        --archive-format no-archive \
        --export-path model_store/ \
        --extra-files rbln_model/
```