### Service Startup on IP2 Node (Work Node)

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Configures and starts the KsanaLLM service on a work node in a multi-node setup. Requires setting environment variables for distributed training and specifying TCP for inter-node communication when using multi-batch.

```bash
# Set the IP2 node as the work node.
export WORLD_SIZE=2
export NODE_RANK=1
export MASTER_HOST=master_node_ip
export MASTER_PORT=master_node_port
# Optimal environment variable configuration
export ENABLE_COMPRESSED_KV=2
export SELECT_ALL_REDUCE_BY_SIZE=1
export PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync
export USE_TCP_DATA_CHANNEL=1
# Service Startup
python serving_server.py \
--config_file ${GIT_PROJECT_REPO_ROOT}/examples/deepseek_fp8_perf.yaml \
--port service_port
```

--------------------------------

### Launch Single-Node Inference Server

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Setup environment and launch the serving server for single-node inference.

```bash
cd ${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/python
ln -s ${GIT_PROJECT_REPO_ROOT}/build/lib .

# download huggingface model for example:
# Note: Make sure git-lfs is installed.
git clone https://huggingface.co/NousResearch/Llama-2-7b-hf

# change the model_dir in ${GIT_PROJECT_REPO_ROOT}/examples/ksana_llm2-7b.yaml if needed

# set environment variable `KLLM_LOG_LEVEL=DEBUG` before run to get more log info
# the serving log locate in log/ksana_llm.log

# ${GIT_PROJECT_REPO_ROOT}/examples/ksana_llm2-7b.yaml's tensor_para_size equal the GPUs/NPUs number
export CUDA_VISIBLE_DEVICES=xx

# launch server
python serving_server.py \
    --config_file ${GIT_PROJECT_REPO_ROOT}/examples/ksana_llm2-7b.yaml \
    --port 8080
# KsanaLLM now supports the OpenAI API protocol and has implemented the core /v1/chat/completions API. 
# you can add specific parameters to enable tool invocation parsing and inference content analysis capabilities.
# tool-choice:
--enable-auto-tool-choice     --tool-call-parser deepseek_v3  \
# reasoning-parser:
--reasoning-parser  deepseek_r1 \
# apply specific chat-template:
--chat-template openaiapi/chat_templates/tool_chat_template_deepseekr1.jinja
```

--------------------------------

### Install Dependencies

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/simple_router/README.md

Installs necessary Python packages for running the service. Ensure pymysql is included for MySQL support.

```bash
pip install fastapi sqlalchemy httpx uvicorn pymysql  # add pymysql for MySQL
```

--------------------------------

### Start vLLM Server

Source: https://github.com/tencent/ksanallm/blob/main/benchmarks/README.md

Configures environment variables and launches the vLLM API server with specified model and hardware parameters.

```bash
export MODEL_PATH=${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/python/Llama-2-7b-hf
export CUDA_VISIBLE_DEVICES=xx

python -m vllm.entrypoints.api_server \
     --model $MODEL_PATH \
     --tokenizer $MODEL_PATH \
     --trust-remote-code \
     --max-model-len 1536 \
     --pipeline-parallel-size 1 \
     --tensor-parallel-size 1 \
     --gpu-memory-utilization 0.94 \
     --disable-log-requests \
     --port 8080
```

--------------------------------

### Start KsanaLLM Serving Server

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/openaiapi/readme_cn.md

Commands to launch the serving server with optional configurations for tool parsing, reasoning models, and chat templates.

```bash
#打开 server 文件夹
cd KsanaLLM/src/ksana_llm/python

python serving_server.py \
    --config_file ${GIT_PROJECT_REPO_ROOT}/examples/ksana_llm.yaml  \
    --host 0.0.0.0 \
    --port 8080

#如果需要解析工具调用，请根据模型添加：
--enable-auto-tool-choice  --tool-call-parser deepseek_v3 \ 
# 如果需要解析推理模型的推理内容，请根据模型添加：
--reasoning-parser qwen3 \
# 如果需要使用固定的 chat-template 请根据 chat-template添加：
--chat-template

# 以DeepSeek_R1为例：
python serving_server.py \
--config_file ${GIT_PROJECT_REPO_ROOT}/examples/deepseek_fp8_perf.yaml \
--port 8080 \
--enable-auto-tool-choice     --tool-call-parser deepseek_v3  \
--reasoning-parser  deepseek_r1 \
--chat-template /workspace/KsanaLLM/examples/chat_templates/tool_chat_template_deepseekr1.jinja
```

--------------------------------

### Service Startup on Master Node

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Starts the KsanaLLM service on the master node. Ensure the configuration file path is correct.

```bash
python serving_server.py \
--config_file ${GIT_PROJECT_REPO_ROOT}/examples/deepseek_fp8_perf.yaml \
--port service_port
```

--------------------------------

### Start KsanaLLM Serving Server

Source: https://github.com/tencent/ksanallm/blob/main/benchmarks/README.md

Launch the KsanaLLM serving server. Specify the configuration file and port. Change the config file to try other options.

```bash
cd ${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/python

export CUDA_VISIBLE_DEVICES=xx

# launch server
python serving_server.py \
    --config_file ../../../examples/ksana_llm2-7b.yaml \
    --port 8080

```

--------------------------------

### Setup Kernel Testing Environment

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/activation/CMakeLists.txt

Copies test scripts to build directories and defines the test target with necessary dependencies.

```cmake
file(GLOB_RECURSE ACTIVATION_TEST_SRCS *test.cu)
# copy python script for make test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/activation/activation_test.py ${CMAKE_CURRENT_BINARY_DIR})
# copy python script for standalone test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/activation/activation_test.py ${CMAKE_BINARY_DIR})
cc_test(llm_kernels_nvidia_kernel_activation_test SRCS ${ACTIVATION_TEST_SRCS} DEPS 
    llm_kernels_nvidia_utils
    llm_kernels_nvidia_kernel_activation)
```

--------------------------------

### Distribute KsanaLLM Wheel Build and Install

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Builds a distributable wheel package for KsanaLLM and installs it using pip. Includes options for customizing CMake arguments during the build process.

```bash
cd ${GIT_PROJECT_REPO_ROOT}

# for distribute wheel
python setup.py bdist_wheel

# or build with other cmake args
export CMAKE_ARGS="
  -DWITH_CUDA=ON
  -DWITH_ACL=OFF
" && python setup.py bdist_wheel

# install wheel
pip install dist/ksana_llm-0.1-*-linux_x86_64.whl

# check install success
pip show -f ksana_llm
python -c "import ksana_llm"
```

--------------------------------

### Perform Tool Calling

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/openaiapi/readme_cn.md

Example of sending a tool definition to the API to enable function calling capabilities.

```shell
curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
-d '{
  "model": "ksana-llm",
  "messages": [
    {
      "role": "user",
      "content": "What is the weather like in Boston today?"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"]
            }
          },
          "required": ["location"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}'
```

--------------------------------

### Request Chat Completions via OpenAI Python Client

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/openaiapi/readme_cn.md

Example of initializing the OpenAI client to communicate with the KsanaLLM service.

```python
from openai import OpenAI

# 配置客户端指向 KsanaLLM 服务
client = OpenAI(
    api_key="dummy-key",  # KsanaLLM 不需要 API key，但客户端需要这个参数
    base_url="http://localhost:8080/v1"
)

# 创建聊天完成请求
response = client.chat.completions.create(
    model="ksana-llm",  # 或使用您加载的具体模型名称
    messages=[
        {"role": "system", "content": "你是一个有帮助的助手。"},
        {"role": "user", "content": "你好，请介绍一下你自己。"}
    ],
    max_tokens=1000
)

print(response.choices[0].message.content)
```

--------------------------------

### Launch Service with Uvicorn

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/simple_router/README.md

Starts the KsanaLLM service using uvicorn. Specify host, port, worker count, and the configuration file path.

```bash
python ./src/ksana_llm/python/simple_router/main.py --host 0.0.0.0 --port 9080 --workers 8 --config /tmp/config.ini
```

--------------------------------

### Launch KsanaLLM Server via CLI

Source: https://context7.com/tencent/ksanallm/llms.txt

Start the inference server using a YAML configuration file. Additional flags enable specialized features like tool calling and reasoning parsers.

```bash
# Basic server launch
python serving_server.py \
    --config_file examples/ksana_llm2-7b.yaml \
    --port 8080

# With OpenAI API tool calling and reasoning support
python serving_server.py \
    --config_file examples/deepseek_fp8_perf.yaml \
    --port 8080 \
    --enable-auto-tool-choice \
    --tool-call-parser deepseek_v3 \
    --reasoning-parser deepseek_r1 \
    --chat-template openaiapi/chat_templates/tool_chat_template_deepseekr1.jinja
```

--------------------------------

### Download Hugging Face Model

Source: https://github.com/tencent/ksanallm/blob/main/benchmarks/README.md

Use this command to clone a Hugging Face model repository. Ensure git-lfs is installed.

```bash
cd ${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/python

# download huggingface model for example:
# Note: Make sure git-lfs is installed.
git clone https://huggingface.co/NousResearch/Llama-2-7b-hf

```

--------------------------------

### Build Docker Container for Nvidia GPU

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Builds a Docker image for Nvidia GPU usage. Ensure nvidia-docker is installed.

```bash
# need install nvidia-docker from https://github.com/NVIDIA/nvidia-container-toolkit
cd docker
docker build -f Dockerfile.gpu -t ksanallm-gpu .
docker run \
    -u root \
    -itd --privileged \
    --shm-size=50g \
    --network host \
    --cap-add=SYS_ADMIN \
    --cap-add=SYS_PTRACE \
    --gpus all \
    ksanallm-gpu bash

# goto KsanaLLM root directory 
pip install -r requirements.txt
```

--------------------------------

### Request Reasoning Content

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/openaiapi/readme_cn.md

Example of a request to retrieve the model's internal reasoning process.

```shell
curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ksana-llm",
    "messages": [
      {
        "role": "developer",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'
```

--------------------------------

### Setup NVIDIA Layernorm Tests

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/layernorm/CMakeLists.txt

Configures the build for layernorm tests, including copying necessary Python scripts for testing purposes. This setup is crucial for validating the layernorm kernel functionality.

```cmake
file(GLOB_RECURSE LAYERNORM_TEST_SRCS *test.cu)
# # copy python script for make test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/layernorm/layernorm_test.py ${CMAKE_CURRENT_BINARY_DIR})
# # copy python script for standalone test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/layernorm/layernorm_test.py ${CMAKE_BINARY_DIR})
cc_test(llm_kernels_nvidia_kernel_layernorm_test SRCS ${LAYERNORM_TEST_SRCS} DEPS 
    llm_kernels_nvidia_utils
    llm_kernels_nvidia_kernel_layernorm)
```

--------------------------------

### Build Docker Container for Huawei Ascend NPU

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Builds a Docker image for Huawei Ascend NPU usage. Requires specific driver and CANN installation. Only supports Ascend NPU + X86 CPU.

```bash
cd docker
docker build -f Dockerfile.npu -t ksanallm-npu .
docker run \
    -u root \
    -itd --privileged \
    --shm-size=50g \
    --network host \
    --cap-add=SYS_ADMIN \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp:unconfined $(find /dev/ -regex ".*/davinci$" | awk '{print " --device "$0}') \
    --device=/dev/devmm_svm \
    --device=/dev/hisi_hdc \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/local/sbin/:/usr/local/sbin/ \
    -v /var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf \
    -v /var/log/npu/slog/:/var/log/npu/slog \
    -v /var/log/npu/profiling/:/var/log/npu/profiling \
    -v /var/log/npu/dump/:/var/log/npu/dump \
    -v /var/log/npu/:/usr/slog \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    ksanallm-npu bash

# install Ascend-cann-toolkit, Ascend-cann-nnal from https://www.hiascend.com/document/detail/zh/canncommercial/80RC2/softwareinst/instg/instg_0000.html?Mode=PmIns&OS=Ubuntu&Software=cannToolKit
# download torch_npu-2.1.0.post6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl from https://www.hiascend.com/document/detail/zh/canncommercial/80RC2/softwareinst/instg/instg_0000.html?Mode=PmIns&OS=Ubuntu&Software=cannToolKit
pip3 install torch==2.1.0+cpu --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu-2.1.0.post6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pip install -r requirements.txt
```

--------------------------------

### Setup Fused MoE Test Environment

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/moe/fused_moe/CMakeLists.txt

Copies necessary Python scripts to the build directory and registers the C++ test target.

```cmake
# for test
file(GLOB_RECURSE fused_moe_test_SRCS
${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/moe/fused_moe/*test.cpp)
# copy python script for make test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/moe/fused_moe/fused_moe.py ${CMAKE_CURRENT_BINARY_DIR})
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/moe/fused_moe/fused_moe.py ${CMAKE_BINARY_DIR})
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/moe/common.py ${CMAKE_CURRENT_BINARY_DIR})
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/moe/common.py ${CMAKE_BINARY_DIR})
cc_test(fused_moe_config_find_test SRCS ${fused_moe_test_SRCS})
```

--------------------------------

### Single-Node 8-GPU Execution with INT4 Model

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Starts the KsanaLLM service on a single node with 8 GPUs, using an INT4 quantized model. Optional environment variables can further enhance performance with a slight accuracy trade-off. Model caching can also be applied.

```bash
# Optimal environment variable configuration
export ENABLE_COMPRESSED_KV=2
export SELECT_ALL_REDUCE_BY_SIZE=1
export PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync
# (Optional) Further Performance Enhancement with Slight Accuracy Degradation
export W4AFP8_MOE_BACKEND=1
# Service Startup
python serving_server.py \
--config_file ${GIT_PROJECT_REPO_ROOT}/examples/deepseek_int4_perf.yaml \
--port service_port
```

--------------------------------

### Basic Usage with --tune

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/moe/fused_moe/readme.md

Use the --tune flag to search for optimal num_warps and num_stages based on MoE parameters and generate a Triton kernel. This example sets various MoE and compute parameters.

```bash
python ./fused_moe.py \
    --output_dir ./ \
    --group_n 128 \
    --group_k 128 \
    --BLOCK_SIZE_M 64 \
    --BLOCK_SIZE_N 128 \
    --BLOCK_SIZE_K 128 \
    --GROUP_SIZE_M 32 \
    --MUL_ROUTED_WEIGHT "True" \
    --top_k 1 \
    --compute_type "FP16" \
    --use_fp8_w8a8 "True" \
    --use_int8_w8a16 "False" \
    --tune
```

--------------------------------

### ABI Compatibility Check

Source: https://github.com/tencent/ksanallm/blob/main/CMakeLists.txt

Queries the installed PyTorch environment to determine the required CXX11 ABI setting for compilation.

```cmake
set(USE_CXX11_ABI, "False")

# prepare ABI version
execute_process(COMMAND python "-c" "import torch;print(torch._C._GLIBCXX_USE_CXX11_ABI,end='');"
  RESULT_VARIABLE _PYTHON_SUCCESS
  OUTPUT_VARIABLE USE_CXX11_ABI)

if(NOT _PYTHON_SUCCESS EQUAL 0)
  message(FATAL_ERROR "run python -c \"import torch;print(torch._C._GLIBCXX_USE_CXX11_ABI,end='');\" failed.")
endif()

# set compiler flags
if("${USE_CXX11_ABI}" STREQUAL "True")
  add_compile_definitions(_GLIBCXX_USE_CXX11_ABI=1)
  message(STATUS "Compile with CXX11 ABI")
else()
  add_compile_definitions(_GLIBCXX_USE_CXX11_ABI=0)
  message(STATUS "Compile without CXX11 ABI")
endif()
```

--------------------------------

### Deep Tune Configuration Example

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/moe/fused_moe/readme.md

This JSON object represents an optimized configuration found by --deep_tune, specifying parameters like BLOCK_SIZE_M, num_warps, and num_stages for a given key (e.g., '128').

```json
{
    "128": {
        "BLOCK_SIZE_M": 64,
        "BLOCK_SIZE_N": 128,
        "BLOCK_SIZE_K": 128,
        "GROUP_SIZE_M": 1,
        "num_warps": 4,
        "num_stages": 3
    }
}
```

--------------------------------

### Setup Fused Add Norm Tests

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/fused_add_norm/CMakeLists.txt

Configures test execution by copying necessary Python scripts and defining the test target with required dependencies.

```cmake
file(GLOB_RECURSE FUSED_ADD_NORM_TEST_SRCS *test.cu)
# # copy python script for make test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/fused_add_norm/fused_add_norm_test.py ${CMAKE_CURRENT_BINARY_DIR})
# # copy python script for standalone test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/fused_add_norm/fused_add_norm_test.py ${CMAKE_BINARY_DIR})
cc_test(llm_kernels_nvidia_kernel_fused_add_norm_test SRCS ${FUSED_ADD_NORM_TEST_SRCS} DEPS 
    llm_kernels_nvidia_utils
    llm_kernels_nvidia_kernel_fused_add_norm)
```

--------------------------------

### Build Docker Container for Tencent Cloud Nvidia GPU Image

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Builds a Docker image using Tencent Cloud's OS image for Nvidia GPUs. Ensure nvidia-docker is installed.

```bash
# need install nvidia-docker from https://github.com/NVIDIA/nvidia-container-toolkit
cd docker
nvidia-docker build -f Dockerfile.tencentos4.gpu -t ksanallm-gpu .
nvidia-docker run \
    -u root \
    -itd --privileged \
    --shm-size=50g \
    --network host \
    --cap-add=SYS_ADMIN \
    --cap-add=SYS_PTRACE \
    ksanallm-gpu bash

# goto KsanaLLM root directory 
pip install -r requirements.txt
```

--------------------------------

### Advanced Usage with --deep_tune and --model_path

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/moe/fused_moe/readme.md

Enables deep tuning and reads model parameters from a specified path. Note that MUL_ROUTED_WEIGHT is ignored in this scenario, and deep_tune requires tune to be enabled. This example also sets top_k to 8 and simulates multi-GPU parallelism with tp_size.

```bash
python ./fused_moe.py \
    --output_dir ./ \
    --group_n 128 \
    --group_k 128 \
    --BLOCK_SIZE_M 64 \
    --BLOCK_SIZE_N 128 \
    --BLOCK_SIZE_K 128 \
    --GROUP_SIZE_M 32 \
    --MUL_ROUTED_WEIGHT "False" \
    --top_k 8 \
    --compute_type "FP16" \
    --use_fp8_w8a8 "True" \
    --use_int8_w8a16 "False" \
    --tune \
    --deep_tune \
    --model_path "/model/DeepSeek" \
    --tp_size 8 \
    --m 20 \
    --even_Ks True
```

--------------------------------

### Build NVIDIA Kernel Assembly Library

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/assemble_tokens_hidden/CMakeLists.txt

Configures and builds the 'llm_kernels_nvidia_kernel_assemble_tokens_hidden' static library. It finds all .cu files, excludes test files, and links against cublas, cudart, and cublasLt libraries. Ensure CUDA toolkit is installed and configured.

```cmake
file(GLOB_RECURSE ASSEMBLE_TOKENS_HIDDEN_SRCS *.cu)
list(FILTER ASSEMBLE_TOKENS_HIDDEN_SRCS EXCLUDE REGEX ".*test.cu")
add_library(llm_kernels_nvidia_kernel_assemble_tokens_hidden STATIC ${ASSEMBLE_TOKENS_HIDDEN_SRCS})
set_property(TARGET llm_kernels_nvidia_kernel_assemble_tokens_hidden PROPERTY POSITION_INDEPENDENT_CODE ON)
set_property(TARGET llm_kernels_nvidia_kernel_assemble_tokens_hidden PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON)
target_link_libraries(llm_kernels_nvidia_kernel_assemble_tokens_hidden PUBLIC -lcublas -lcudart -lcublasLt)
```

--------------------------------

### Get Python Include Directory

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/CMakeLists.txt

Executes a Python command to obtain the include directory for the Python installation. This is necessary for compiling C++ extensions that interface with Python.

```cmake
execute_process(COMMAND ${PYTHON_PATH} "-c" "from __future__ import print_function; from distutils import sysconfig;print(sysconfig.get_python_inc());"
  RESULT_VARIABLE _PYTHON_SUCCESS
  OUTPUT_VARIABLE PY_INCLUDE_DIR)

if(NOT _PYTHON_SUCCESS MATCHES 0)
  message(FATAL_ERROR "Python config Error.")
endif()
```

--------------------------------

### Determine PyTorch Installation Directory

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/CMakeLists.txt

Retrieves the installation directory of PyTorch using a Python command. This path is crucial for CMake to find PyTorch headers and libraries.

```cmake
execute_process(COMMAND ${PYTHON_PATH} "-c" "from __future__ import print_function; import os; import torch;print(os.path.dirname(torch.__file__),end='');"
  RESULT_VARIABLE _PYTHON_SUCCESS
  OUTPUT_VARIABLE TORCH_DIR)

if(NOT _PYTHON_SUCCESS MATCHES 0)
  message(FATAL_ERROR "Torch config Error.")
endif()

list(APPEND CMAKE_PREFIX_PATH ${TORCH_DIR})
```

--------------------------------

### GET /health

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/openaiapi/readme_cn.md

Checks the health status of the KsanaLLM service.

```APIDOC
## GET /health

### Description
Checks the service health status.

### Method
GET

### Endpoint
/health
```

--------------------------------

### Configure Build Directories and Includes

Source: https://github.com/tencent/ksanallm/blob/main/CMakeLists.txt

Sets output directories for binaries and libraries, and configures include paths for various hardware backends.

```cmake
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)

# set include headers
set(COMMON_HEADER_DIRS
  ${PROJECT_SOURCE_DIR}
  ${PROJECT_SOURCE_DIR}/src
)

# set headers
include_directories(
  ${COMMON_HEADER_DIRS}
  ${CUDA_INC_DIRS}
  ${ACL_INC_DIRS}
  ${TOPS_INC_DIRS}
)

# set linked libraries
link_directories(
  ${CUDA_LIB_DIRS}
  ${ACL_LIB_DIRS}
  ${TOPS_LIB_DIRS}
)
```

--------------------------------

### GET /models

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/python/openaiapi/readme_cn.md

Retrieves a list of available models or details for a specific model.

```APIDOC
## GET /models

### Description
Lists all available models.

### Method
GET

### Endpoint
/models

## GET /models/{model_id}

### Description
Retrieves detailed information for a specific model.

### Method
GET

### Endpoint
/models/{model_id}

### Parameters
#### Path Parameters
- **model_id** (string) - Required - The unique identifier of the model.
```

--------------------------------

### Build KsanaLLM for Huawei Ascend NPU

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Configure and build the project specifically for Huawei Ascend NPU hardware.

```bash
cmake -DWITH_TESTING=ON -DWITH_CUDA=OFF -DWITH_ACL=ON .. && make -j32
```

--------------------------------

### Enable Model Caching for Faster Startup

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Accelerates service startup by generating and utilizing a cached model. These environment variables must be set on all nodes for both cache generation and utilization.

```bash
export ENABLE_MODEL_CACHE=1
export MODEL_CACHE_PATH=/xxx_cache_model_dir/
```

--------------------------------

### Compile for NVIDIA H20

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Build the project specifically for NVIDIA H20 hardware.

```bash
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release ..  && make -j

cd ${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/python
ln -s ${GIT_PROJECT_REPO_ROOT}/build/lib .
```

--------------------------------

### Enable Model Caching for Faster Startup

Source: https://context7.com/tencent/ksanallm/llms.txt

Enable model caching by setting environment variables. The first startup generates the cache, while subsequent startups load from it, significantly speeding up initialization.

```bash
# Enable model caching (set on all nodes for distributed setup)
export ENABLE_MODEL_CACHE=1
export MODEL_CACHE_PATH=/data/model_cache/

# First startup: generates cache
python serving_server.py --config_file examples/deepseek_fp8_perf.yaml --port 8080

# Subsequent startups: loads from cache (faster)
python serving_server.py --config_file examples/deepseek_fp8_perf.yaml --port 8080
```

--------------------------------

### Set Minimum CMake Version

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/endpoints/wrapper/trpc/CMakeLists.txt

Specifies the minimum required version of CMake for this project. Ensure your CMake installation meets this requirement.

```cmake
cmake_minimum_required(VERSION 3.13)
```

--------------------------------

### Build KsanaLLM with Internal Libraries

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Enable internal Tencent libraries during the build process.

```bash
cmake -DSM=86 -DWITH_TESTING=ON -DWITH_INTERNAL_LIBRARIES=ON .. && make -j32
```

--------------------------------

### Define Test Dependencies and Main Source

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/models/CMakeLists.txt

Sets up dependencies and the main test source file when standalone testing is enabled. This is a prerequisite for defining individual test cases.

```cmake
if(WITH_STANDALONE_TEST)
  set(MODELS_TEST_DEPS models runtime data_hub)
  set(MODEL_TEST_MAIN ${PROJECT_SOURCE_DIR}/tests/test.cpp)

  if(WITH_CUDA)
    set(MODELS_TEST_LIBS ${TORCH_LIBRARIES} ${TORCH_PYTHON_LIBRARY})
  else()
    set(MODELS_TEST_LIBS "")
  endif()
endif()
```

--------------------------------

### Find PyTorch Package

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/CMakeLists.txt

Instructs CMake to find the PyTorch package, which is required for building the project. This command assumes PyTorch has been installed and configured correctly.

```cmake
find_package(Torch REQUIRED)
```

--------------------------------

### CMake Build Configuration

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/cpp/CMakeLists.txt

Defines the minimum CMake version, includes external dependencies, and adds the serving_forward_client executable.

```cmake
# Copyright 2024 Tencent Inc.  All rights reserved.
#
# ==============================================================================
cmake_minimum_required(VERSION 3.13)

include(external/httplib)
include(external/base64)
include(external/msgpack)

include_directories(
  ${PROJECT_SOURCE_DIR}/src
)

add_executable(serving_forward_client ${PROJECT_SOURCE_DIR}/src/ksana_llm/cpp/serving_forward_client.cpp)
```

--------------------------------

### Include Test Module

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/CMakeLists.txt

Includes the CMake test module, which likely provides testing utilities for the project. No specific setup is required beyond this line.

```cmake
include(test)
```

--------------------------------

### Performance Testing

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Run model performance benchmarks, optionally with profiling enabled.

```bash
cd ${GIT_PROJECT_REPO_ROOT}/build 
./bin/run_model_performance --runtime-config ${GIT_PROJECT_REPO_ROOT}/examples/llama7b/ksana_llm_tp.yaml --perf-config ${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/model_performance/test_config.csv 

# enable nsys when using Cuda
export ENABLE_PROFILE_EVENT=1 # enale profile event like NVTX on Cuda
nsys profile ./bin/run_model_performance --runtime-config ${GIT_PROJECT_REPO_ROOT}/examples/llama7b/ksana_llm_tp.yaml --perf-config ${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/model_performance/test_config.csv 
unset ENABLE_PROFILE_EVENT # after using nsys
```

--------------------------------

### Enable CUDA Support in CMake

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/CMakeLists.txt

Conditionally enables CUDA support and adds the NVIDIA subdirectory if WITH_CUDA is defined. Ensure CUDA is installed and configured for your system.

```cmake
if(WITH_CUDA)
  add_definitions(-DENABLE_FLASHINFER=ON)
  add_subdirectory(nvidia)
endif()
```

--------------------------------

### Configure New Qwen Config Parser Test (CUDA)

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/models/CMakeLists.txt

Sets up the `new_qwen_config_parser_test` using `cpp_test` for CUDA builds. This test focuses on the configuration parsing for Qwen models.

```cmake
cpp_test(new_qwen_config_parser_test SRCS ${PROJECT_SOURCE_DIR}/src/ksana_llm/models/qwen/new_qwen_config_parser_test.cpp ${MODEL_TEST_MAIN} DEPS ${MODELS_TEST_DEPS} LIBS ${MODELS_TEST_LIBS})
```

--------------------------------

### Configure Utility Library Build

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/utils/CMakeLists.txt

Defines the build process for the utils library, including source file discovery and conditional linking for CUDA, ACL, and TOPS backends.

```cmake
cmake_minimum_required(VERSION 3.13)

file(GLOB_RECURSE utils_SRCS
  ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/*.cpp)
list(FILTER utils_SRCS EXCLUDE REGEX ".*test.cpp")
list(FILTER utils_SRCS EXCLUDE REGEX ".*/nvidia/.*.cpp")
list(FILTER utils_SRCS EXCLUDE REGEX ".*/ascend/.*.cpp")
list(FILTER utils_SRCS EXCLUDE REGEX ".*/zixiao/.*.cpp")
message(STATUS "utils_SRCS: ${utils_SRCS}")

set(utils_nvidia_SRCS "")
set(utils_nvidia_LIBS "")

if(WITH_CUDA)
  file(GLOB_RECURSE utils_nvidia_SRCS
    ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/nvidia/*.cpp
    ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/nvidia/*.cu)
  list(APPEND utils_nvidia_LIBS ${NCCL_LIBRARIES} -lcudart -lcublas -lcublasLt -lcuda -lnvrtc -lcudadevrt
    llm_kernels_nvidia_utils)
endif()

set(utils_ascend_SRCS "")
set(utils_ascend_LIBS "")

if(WITH_ACL)
  file(GLOB_RECURSE utils_ascend_SRCS ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/ascend/*.cpp)
  list(APPEND utils_ascend_LIBS ${ACL_SHARED_LIBS} hccl llm_kernels_ascend_utils_common)
endif()

set(utils_zixiao_SRCS "")
set(utils_zixiao_LIBS "")

if(WITH_TOPS)
  file(GLOB_RECURSE utils_zixiao_SRCS ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/zixiao/*.cpp)
  list(APPEND utils_zixiao_LIBS ${ZIXIAO_SHARED_LIBS} topsrt llm_kernels_zixiao_utils_common)
endif()

add_library(utils STATIC ${utils_SRCS} ${utils_nvidia_SRCS} ${utils_ascend_SRCS} ${utils_zixiao_SRCS})
add_dependencies(utils fmt gflags)
target_link_libraries(utils PUBLIC
  -lpthread -ldl gflags loguru yaml-cpp profiler ${utils_nvidia_LIBS} ${utils_ascend_LIBS} ${utils_zixiao_LIBS}
  absl::base absl::strings re2 Boost::dll ${TORCH_LIBRARIES} ${PYTHON_LIBRARY} xgrammar
)

if(WITH_CUDA)
  set_target_properties(utils PROPERTIES CUDA_SEPARABLE_COMPILATION ON CUDA_RESOLVE_DEVICE_SYMBOLS ON
    POSITION_INDEPENDENT_CODE ON)
  target_include_directories(utils PRIVATE ${TORCH_INCLUDE_DIRS})
  if(ENABLE_DEEPSEEK_DEEPGEMM_FLAG)
    target_include_directories(utils PRIVATE ${DEEPGEMM_INCLUDE_DIRS})
    target_compile_definitions(utils PRIVATE
      DEEPGEMM_LIBRARY_ROOT_PATH="${DEEPGEMM_LIBRARY_ROOT_PATH}"
      DEEPGEMM_CUDA_HOME="${CUDA_TOOLKIT_ROOT_DIR}")
  endif()
endif()

# for test
if(WITH_STANDALONE_TEST)
  file(GLOB_RECURSE utils_test_SRCS
    ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/*test.cpp
    ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/config/*test.cpp
    ${PROJECT_SOURCE_DIR}/tests/test.cpp)
  list(FILTER utils_test_SRCS EXCLUDE REGEX ".*environment_test.cpp")
  message(STATUS "utils_test_SRCS: ${utils_test_SRCS}")
  cpp_test(utils_test SRCS ${utils_test_SRCS} DEPS runtime utils)
  cpp_test(environment_test SRCS ${PROJECT_SOURCE_DIR}/src/ksana_llm/utils/environment_test.cpp DEPS runtime utils)
endif()
```

--------------------------------

### Check PyTorch Version

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/CMakeLists.txt

Executes a Python command to get the PyTorch version and checks if it meets the minimum requirement of 1.5.0 for TorchScript mode. If not, it halts the build.

```cmake
execute_process(COMMAND ${PYTHON_PATH} "-c" "from __future__ import print_function; import torch; print(torch.__version__,end='');"
  RESULT_VARIABLE _PYTHON_SUCCESS
  OUTPUT_VARIABLE TORCH_VERSION)

if(TORCH_VERSION VERSION_LESS "1.5.0")
  message(FATAL_ERROR "PyTorch >= 1.5.0 is needed for TorchScript mode.")
endif()
```

--------------------------------

### Enable ACL Support in CMake

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/CMakeLists.txt

Conditionally adds the Ascend (ACL) subdirectory and sets Ascend-specific libraries if WITH_ACL is defined. Requires Ascend drivers and libraries to be installed.

```cmake
if(WITH_ACL)
  add_subdirectory(ascend)
  set(llm_kernels_ascend_libs ${llm_kernels_ascend_libs} PARENT_SCOPE)
endif()
```

--------------------------------

### Deploy Distributed Multi-Node Cluster

Source: https://context7.com/tencent/ksanallm/llms.txt

Configures environment variables and launches the serving server across master and worker nodes.

```bash
# Master node (IP1)
export WORLD_SIZE=2
export NODE_RANK=0
export MASTER_HOST=192.168.1.100
export MASTER_PORT=29500
export ENABLE_COMPRESSED_KV=2
export USE_TCP_DATA_CHANNEL=1

python serving_server.py \
    --config_file examples/deepseek_fp8_perf.yaml \
    --port 8080

# Worker node (IP2)
export WORLD_SIZE=2
export NODE_RANK=1
export MASTER_HOST=192.168.1.100
export MASTER_PORT=29500
export ENABLE_COMPRESSED_KV=2
export USE_TCP_DATA_CHANNEL=1

python serving_server.py \
    --config_file examples/deepseek_fp8_perf.yaml \
    --port 8080
```

--------------------------------

### Health Check Endpoint

Source: https://context7.com/tencent/ksanallm/llms.txt

Monitor service health through the built-in health check endpoint. This function sends a GET request to the /health endpoint and checks for a 200 status code.

```python
import requests

def check_health(port=8080):
    try:
        response = requests.get(f"http://localhost:{port}/health", timeout=5)
        if response.status_code == 200:
            result = response.json()
            print(f"Service status: {result['status']}")
            return True
    except requests.exceptions.RequestException as e:
        print(f"Health check failed: {e}")
    return False
```

--------------------------------

### Create Connector Static Library

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/connector/CMakeLists.txt

Creates a static library named 'connector' using the discovered source files. It also adds dependencies on cppzmq and libzmq, and configures include directories.

```cmake
# Create the connector library
add_library(connector STATIC ${CONNECTOR_SRCS})

add_dependencies(connector cppzmq libzmq)

target_include_directories(connector
    PUBLIC
        ${PROJECT_SOURCE_DIR}/src
        ${CMAKE_CURRENT_SOURCE_DIR}
        ${LIBZMQ_INCLUDE_DIR}
        ${CPPZMQ_INCLUDE_DIR}
)

target_link_libraries(connector PRIVATE utils cppzmq libzmq TBB::tbb ${CURL_LIBRARIES})
```

--------------------------------

### Platform Support Configuration

Source: https://github.com/tencent/ksanallm/blob/main/CMakeLists.txt

Configures project languages and paths based on the selected hardware backend (CUDA, ACL, or TOPS).

```cmake
if(WITH_CUDA)
  # support NVIDIA GPU
  project(ksana_llm LANGUAGES CXX CUDA)
elseif(WITH_ACL)
  # support Huawei NPU
  set(ASCEND_PRODUCT_TYPE "ascend910")
  set(ASCEND_PLATFORM_NAME "Ascend910B2C")
  set(ASCEND_CORE_TYPE "AiCore")
  set(ASCEND_RUN_MODE "ONBOARD")
  set(ASCEND_INSTALL_PATH "/usr/local/Ascend/ascend-toolkit/latest")
  set(CCE_CMAKE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake/module)
  list(APPEND CMAKE_MODULE_PATH ${CCE_CMAKE_PATH})

  # NOTE(karlluo): languages cce need configured cmake module before
  project(ksana_llm LANGUAGES CCE CXX)

elseif(WITH_TOPS)
  # support Tencent ZiXiao
  set(ARCH "gcu300")
  set(CMAKE_TOPS_COMPILER_TOOLKIT_ROOT "/opt/tops")
  list(APPEND CMAKE_MODULE_PATH "/opt/tops/cmake_modules")
  include(FindTOPS)
  project(ksana_llm LANGUAGES CXX TOPS)
else()
  message(FATAL_ERROR "Support platform is not selected. Select NVIDIA GPU with -DWITH_CUDA=ON, Huawei NPU with -DWITH_ACL=ON and Tencent ZiXiao with -DWITH_TOPS=ON")
endif()
```

--------------------------------

### Configure Connector Test Executable

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/connector/CMakeLists.txt

If testing is enabled (WITH_TESTING), this section discovers test source files, creates a test executable, sets its output directory, and links it with the connector library and testing frameworks.

```cmake
# Add testing if enabled
if(WITH_TESTING)
    file(GLOB_RECURSE TEST_SRCS ${PROJECT_SOURCE_DIR}/src/ksana_llm/connector/*test.cpp)
    add_executable(connector_test ${TEST_SRCS})
    set_target_properties(connector_test PROPERTIES
        RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
    )
    target_compile_options(connector_test PRIVATE -fno-access-control -g -O0)

    target_link_libraries(connector_test PRIVATE connector gtest runtime gtest_main gmock ${CURL_LIBRARIES} TBB::tbb)

    if(WITH_CUDA)
        target_link_libraries(connector_test PRIVATE cuda )
    endif()

    if(WITH_INTERNAL_LIBRARIES)
        target_link_libraries(connector_test PRIVATE polaris-cpp::polaris_api polaris-cpp::polaris_cl5_api)
    endif()
    add_test(NAME connector_test COMMAND connector_test)
endif()
```

--------------------------------

### Configure Expert Parameter Weight Test

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/models/CMakeLists.txt

Sets up the `expert_para_weight_test` using `cpp_test`. It includes specific source files and dependencies, and also lists `data_hub` as a dependency.

```cmake
cpp_test(expert_para_weight_test SRCS ${PROJECT_SOURCE_DIR}/src/ksana_llm/models/common_moe/common_moe_weight_test.cpp ${MODEL_TEST_MAIN} DEPS ${MODELS_TEST_DEPS}  LIBS ${MODELS_TEST_LIBS} data_hub)
```

--------------------------------

### Configure Ksanallm Plugins Library

Source: https://github.com/tencent/ksanallm/blob/main/src/ksana_llm/plugins/CMakeLists.txt

Defines the build process for plugin source files, excluding any files ending in test.cpp.

```cmake
cmake_minimum_required(VERSION 3.13)

file(GLOB_RECURSE plugins_SRCS
	${PROJECT_SOURCE_DIR}/src/ksana_llm/plugins/*.cpp)
list(FILTER plugins_SRCS EXCLUDE REGEX ".*test.cpp")
message(STATUS "plugins_SRCS: ${plugins_SRCS}")

add_library(plugins STATIC ${plugins_SRCS})
```

--------------------------------

### Configure NVIDIA Kernel Library and Tests

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/adjust_mem/CMakeLists.txt

Defines the static library for NVIDIA kernels and sets up the necessary test infrastructure using CMake.

```cmake
# Copyright 2025 Tencent Inc.  All rights reserved.

# set kernels target
file(GLOB_RECURSE CONCAT_SRCS *.cu)
list(FILTER CONCAT_SRCS EXCLUDE REGEX ".*test.cu")
add_library(llm_kernels_nvidia_kernel_adjust_mem STATIC ${CONCAT_SRCS})
set_property(TARGET llm_kernels_nvidia_kernel_adjust_mem PROPERTY POSITION_INDEPENDENT_CODE ON)
set_property(TARGET llm_kernels_nvidia_kernel_adjust_mem PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON)
target_link_libraries(llm_kernels_nvidia_kernel_adjust_mem PUBLIC -lcublas -lcudart -lcublasLt)

# for test
file(GLOB_RECURSE ADJUST_TEST_SRCS *test.cu)
# copy python script for make test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/adjust_mem/adjust_mem_test.py ${CMAKE_CURRENT_BINARY_DIR})
# copy python script for standalone test
execute_process(COMMAND cp -r ${PROJECT_SOURCE_DIR}/csrc/kernels/nvidia/adjust_mem/adjust_mem_test.py ${CMAKE_BINARY_DIR})
cc_test(llm_kernels_nvidia_kernel_adjust_mem_test SRCS ${ADJUST_TEST_SRCS} DEPS 
    llm_kernels_nvidia_utils
    llm_kernels_nvidia_kernel_adjust_mem)
```

--------------------------------

### Run Inference Clients

Source: https://github.com/tencent/ksanallm/blob/main/README.md

Execute client scripts to test inference capabilities.

```bash
# open another session
cd ${GIT_PROJECT_REPO_ROOT}/src/ksana_llm/python
python serving_generate_client.py --port 8080
```

```bash
python serving_forward_client.py --port 8080
```

--------------------------------

### Build NVIDIA Kernel Expansion Library

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/expand/CMakeLists.txt

Compiles all .cu files (excluding test files) into a static library for NVIDIA kernel expansion. Requires CUDA toolkit and specific libraries.

```cmake
# set kernels target
file(GLOB_RECURSE EXPAND_SRCS *.cu)
list(FILTER EXPAND_SRCS EXCLUDE REGEX ".*test.cu")
add_library(llm_kernels_nvidia_kernel_expand STATIC ${EXPAND_SRCS})
set_property(TARGET llm_kernels_nvidia_kernel_expand PROPERTY POSITION_INDEPENDENT_CODE ON)
set_property(TARGET llm_kernels_nvidia_kernel_expand PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON)
target_link_libraries(llm_kernels_nvidia_kernel_expand PUBLIC -lcublas -lcudart -lcublasLt
                      llm_kernels_nvidia_utils)
```

--------------------------------

### CMake Build Configuration for atb_plugin_operations

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/ascend/atb_plugin_operations/CMakeLists.txt

Defines the minimum CMake version, collects source files, and creates a static library with linked dependencies.

```cmake
cmake_minimum_required(VERSION 3.14)

file(GLOB_RECURSE atb_plugin_operations_SRCS ./*.cc)

message(STATUS "atb_plugin_operations_SRCS: ${atb_plugin_operations_SRCS}")

add_library(atb_plugin_operations STATIC ${atb_plugin_operations_SRCS})
target_link_libraries(atb_plugin_operations PRIVATE ${ACL_SHARED_LIBS} ${ACL_SHARED_LIBS})
```

--------------------------------

### Build NVIDIA Kernel Library

Source: https://github.com/tencent/ksanallm/blob/main/3rdparty/LLM_kernels/csrc/kernels/nvidia/logits_topk/CMakeLists.txt

Configures and builds the 'llm_kernels_nvidia_kernel_logits_topk' static library from CUDA source files. It excludes test files and links against necessary NVIDIA libraries.

```cmake
file(GLOB_RECURSE LOGITS_TOPK_SRCS *.cu)
list(FILTER LOGITS_TOPK_SRCS EXCLUDE REGEX ".*test.cu")
add_library(llm_kernels_nvidia_kernel_logits_topk STATIC ${LOGITS_TOPK_SRCS})
set_property(TARGET llm_kernels_nvidia_kernel_logits_topk PROPERTY POSITION_INDEPENDENT_CODE ON)
set_property(TARGET llm_kernels_nvidia_kernel_logits_topk PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON)
target_link_libraries(llm_kernels_nvidia_kernel_logits_topk PUBLIC -lcublas -lcudart -lcublasLt)
```

--------------------------------

### Run Throughput Benchmark

Source: https://github.com/tencent/ksanallm/blob/main/benchmarks/README.md

Executes the throughput benchmark script against a running vLLM server and redirects output to a log file.

```bash
python benchmark_throughput.py --port 8080  --input_csv benchmark_input.csv  \
    --model_type llama \
    --tokenizer_path $MODEL_PATH  \
    --backend vllm \
    --perf_csv vllm_perf.csv > vllm_stdout.txt 2>&1
```