### Install headers with CMake Source: https://github.com/ascend/mssanitizer/blob/master/csrc/include/CMakeLists.txt Collects all header files in the current directory and installs them to the include directory under the installation prefix. ```cmake file(GLOB HEADER_FILES "*.h") install(FILES ${HEADER_FILES} DESTINATION ${CMAKE_INSTALL_PREFIX}/include ) ``` -------------------------------- ### Install bash completion script Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md Installs the msSanitizer bash completion script to the user-level bash completion directory so that auto-completion activates automatically on login. ```shell # 保证 bash 补全目录已创建 mkdir -p ~/.local/share/bash-completion/completions # 拷贝 mssanitizer 自动补全脚本 cp ${ASCEND_HOME_PATH}/tools/mssanitizer/bin/mssanitizer-bash-completion ~/.local/share/bash-completion/completions/mssanitizer ``` -------------------------------- ### mssanitizer launch log output Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_synccheck/README.md Example output from mssanitizer when it successfully launches the operator. It shows the log file path and the start of the synccheck sanitizer on the kernel. ```text [mssanitizer] logging to file: ./mindstudio_sanitizer_log/mssanitizer_XXX.log [mssanitizer] Start synccheck sanitizer on kernel "copy_demo(unsigned char*, unsigned char*, unsigned char*)" ``` -------------------------------- ### initcheck output: uninitialized read error Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/quick_start/mssanitizer_quick_start.md Example output from the initcheck tool showing an uninitialized read of size 256. ```text ====== ERROR: uninitialized read of size 256 ====== at 0x400 on UB in AddCustom_ab1b6750d7f510985325b603cb06dc8b_0 ====== in block aiv(0-7) on device 3 ====== code in pc current 0x1e34 (serialNo:241) ====== #0 /usr/local/Ascend/cann-8.5.0/aarch64-linux/asc/impl/basic_api/dav_c220/kernel_operator_data_copy_impl.h:124:9 ====== #1 /usr/local/Ascend/cann-8.5.0/aarch64-linux/asc/impl/basic_api/kernel_operator_data_copy_intf_impl.h:265:9 ====== #2 /usr/local/Ascend/cann-8.5.0/aarch64-linux/asc/impl/basic_api/kernel_operator_data_copy_intf_impl.h:736:5 ====== #3 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:126:9 ====== #4 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:63:13 ====== #5 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:167:8 ``` -------------------------------- ### Build demo with CMake and mssanitizer Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_memcheck_advanced/README.md Build the demo with CMake. The first block uses the default mssanitizer location; the second explicitly sets the include directory and hook library when the tool is installed elsewhere. The build includes -g --cce-enable-sanitizer and produces build/demo. ```bash source /usr/local/Ascend/ascend-toolkit/set_env.sh cmake -S . -B build -DCMAKE_ASC_ARCHITECTURES=dav-2201 cmake --build build -j ``` ```bash cmake -S . -B build \ -DCMAKE_ASC_ARCHITECTURES=dav-2201 \ -DMSSANITIZER_ACL_INCLUDE_DIR=/path/to/mssanitizer/include/acl \ -DMSSANITIZER_ACL_HOOK_LIBRARY=/path/to/libascend_acl_hook.so ``` -------------------------------- ### Bash completion example output Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md Example output of bash auto-completion: pressing `` after `mssanitizer --` lists all options, and after `--log-level` lists the valid values (error, info, warn). ```shell $ mssanitizer -- --block-id --check-cann-heap --check-unused-memory --full-backtrace --kernel-name --log-file --max-debuglog-size --version --cache-size --check-device-heap --demangle --help --leak-check --log-level --tool $ mssanitizer --log-level error info warn ``` -------------------------------- ### PyTorchInvocation project directory structure Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md Directory structure of the PyTorch invocation project, showing the files needed to run the sample and start the tool. ```text PytorchInvocation ├── op_plugin_patch ├── run_op_plugin.sh // 执行样例时需要使用 └── test_ops_custom.py // 启动工具时需要使用 ``` -------------------------------- ### Run mssanitizer racecheck on demo Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck/README.md Launches the operator with the mssanitizer tool in racecheck mode. On success, it logs to a file and starts the racecheck sanitizer on the kernel 'raw_hazard_demo'. ```bash mssanitizer -t racecheck ./demo ``` -------------------------------- ### 执行卸载 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 执行卸载操作。{tools_name} 配置为需卸载的工具名称,可通过 python ms_install.py help 命令查询。 ```bash python ms_install.py uninstall {tools_name} ``` -------------------------------- ### main.cpp 包含 mstx 头文件 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/api_reference/mssanitizer_api_reference.md 在 main.cpp 中引入 mstx/ms_tools_ext.h 头文件,头文件地址为 ${INSTALL_DIR}/include/mstx。此步骤与 CMakeLists.txt 配置共同完成 mstx 接口的启用。 ```c ... #include "mstx/ms_tools_ext.h" ... ``` -------------------------------- ### 编译命令(默认路径) Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 进入样例目录后,设置环境变量并执行 cmake 配置与构建。生成的可执行文件为 build/demo。 ```bash source /usr/local/Ascend/ascend-toolkit/set_env.sh cmake -S . -B build -DCMAKE_ASC_ARCHITECTURES=dav-2201 cmake --build build -j ``` -------------------------------- ### Memory Anomaly Report Example Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md Example output from msSanitizer memcheck showing an illegal read of size 368 at sample.py:18:45, matching the constructed fault scenario. ```text $ mssanitizer -t memcheck -- python sample.py [mssanitizer] logging to file: ./mindstudio_sanitizer_log/mssanitizer_20250522093805_922880.log Failed ====== ERROR: illegal read of size 368 ====== at 0x12c0c0053190 on GM in triton_add ====== in block aiv(1) on device 0 ====== code in pc current 0x1b0 (serialNo:524) ====== #0 sample.py:18:45 ``` -------------------------------- ### Run sample and view build/execution output Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md Run the sample script and observe the build and execution output, including the successful compilation and execution of the operator and the final 'test pass' message. ```text bash run_op_plugin.sh -- CMAKE_CCE_COMPILER: ${INSTALL_DIR}/toolkit/tools/ccec_compiler/bin/ccec -- CMAKE_CURRENT_LIST_DIR: ${INSTALL_DIR}/AddKernelInvocation/cmake/Modules -- ASCEND_PRODUCT_TYPE: Ascendxxxyy -- ASCEND_CORE_TYPE: VectorCore -- ASCEND_INSTALL_PATH: /usr/local/Ascend/cann -- The CXX compiler identification is GNU 10.3.1 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: ${INSTALL_DIR}/AddKernelInvocation/build Scanning dependencies of target add_npu [ 33%] Building CCE object cmake/npu/CMakeFiles/add_npu.dir/__/__/add_custom.cpp.o [ 66%] Building CCE object cmake/npu/CMakeFiles/add_npu.dir/__/__/main.cpp.o [100%] Linking CCE executable ../../../add_npu [100%] Built target add_npu ${INSTALL_DIR}/AddKernelInvocation INFO: compile op on ONBOARD succeed! INFO: execute op on ONBOARD succeed! test pass ``` -------------------------------- ### CMakeLists.txt 配置:链接 libms_tools_ext.so 库 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/api_reference/mssanitizer_api_reference.md 在 AclNNInvocation 示例的 CMakeLists.txt 中新增库文件 libms_tools_ext.so,地址为 ${INSTALL_DIR}/lib64/libms_tools_ext.so。此配置用于使能 mstx 接口。 ```cmake # Header path include_directories( ... ${CUST_PKG_PATH}/include ) ... target_link_libraries( ... dl ) ``` -------------------------------- ### Directory structure of sample_initcheck Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_initcheck/README.md Shows the directory layout of the sample project, including the build file, the Ascend C implementation, and the README. ```text ├── sample_initcheck/ │ ├── CMakeLists.txt // 编译工程文件 │ ├── initcheck.asc // Ascend C 样例实现(含未初始化读取注入) │ └── README.md // 本说明文件 ``` -------------------------------- ### Illegal read error report example Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md Example of an illegal read error report from memory detection. It shows the error type and size, the memory address and kernel name, the block index, the PC and serial number, and the call stack with file, line, and column. The call stack is only present when the -g compile option is added. ```text ====== ERROR: illegal read of size 224 // 异常的基本信息,包含非法读写的类型以及被非法访问的字节数,非法读写包括read(非法读取)和write(非法写入) ====== at 0x12c0c0015000 on GM in add_custom_kernel // 异常发生的内存位置信息,包含发生的核函数名、地址空间与内存地址,此处的内存地址指一次内存访问中的首地址 ====== in block aiv(0) on device 0 // 异常代码对应Vector核的block索引 ====== code in pc current 0x77c (serialNo:10) // 当前异常发生的pc指针和调用api行为的序列号 ====== #0 ${ASCEND_HOME_PATH}/compiler/tikcpp/tikcfw/impl/dav_c220/kernel_operator_data_copy_impl.h:58:9 // 以下为异常发生代码的调用栈,包含文件名、行号和列号 ====== #1 ${ASCEND_HOME_PATH}/compiler/tikcpp/tikcfw/inner_interface/inner_kernel_operator_data_copy_intf.cppm:58:9 ====== #2 ${ASCEND_HOME_PATH}/compiler/tikcpp/tikcfw/inner_interface/inner_kernel_operator_data_copy_intf.cppm:443:5 ====== #3 illegal_read_and_write/add_custom.cpp:18:5 ``` -------------------------------- ### 模板库场景:添加编译选项 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/compile_option_config.md 在模板库场景中,修改 examples/CMakeLists.txt 文件,新增 -g 和 --cce-enable-sanitizer 编译选项以启用全量检测。 ```cmake set(BISHENG_COMPILER_OPTIONS -g --cce-enable-sanitizer) ``` -------------------------------- ### 自动复验脚本(默认日志目录) Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 运行 test_sample.sh 脚本,重新编译样例并依次执行三个场景,校验竞争类型、block、Device、跨卡汇总结论以及程序是否异常退出。日志默认保存在 test_output/。 ```bash bash test_sample.sh ``` -------------------------------- ### racecheck output: WAR hazard error Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/quick_start/mssanitizer_quick_start.md Example output from the racecheck tool showing a potential WAR hazard detected at UB. ```text ====== ERROR: Potential WAR hazard detected at UB in AddCustom_ab1b6750d7f510985325b603cb06dc8b_0 on device 0: ====== PIPE_MTE3 Read at WAR()+0x400 in block 0 (aiv) on device 0 at pc current 0x1e28 (serialNo:31) ====== #0 /usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux/tikcpp/tikcfw/impl/dav_c220/kernel_operator_data_copy_impl.h:124:9 ====== #1 /usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux/tikcpp/tikcfw/impl/kernel_operator_data_copy_intf_impl.h:204:9 ====== #2 /usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux/tikcpp/tikcfw/impl/kernel_operator_data_copy_intf_impl.h:573:5 ====== #3 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:128:10 ====== #4 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:63:14 ====== #5 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:169:9 ``` -------------------------------- ### Step-by-step build commands Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/development_guide/develop_guide.md Step-by-step build process: download dependencies, compile locally, and clean the build directory for a full rebuild. Output is generated in the artifacts directory. ```bash python3 download_dependencies.py ``` ```bash python3 build.py local ``` ```bash rm -rf build ``` -------------------------------- ### memcheck output: out of bounds warning Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/quick_start/mssanitizer_quick_start.md Example output from the memcheck tool showing an out-of-bounds write of size 256 on GM memory. ```text ====== WARNING: out of bounds of size 256 ====== at 0x12c0c0026000 on GM when writing data in AddCustom_ab1b6750d7f510985325b603cb06dc8b_0 ====== in block aiv(1) on device 0 ====== code in pc current 0x1e28 (serialNo:87) ====== #0 /usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux/tikcpp/tikcfw/impl/dav_c220/kernel_operator_data_copy_impl.h:124:9 ====== #1 /usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux/tikcpp/tikcfw/impl/kernel_operator_data_copy_intf_impl.h:204:9 ====== #2 /usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux/tikcpp/tikcfw/impl/kernel_operator_data_copy_intf_impl.h:573:5 ====== #3 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:128:10 ====== #4 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:63:14 ====== #5 /root/ot_demo/workspace/src/AddCustom/op_kernel/add_custom.cpp:169:9 ``` -------------------------------- ### 目录结构 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_memcheck/README.md 展示了 sample_memcheck 样例的目录结构,包含 CMakeLists.txt、memcheck.asc 和 README.md 文件。 ```text ├── sample_memcheck/ │ ├── CMakeLists.txt // 编译工程文件 │ ├── memcheck.asc // Ascend C 样例实现(含越界写入注入) │ └── README.md // 本说明文件 ``` -------------------------------- ### Activate bash completion in current session Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md Activates bash auto-completion for msSanitizer in the current shell session. Requires the `bash-completion` package to be installed. ```shell source ${ASCEND_HOME_PATH}/tools/mssanitizer/bin/mssanitizer-bash-completion ``` -------------------------------- ### 编译算子命令 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_memcheck/README.md 在 sample_memcheck/ 目录下执行,用于编译生成算子二进制文件 demo。 ```bash mkdir -p build && cd build cmake -DCMAKE_ASC_ARCHITECTURES=dav-2201 .. make -j96 ``` -------------------------------- ### Run mssanitizer in synccheck mode Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_synccheck/README.md Launches the operator with mssanitizer in synccheck mode. The tool will output logs indicating the start of the synccheck sanitizer on the kernel. ```bash mssanitizer -t synccheck ./demo ``` -------------------------------- ### 内核调用符场景:修改 CMakeCCEInformation.cmake Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/compile_option_config.md 在内核调用符场景中,编辑 cmake/Modules/CMakeCCEInformation.cmake 文件,增加 ${_CMAKE_COMPILE_AS_CCE_FLAG} 链接选项。 ```cmake if(NOT CMAKE_CCE_LINK_EXECUTABLE) set(CMAKE_CCE_LINK_EXECUTABLE " ${CMAKE_LIBRARY_CREATE_CCE_FLAGS} ${_CMAKE_COMPILE_AS_CCE_FLAG} -o ${__IMPLICIT_LINKS}") endif() ``` -------------------------------- ### 使用cmake ../cmake构建安装包 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/support/mssanitizer_faq.md 源码编译安装时,必须使用cmake ../cmake而非cmake ..,否则不会生成.run安装包。 ```shell mkdir build cd build cmake ../cmake make -j$(nproc) ``` -------------------------------- ### CMake build configuration for mssanitizer Source: https://github.com/ascend/mssanitizer/blob/master/csrc/CMakeLists.txt CMake configuration for the mssanitizer project. It sets include directories, compiles with -Werror, builds the shared library ascend_san and the executable mssanitizer, and installs binaries and headers. ```cmake get_filename_component(CSRC_INC_DIR . ABSOLUTE) get_filename_component(FRAMEWORK_INC_DIR ./core/framework ABSOLUTE) get_filename_component(HOOK_INC_DIR ./hooks ABSOLUTE) # 强制拦截业务代码中的所有编译告警 add_compile_options("-Werror") add_library(ascend_san SHARED) add_subdirectory(address_sanitizer) add_subdirectory(race_sanitizer) add_subdirectory(sync_sanitizer) add_subdirectory(register_sanitizer) add_subdirectory(core) add_subdirectory(hooks) add_subdirectory(plugin) add_subdirectory(include) target_include_directories( ascend_san PUBLIC ${CMAKE_CURRENT_SOURCE_DIR} ${SECUREC_INC_DIR} ${JSON_INC} ${CSRC_INC_DIR} ${SANITIZER_INC_DIR} ) target_link_libraries(ascend_san c_sec pthread stdc++fs dl) target_link_options(ascend_san PRIVATE "-Wl,-z,defs") add_executable(mssanitizer ${CMAKE_CURRENT_SOURCE_DIR}/main.cpp) target_link_options(mssanitizer PRIVATE "-pie") target_link_libraries(mssanitizer ascend_san) set_target_properties(mssanitizer PROPERTIES OUTPUT_NAME "mssanitizer.bin") set_target_properties(ascend_san PROPERTIES VERSION ${PROJECT_VERSION}) install(TARGETS mssanitizer ascend_san ascend_hal_hook ascend_acl_hook RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX}/bin LIBRARY DESTINATION ${CMAKE_INSTALL_PREFIX}/lib64 ) install(DIRECTORY ${HOOK_INC_DIR}/api/acl DESTINATION ${CMAKE_INSTALL_PREFIX}/include FILES_MATCHING PATTERN "*.h" ) install(FILES ${HOOK_INC_DIR}/api/sanitizer.h DESTINATION ${CMAKE_INSTALL_PREFIX}/include ) ``` -------------------------------- ### Run misaligned access memcheck test Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_memcheck_advanced/README.md Run the general misaligned access test and expect an ERROR on UB. Fix by ensuring UB normal DMA addresses are 32-byte aligned and not starting from a non-aligned sub-tensor at float granularity. ```bash mssanitizer --tool=memcheck ./build/demo misaligned ``` ```text ERROR: misaligned access of size ... on UB ... in "misaligned_access..." ``` -------------------------------- ### 解决 acl/acl.h 头文件找不到问题 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 当编译过程中出现 acl/acl.h 头文件找不到的问题时,手动在对应的cmake/npu/CMakeLists.txt中增加对应头文件路径。如果是x86环境,请将路径中的"aarch64-linux"换成"x86_64-linux"。 ```cmake target_include_directories(${smoke_testcase}_npu PRIVATE ${$ASCEND_HOME_PATH}/aarch64-linux/include/ ) ``` -------------------------------- ### 使能AscendCL接口检测内存泄漏 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 使用msSanitizer检测工具拉起待检测程序,使能AscendCL系列接口进行泄漏检测,用于判断用户代码调用AscendCL接口是否存在泄漏。 ```shell mssanitizer --check-cann-heap=yes --leak-check=yes ./add_npu ``` -------------------------------- ### Package sanitizer with CMake custom command and target Source: https://github.com/ascend/mssanitizer/blob/master/cmake/CMakeLists.txt Defines a custom command that parses filelist.xml, creates a self-extracting installer with makeself, sets permissions, and touches a flag file. The custom target package_sanitizer depends on build_sanitizer and the flag file, and is built by default (ALL). ```cmake add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/package.flag COMMAND cd ${ROOT_DIR}/package/script/ && ${PYTHON} parser.py -x ${ROOT_DIR}/package/conf/filelist.xml --delivery_path ${CMAKE_INSTALL_PREFIX} -o ${CMAKE_SYSTEM_PROCESSOR} COMMAND cd ${CMAKE_INSTALL_PREFIX} && ${makeselfTool} --header ${makeselfHeader} --help-header ${helpInfo} --pigz --complevel 4 --nocrc --nomd5 --sha256 --chown --tar-quietly ${NAME} ${packageName} ${comments} ${installShell} COMMAND chmod 750 ${CMAKE_INSTALL_PREFIX}/*.run COMMAND touch ${CMAKE_CURRENT_BINARY_DIR}/package.flag DEPENDS ${PACAKGE_SCRIPT_FILES} ${makeselfTool} ${makeselfHeader} WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} ) ``` ```cmake add_custom_target(package_sanitizer ALL DEPENDS build_sanitizer ${CMAKE_CURRENT_BINARY_DIR}/package.flag) ``` -------------------------------- ### Directory structure of sample_synccheck Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_synccheck/README.md Shows the file layout of the sample_synccheck directory, including the CMakeLists.txt build file, the synccheck.asc Ascend C implementation, and the README.md documentation. ```text ├── sample_synccheck/ │ ├── CMakeLists.txt // 编译工程文件 │ ├── synccheck.asc // Ascend C 样例实现(含同步异常注入) │ └── README.md // 本说明文件 ``` -------------------------------- ### 拉起算子命令 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_initcheck_multiscene/README.md 使用 mssanitizer 工具以 initcheck 模式拉起算子 demo。 ```bash mssanitizer -t initcheck ./demo ``` -------------------------------- ### One-click build with build.py Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/development_guide/develop_guide.md Runs the one-click build script in the project root. It automatically downloads dependencies, compiles, and packages the project. ```bash python3 build.py ``` -------------------------------- ### 使能Device接口检测Host侧内存泄漏 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 使用msSanitizer检测工具拉起待检测程序,使能Device系列接口进行泄漏检测,用于判断内存泄漏是否发生在Host侧。待检测程序路径可配置为绝对路径或相对路径。 ```shell mssanitizer --check-device-heap=yes --leak-check=yes ./add_npu ``` -------------------------------- ### Directory structure of sample_synccheck_redundant Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_synccheck_redundant/README.md Shows the project layout for the sample, including the CMakeLists.txt build file, the Ascend C implementation with injected redundant sync instructions, and this README. ```text ├── sample_synccheck_redundant/ │ ├── CMakeLists.txt // 编译工程文件 │ ├── synccheck_redundant.asc // Ascend C 样例实现(含冗余同步指令注入) │ └── README.md // 本说明文件 ``` -------------------------------- ### 内核调用符场景:添加链接选项 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/compile_option_config.md 在内核调用符场景中,编辑 cmake/npu/CMakeLists.txt 文件,添加链接选项 --cce-fatobj-link 和 --cce-enable-sanitizer。 ```cmake target_link_options(${smoke_testcase}_npu PRIVATE --cce-fatobj-link --cce-enable-sanitizer ) ``` -------------------------------- ### msOpGen 场景:生成算子工程 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/compile_option_config.md 在 msOpGen 算子工程场景中,运行 install.sh 脚本生成自定义算子工程,-v 参数指定芯片类型。 ```bash bash install.sh -v Ascendxxxyy # xxxyy为用户实际使用的具体芯片类型 ``` -------------------------------- ### 执行卸载命令 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/support/mssanitizer_faq.md 使用下载的脚本执行卸载操作,其中{tools_name}需替换为实际工具名称。 ```bash python ms_install.py uninstall {tools_name} ``` -------------------------------- ### Sample project directory structure Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md Directory structure of the sample project used for verification. It includes input/output directories, header files, source files, and scripts for data generation and result verification. ```text ├──input // 存放脚本生成的输入数据目录 ├──output // 存放算子运行输出数据和真值数据的目录 ├── inc // 头文件目录 │ ├── common.h // 声明公共方法类,用于读取二进制文件 │ ├── operator_desc.h // 算子描述声明文件,包含算子输入/输出,算子类型以及输入描述与输出描述 │ ├── op_runner.h // 算子运行相关信息声明文件,包含算子输入/输出个数,输入/输出大小等 ├── src │ ├── CMakeLists.txt // 编译规则文件 │ ├── common.cpp // 公共函数,读取二进制文件函数的实现文件 │ ├── main.cpp // 单算子调用应用的入口 │ ├── operator_desc.cpp // 构造算子的输入与输出描述 │ ├── op_runner.cpp // 单算子调用主体流程实现文件 ├── scripts │ ├── verify_result.py // 真值对比文件 │ ├── gen_data.py // 输入数据和真值数据生成脚本文件 │ ├── acl.json // acl配置文件 ``` -------------------------------- ### 构建单算子可执行文件 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 构建单算子可执行文件,以Add算子为例。执行完成后在工程目录下生成NPU侧可执行文件`__npu_`。 ```shell bash run.sh -r npu -v ``` -------------------------------- ### 目录结构示例 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_initcheck_multiscene/README.md 展示样例工程的文件结构,包含 CMakeLists.txt、initcheck_multiscene.asc 和 README.md。 ```text ├── sample_initcheck_multiscene/ │ ├── CMakeLists.txt // 编译工程文件 │ ├── initcheck_multiscene.asc // Ascend C 样例实现(含 GM/UB 两种未初始化场景注入) │ └── README.md // 本说明文件 ``` -------------------------------- ### 内核调用符场景:添加编译选项 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/compile_option_config.md 在内核调用符场景中,编辑 cmake/npu/CMakeLists.txt 文件,添加 --cce-enable-sanitizer 和 -g 编译选项。--cce-enable-sanitizer 与 --sanitizer 等价,-g 用于生成定位信息。 ```cmake target_compile_options(${smoke_testcase}_npu PRIVATE -O2 -std=c++17 --cce-enable-sanitizer -g ) ``` -------------------------------- ### 下载卸载脚本 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/support/mssanitizer_faq.md 在联网环境下下载msSanitizer卸载脚本。若环境不允许联网,需先在可联网环境下载后拷贝到目标设备。 ```bash curl -O https://inst.obs.cn-north-4.myhuaweicloud.com/26.0.0/ms_install.py ``` -------------------------------- ### 自动复验脚本(指定日志目录) Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 运行 test_sample.sh 脚本并传入自定义日志目录,例如 /tmp/racecheck_advanced_logs。 ```bash bash test_sample.sh /tmp/racecheck_advanced_logs ``` -------------------------------- ### 下载卸载脚本 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 需要联网环境才能下载。若环境不允许联网或处于离线状态,请先在可联网的环境下载该脚本后拷贝到目标设备。 ```bash curl -O https://inst.obs.cn-north-4.myhuaweicloud.com/26.2.0/ms_install.py ``` -------------------------------- ### 编译命令(指定 MSTX include 目录) Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 当 MSTX 头文件不在 CANN 默认搜索路径时,通过 -DMSTX_INCLUDE_DIR 显式指定其上级 include 目录。 ```bash cmake -S . -B build \ -DCMAKE_ASC_ARCHITECTURES=dav-2201 \ -DMSTX_INCLUDE_DIR=/path/to/mstx/include ``` -------------------------------- ### 通过环境变量指定 ACL 包装文件路径 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_memcheck_advanced/README.md 当 mssanitizer 的 ACL 包装文件不在默认安装目录时,通过 MSSANITIZER_ACL_INCLUDE_DIR 和 MSSANITIZER_ACL_HOOK_LIBRARY 环境变量传入路径,然后运行脚本。 ```bash MSSANITIZER_ACL_INCLUDE_DIR=/path/to/mssanitizer/include/acl \ MSSANITIZER_ACL_HOOK_LIBRARY=/path/to/libascend_acl_hook.so \ bash test_sample.sh ``` -------------------------------- ### mssanitizer startup log Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_initcheck/README.md Expected output when mssanitizer successfully launches the operator in initcheck mode. ```text [mssanitizer] logging to file: ./mindstudio_sanitizer_log/mssanitizer_XXX.log [mssanitizer] Start initcheck sanitizer on kernel "uninit_read_demo(unsigned char*, unsigned char*)" ``` -------------------------------- ### 准备 run 包 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 确保 run 包具备可执行权限。 ```bash chmod +x mindstudio-sanitizer_*.run ``` -------------------------------- ### 执行编译 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 在仓库根目录执行,自动完成依赖下载与构建,构建成功后安装包生成在 artifacts/ 目录下。 ```bash cd ~/mssanitizer python3 build.py ``` -------------------------------- ### 卸载 run 包已删除时 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 通过卸载脚本执行卸载操作。 ```shell bash $HOME/Ascend/cann/share/info/mindstudio-sanitizer/script/uninstall.sh ``` -------------------------------- ### 启用同步检测:mssanitizer --tool=synccheck Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md 启用同步检测功能(synccheck)。启动工具后,会在当前目录下自动生成工具运行日志文件mssanitizer__{TIMESTAMP}__{_PID_}.log,当用户程序运行完成后,界面会打印异常报告。 ```shell mssanitizer --tool=synccheck application # application为用户程序 ``` -------------------------------- ### 竞争检测命令 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 使用msSanitizer检测工具拉起单算子可执行文件(以_add_npu_为例)进行竞争检测。需指定 --tool=racecheck。 ```shell mssanitizer --tool=racecheck ./add_npu # 竞争检测需指定 --tool=racecheck ``` -------------------------------- ### 两种编译选项等价示例 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/support/mssanitizer_faq.md 展示--cce-enable-sanitizer和--sanitizer两种编译选项的等价用法,均可用于使能异常检测。 ```cmake # 以下两种写法效果相同 target_compile_options(${smoke_testcase}_npu PRIVATE -O2 --cce-enable-sanitizer -g) target_compile_options(${smoke_testcase}_npu PRIVATE -O2 --sanitizer -g) ``` -------------------------------- ### 目录结构示例 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 展示样例的目录结构,包含 CMakeLists.txt、README.md、racecheck_advanced.asc 和 test_sample.sh。 ```text sample_racecheck_advanced/ ├── CMakeLists.txt ├── README.md ├── racecheck_advanced.asc └── test_sample.sh ``` -------------------------------- ### 内存检测命令 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 使用msSanitizer检测工具拉起单算子可执行文件(以_add_npu_为例)进行内存检测。需指定 --tool=memcheck。 ```shell mssanitizer --tool=memcheck ./add_npu # 内存检测需指定 --tool=memcheck ``` -------------------------------- ### 执行卡间场景 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 卡间检测默认关闭,必须显式开启 --check-cross-npu-races=yes。预期输出包含 WAW、WAR、RAW,并显示 device 0 和 device 1。 ```bash mssanitizer --tool=racecheck --check-cross-npu-races=yes ./build/demo cross-npu ``` -------------------------------- ### 克隆 msSanitizer 仓库 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 在 ctr_in.py 打开的交互式容器 Shell 中执行,用于获取 msSanitizer 源码。 ```bash cd ~ git clone https://gitcode.com/Ascend/mssanitizer.git ``` -------------------------------- ### 验证安装 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 执行 mssanitizer --help 验证工具是否安装成功。若输出不报错且能显示帮助信息,则安装成功。 ```shell mssanitizer --help ``` -------------------------------- ### AscendCL接口内存泄漏检测输出 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 当AscendCL接口调用出现内存泄漏时,msSanitizer输出的错误信息。结果表明调用AscendCL接口时共有一处分配了内存但未释放,导致内存泄漏32768字节。 ```text ====== ERROR: LeakCheck: detected memory leaks ====== Direct leak of 32768 byte(s) ====== at 0x124080024000 on GM allocated in :0 (serialNo:0) ====== SUMMARY: 32768 byte(s) leaked in 1 allocation(s) ``` -------------------------------- ### 启用未初始化检测(initcheck) Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md 启用未初始化检测功能(initcheck)。启动工具后,会在当前目录下自动生成工具运行日志文件mssanitizer__{TIMESTAMP}__{_PID_}.log。由于硬件限制,某些指令仅支持以Block形式进行数据搬运,当实际数据量不是Block大小的整数倍时,可能带入无效数据(脏数据),导致报告初始化异常,用户需自行判断。未初始化检测结果准确的前提是不存在数据竞争,建议先处理竞争问题。当前仅支持Atlas A2系列、Atlas A3系列以及Ascend 950PR&950DT系列产品,其中Ascend 950PR&950DT系列不支持UB地址空间未初始化检测。 ```shell mssanitizer --tool=initcheck application # application为用户程序 ``` -------------------------------- ### Sample Triton Operator (sample.py) Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md A sample Triton operator implementation (sample.py) used to demonstrate the detection flow. It defines a pointwise add kernel and a test case that runs it on NPU tensors. ```py # file name: sample.py import triton import triton.language as tl import torch def torch_pointwise(x0, x1): res = x0 + x1 return res @triton.jit def triton_add(in_ptr0, in_ptr1, out_ptr0, XBLOCK: tl.constexpr, XBLOCK_SUB: tl.constexpr): offset = tl.program_id(0) * XBLOCK base1 = tl.arange(0, XBLOCK_SUB) loops1: tl.constexpr = (XBLOCK + XBLOCK_SUB - 1) // XBLOCK_SUB for loop1 in range(loops1): x0 = offset + (loop1 * XBLOCK_SUB) + base1 tmp0 = tl.load(in_ptr0 + (x0), None) tmp1 = tl.load(in_ptr1 + (x0), None) tmp2 = tmp0 + tmp1 tl.store(out_ptr0 + (x0), tmp2, None) def test_case(dtype, shape, ncore, xblock, xblock_sub): x0 = torch.randn(shape, dtype=dtype).npu() x1 = torch.randn(shape, dtype=dtype).npu() y_ref = torch_pointwise(x0, x1) y_cal = torch.zeros(shape, dtype=dtype).npu() triton_add[ncore, 1, 1](x0, x1, y_cal, xblock, xblock_sub) print("Pass" if torch.equal(y_ref, y_cal) else "Failed") if __name__ == "__main__": test_case(torch.float32, (2, 4096, 8), 2, 32768, 1024) ``` -------------------------------- ### Host侧内存泄漏检测输出 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md 当Host侧存在内存泄漏时,msSanitizer输出的错误信息。结果表明Host侧共有一处分配了内存但未释放,导致内存泄漏32800字节。 ```text ====== ERROR: LeakCheck: detected memory leaks ====== Direct leak of 32800 byte(s) ====== at 0x124080024000 on GM allocated in :0 (serialNo:0) ====== SUMMARY: 32800 byte(s) leaked in 1 allocation(s) ``` -------------------------------- ### Run msSanitizer initcheck Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/quick_start/mssanitizer_quick_start.md Runs the msSanitizer initialization check tool to detect uninitialized memory reads. ```bash cd ~/ot_demo/workspace/src/caller mssanitizer --tool=initcheck -- bash run.sh ``` -------------------------------- ### 内核调用符场景:下载样例代码 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/compile_option_config.md 下载内核调用符场景的样例代码,指定分支版本 8.0.RC2,并切换到 AddKernelInvocation 目录。 ```sh git clone https://gitee.com/ascend/samples.git -b 8.0.RC2 cd operator/AddCustomSample/KernelLaunch/AddKernelInvocation ``` -------------------------------- ### 拉起算子命令 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_memcheck/README.md 使用 mssanitizer 工具以 memcheck 模式拉起算子,memcheck 为默认模式,可省略 -t 参数。 ```bash mssanitizer -t memcheck ./demo ``` -------------------------------- ### 配置ASLR级别2 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/legal/security_statement.md 配置系统ASLR级别为2(全随机地址空间布局随机化)。需要root权限执行,写入/proc/sys/kernel/randomize_va_space。 ```shell echo 2 > /proc/sys/kernel/randomize_va_space ``` -------------------------------- ### 执行流水间场景 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 运行 mssanitizer 的 racecheck 工具检测流水间竞争。预期输出包含 WAW、WAR、RAW 三种潜在竞争。 ```bash mssanitizer --tool=racecheck ./build/demo pipeline ``` -------------------------------- ### msOpGen 场景:添加编译选项 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/compile_option_config.md 在 msOpGen 算子工程场景中,编辑 op_kernel/CMakeLists.txt 文件,添加 -sanitizer 编译选项。 ```cmake add_ops_compile_options(ALL OPTIONS -sanitizer) ``` -------------------------------- ### Clone Ascend samples repository Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/best_practices/mssanitizer_basic_cases.md Clone the Ascend samples repository at the specified branch version. This is required before using the AclNNInvocation sample project for operator detection. ```shell git clone https://gitee.com/ascend/samples.git -b v0.2-8.0.0.beta1 ``` -------------------------------- ### 执行核间场景 Source: https://github.com/ascend/mssanitizer/blob/master/example/sample_racecheck_advanced/README.md 运行 mssanitizer 的 racecheck 工具检测核间竞争。预期输出包含 WAW、WAR、RAW,并显示 block 0 和 block 1。 ```bash mssanitizer --tool=racecheck ./build/demo cross-core ``` -------------------------------- ### 启用卡间竞争检测(--check-cross-npu-races=yes) Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md 额外开启卡间竞争检测功能。 ```shell mssanitizer --tool=racecheck --check-cross-npu-races=yes application ``` -------------------------------- ### msSanitizer command format Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/user_guide/mssanitizer_user_guide.md Shows the general command-line format for invoking the msSanitizer tool. Use `--` before the user program when it has its own command-line arguments. ```shell mssanitizer [] [--] [] ``` -------------------------------- ### ASCEND_CUSTOM_OPP_PATH warning Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/quick_start/mssanitizer_quick_start.md Warning shown during redeployment when the ASCEND_CUSTOM_OPP_PATH environment variable is incorrect or contains multiple colon-separated paths. Requires manually unsetting the variable before redeploying. ```text [ERROR] environment variable ASCEND_CUSTOM_OPP_PATH=/home/gitcode/samples/operator/ascendc/0_introduction/12_matmulleakyrelu_frameworklaunch/CustomOp/build_out/test/vendors/customize: is set and has multiple path in it (colon inside), which will cause the custom op installed incorrectly. Please use the --install-path option to specify an installation path instead. ``` -------------------------------- ### Run unit tests with build.py test Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/development_guide/develop_guide.md Runs unit tests. A return code of 0 and no failed test cases indicate success. If any FAILED cases appear, check the logs to diagnose. ```bash python3 build.py test ``` -------------------------------- ### 安装 run 包 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/install_guide/mssanitizer_install_guide.md 执行安装操作。若已安装旧版本,会提示是否替换,输入 y 可覆盖安装。 ```shell ./mindstudio-sanitizer_*.run --run ``` -------------------------------- ### 设置环境变量关闭PyTorch NPU内存池 Source: https://github.com/ascend/mssanitizer/blob/master/docs/zh/support/mssanitizer_faq.md 在PyTorch场景下进行内存检测前,设置此环境变量以关闭NPU内存池,避免内存池复用干扰检测逻辑。Triton算子调用场景也需设置此变量。 ```shell export PYTORCH_NO_NPU_MEMORY_CACHING=1 ```