### Install and Start Development Server

Source: https://github.com/microsoft/superbenchmark/blob/main/website/README.md

Installs project dependencies and starts a local development server for the SuperBench website. Changes are reflected live.

```bash
cd website
npm install
npm start
```

--------------------------------

### Installation Command

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cpu_copy_performance/CMakeLists.txt

Installs the compiled cpu_copy executable to the bin directory.

```cmake
install(TARGETS cpu_copy RUNTIME DESTINATION bin)
```

--------------------------------

### Micro-Benchmark Example

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/superbench-config.mdx

An example configuration for a micro-benchmark named 'kernel-launch', specifying enable status, timeout, modes, and parameters for warm-up and steps.

```yaml
kernel-launch:
  enable: true
  timeout: 120
  modes:
    - name: local
      proc_num: 8
      prefix: CUDA_VISIBLE_DEVICES={proc_rank}
      parallel: yes
  parameters:
    num_warmup: 100
    num_steps: 2000000
    interval: 2000
```

--------------------------------

### Installing Benchmark Binaries and Libraries

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cublas_function/CMakeLists.txt

Installs the main benchmark executable and the cublas library to specified runtime destinations.

```cmake
install(TARGETS cublas_benchmark ${TARGET_NAME} RUNTIME DESTINATION bin LIBRARY DESTINATION lib)
```

--------------------------------

### Install Target

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cuda_decode_performance/CMakeLists.txt

Installs the target executable to the 'bin' directory and libraries to the 'lib' directory.

```cmake
install(TARGETS ${PROJECT_NAME} RUNTIME DESTINATION bin LIBRARY DESTINATION lib)
```

--------------------------------

### Installation Rule for Executable

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/gpu_stream/CMakeLists.txt

Installs the built 'gpu_stream' executable to the 'bin' directory in the runtime destination. This makes the benchmark easily accessible after installation.

```cmake
install(TARGETS gpu_stream RUNTIME DESTINATION bin)
```

--------------------------------

### Install Target

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/gpu_copy_performance/CMakeLists.txt

Installs the built 'gpu_copy' executable to the 'bin' directory.

```cmake
install(TARGETS gpu_copy RUNTIME DESTINATION bin)
```

--------------------------------

### Monitor Configuration Example

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/monitor.md

Enable the monitor module and set sampling parameters in the configuration file.

```yaml
superbench:
  monitor:
    enable: bool
    sample_duration: int
    sample_interval: int
```

--------------------------------

### Collect System Info on Local Machine

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/system-config.md

Use this command to start collecting system information on a local machine. Requires root privilege and SuperBench installation.

```bash
sb node info --output-dir ${output-dir}
```

--------------------------------

### Rule File Example

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/result-summary.md

An example of a SuperBench rule file demonstrating how to define rules for kernel launch, NCCL, and IB loopback benchmarks, including statistics, categories, and aggregation.

```yaml
# SuperBench rules
version: v0.12
superbench:
  rules:
    kernel_launch:
      statistics:
        - mean
        - p90
        - min
        - max
      aggregate: True
      categories: KernelLaunch
      metrics:
        - kernel-launch/event_time
        - kernel-launch/wall_time
    nccl:
      statistics: mean
      categories: NCCL
      metrics:
        - nccl-bw/allreduce_8388608_busbw
    ib-loopback:
      statistics: mean
      categories: RDMA
      metrics:
        - ib-loopback/IB_write_8388608_Avg_\d+
      aggregate: ib-loopback/IB_write_.*_Avg_(\d+)
```

--------------------------------

### Local Mode Example

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/superbench-config.mdx

An example configuration for the 'local' benchmark mode. This sets the number of processes and a prefix for command execution.

```yaml
name: local
proc_num: 8
prefix: CUDA_VISIBLE_DEVICES={proc_rank}
parallel: yes
```

--------------------------------

### SuperBench Configuration Example

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/superbench-config.mdx

An example illustrating a typical SuperBench configuration, showing how to enable benchmarks, configure monitoring, define variables, and specify benchmark settings.

```yaml
version: v0.12
superbench:
  enable: benchmark_1
  monitor:
    enable: false
    sample_duration: 10
    sample_interval: 1
  var:
    var_1: value
  benchmarks:
    benchmark_1:
      enable: true
      modes:
      - name: local
```

--------------------------------

### CMake Minimum Version and Project Setup

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/gpu_stream/CMakeLists.txt

Sets the minimum required CMake version and defines the project name and languages. This is a standard starting point for CMake projects.

```cmake
cmake_minimum_required(VERSION 3.18)

project(gpu_stream LANGUAGES CXX)
```

--------------------------------

### Link Libraries and Install Target

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/ib_validation_performance/CMakeLists.txt

Links the executable against MPI and Boost libraries, and specifies installation rules.

```cmake
target_link_libraries(ib_validation PUBLIC MPI::MPI_CXX ${Boost_LIBRARIES})
install(TARGETS ib_validation RUNTIME DESTINATION bin)
```

--------------------------------

### Expanded Configuration without Anchors and Aliases

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/superbench-config.mdx

Shows the equivalent configuration to the example using anchors and aliases, but with all parameters explicitly defined for each benchmark.

```yaml
superbench:
  benchmarks:
    model-benchmarks:foo:
      models:
        - resnet50
      parameters:
        num_warmup: 16
        num_steps: 128
        batch_size: 128
    model-benchmarks:bar:
      models:
        - vgg19
      parameters:
        num_warmup: 16
        num_steps: 128
        batch_size: 128
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/developer-guides/development.md

Clone the Superbenchmark repository with submodules and install development dependencies using pip.

```bash
git clone --recurse-submodules -j8 https://github.com/microsoft/superbenchmark
cd superbenchmark

python3 -m pip install -e .[develop]
```

--------------------------------

### Model-Benchmark Example

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/superbench-config.mdx

An example configuration for a model-benchmark with the annotation 'resnet', specifying enable status, timeout, modes, frameworks, models, and parameters for duration, warm-up, steps, batch size, precision, and model action.

```yaml
model-benchmarks:resnet:
  enable: true
  timeout: 1800
  modes:
    - name: torch.distributed
      proc_num: 8
      node_num: 1
  frameworks:
    - pytorch
  models:
    - resnet50
    - resnet101
    - resnet152
  parameters:
    duration: 0
    num_warmup: 16
    num_steps: 128
    batch_size: 128
    precision:
      - float32
      - float16
    model_action:
      - train
```

--------------------------------

### Basic CMake Project Setup

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/dist_inference_cpp/CMakeLists.txt

Sets the minimum CMake version and defines the project name and languages.

```cmake
cmake_minimum_required(VERSION 3.18)

project(dist_inference LANGUAGES CXX)
```

--------------------------------

### Install sshpass

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/installation.mdx

Install sshpass on the control node if you plan to use password authentication for SSH connections to managed nodes.

```bash
sudo apt-get install sshpass
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/developer-guides/development.md

Install pre-commit hooks to automatically run checks before committing code. This ensures code quality and consistency.

```bash
pre-commit install --install-hooks
```

--------------------------------

### Get System Product Name (Virtual Machine)

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/system-config.md

Retrieves the product name or virtual machine identifier. This command is part of system information collection.

```bash
dmidecode -s system-product-name
```

--------------------------------

### CUDA Environment Setup

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cpu_copy_performance/CMakeLists.txt

Configures the build for CUDA if found. Includes common CUDA settings, adds the executable, sets CUDA architectures, and links the NUMA library.

```cmake
if(CUDAToolkit_FOUND)
    message(STATUS "Found CUDA: " ${CUDAToolkit_VERSION})

    include(../cuda_common.cmake)
    add_executable(cpu_copy cpu_copy.cpp)
    set_property(TARGET cpu_copy PROPERTY CUDA_ARCHITECTURES ${NVCC_ARCHS_SUPPORTED})
    target_link_libraries(cpu_copy numa)
else()
    # ROCm environment
    include(../rocm_common.cmake)
    find_package(hip QUIET)
    if(hip_FOUND)
        message(STATUS "Found ROCm: " ${HIP_VERSION})

        # Convert cuda code to hip code in cpp
        execute_process(COMMAND hipify-perl -print-stats -o cpu_copy.cpp cpu_copy.cu WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/)

        # link hip device lib
        add_executable(cpu_copy cpu_copy.cpp)

        include(CheckSymbolExists)
        check_symbol_exists("hipDeviceMallocUncached" "hip/hip_runtime_api.h" HIP_UNCACHED_MEMORY)
        if(${HIP_UNCACHED_MEMORY})
            target_compile_definitions(cpu_copy PRIVATE HIP_UNCACHED_MEMORY)
        endif()

        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O2")
        target_link_libraries(cpu_copy numa hip::device)
    else()
        message(FATAL_ERROR "No CUDA or ROCm environment found.")
    endif()
endif()
```

--------------------------------

### Registering Microbenchmark

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/design-docs/benchmarks.md

Example of registering a microbenchmark, such as 'kernel-launch', using the BenchmarkRegistry.register_benchmark method.

```python
BenchmarkRegistry.register_benchmark('kernel-launch', KernelLaunch)
```

--------------------------------

### Run SuperBench CLI

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/installation.mdx

After successful installation, you can invoke the SuperBench Command Line Interface (CLI) using the 'sb' command.

```bash
sb
```

--------------------------------

### Get System Information Summary

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/system-config.md

Provides a short summary of system information, including kernel version. This command is part of system information collection.

```bash
uname
```

--------------------------------

### Example Rule File for Data Diagnosis

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/data-diagnosis.md

Illustrates various rules for data diagnosis, including failure checks, variance analysis for different categories like KernelLaunch, Mem, NCCL, Model, and CNN, as well as multi-rule combinations.

```yaml
# SuperBench rules
version: v0.12
superbench:
  rules:
    failure-rule:
      function: failure_check
      criteria: lambda x:x>0
      categories: Failed
      metrics:
        - kernel-launch/return_code
        - mem-bw/return_code
        - nccl-bw/return_code
        - ib-loopback/return_code
    rule0:
    # Rule 0: If KernelLaunch suffers > 5% downgrade, label it as defective
      function: variance
      criteria: lambda x:x>0.05
      categories: KernelLaunch
      metrics:
        - kernel-launch/event_time:\d+
        - kernel-launch/wall_time:\d+
    rule1:
    # Rule 1: If H2D_Mem_BW or D2H_Mem_BW test suffers > 5% downgrade, label it as defective
      function: variance
      criteria: lambda x:x<-0.05
      categories: Mem
      metrics:
        - mem-bw/H2D_Mem_BW:\d+
        - mem-bw/D2H_Mem_BW:\d+
    rule2:
    # Rule 2: If NCCL_BW suffers > 5% downgrade, label it as defective
      function: variance
      criteria: lambda x:x<-0.05
      categories: NCCL
      metircs:
        - nccl-bw/allreduce_8589934592_busbw:0
    rule3:
    # Rule 3: If GPT-2, BERT suffers > 5% downgrade, label it as defective
      function: variance
      criteria: lambda x:x<-0.05
      categories: Model
      metrics:
        - bert_models/pytorch-bert-base/throughput_train_float(32|16)
        - bert_models/pytorch-bert-large/throughput_train_float(32|16)
        - gpt_models/pytorch-gpt-large/throughput_train_float(32|16)
    rule4:
      function: variance
      criteria: "lambda x:x<-0.05"
      store: True
      categories: CNN
      metrics:
        - resnet_models/pytorch-resnet.*/throughput_train_.*
    rule5:
      function: variance
      criteria: "lambda x:x<-0.05"
      store: True
      categories: CNN
      metrics:
        - vgg_models/pytorch-vgg.*/throughput_train_.*\
    rule6:
      function: multi_rules
      criteria: 'lambda label: bool(label["rule4"]+label["rule5"]>=2)'
      categories: CNN
    rule7:
      categories: MODEL_DIST
      store: True
      metrics:
        - model-benchmarks:stress-run.*/pytorch-gpt2-large/fp32_train_throughput
    rule8:
      function: multi_rules
      criteria: 'lambda label: bool(min(label["rule7"].values()))<1)'
      categories: MODEL_DIST

```

--------------------------------

### Print SuperBench CLI Version

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Displays the current version of the SuperBench CLI. This is a fundamental command for checking the installed version.

```bash
sb version
```

--------------------------------

### Registering E2E Model Benchmarks

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/design-docs/benchmarks.md

Example of how to register E2E model benchmarks using the BenchmarkRegistry.register_benchmark method, specifying benchmark name, class, and arguments.

```python
BenchmarkRegistry.register_benchmark('bert-large', PytorchBERT, args='--hidden_size=1024 --num_hidden_layers=24 --num_attention_heads=16 --intermediate_size=4096')
BenchmarkRegistry.register_benchmark('bert-base', PytorchBERT, args='--hidden_size=768 --num_hidden_layers=12 --num_attention_heads=12 --intermediate_size=3072')
```

--------------------------------

### Get System Information

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/developer-guides/contributing.md

Execute this Python script to automatically gather system information for benchmark results. The script is located in the `superbench/tools` folder.

```python
python system_info.py
```

--------------------------------

### Get Docker Server and Client Versions

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/system-config.md

Retrieves both the server and client versions of the Docker engine. This command is part of system information collection.

```bash
docker version
```

--------------------------------

### Get Operating System Version

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/system-config.md

Retrieves the version of the currently running operating system. This command is part of system information collection.

```bash
cat /proc/version
```

--------------------------------

### Get Local Node System Info

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Retrieves system information from the local node using the `sb node info` command. Optionally specify an output directory.

```bash
sb node info [--output-dir]
```

```bash
sb node info --output-dir outputs
```

--------------------------------

### CMake Minimum Version and Project Setup

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cublaslt_gemm/CMakeLists.txt

Sets the minimum required CMake version and defines the project name and languages. It also configures the CUDA runtime library to be shared.

```cmake
cmake_minimum_required(VERSION 3.18)
set(CMAKE_CUDA_RUNTIME_LIBRARY SHARED)
project(cublaslt_gemm LANGUAGES CXX)
```

--------------------------------

### Generate Baseline JSON

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/baseline-generation.md

Use this command to generate a baseline JSON file from raw benchmark data and rule files. Ensure SuperBench is installed and raw data/rule files are prepared.

```bash
sb result generate-baseline --data-file ./results-summary.jsonl --summary-rule-file ./summary-rule.yaml --diagnosis-rule-file ./diagnosis-rule.yaml --output-dir ${output-dir}
```

--------------------------------

### Creating and Launching a Benchmark

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/design-docs/benchmarks.md

Demonstrates the process of creating a benchmark context and then launching the benchmark using the BenchmarkRegistry. Requires benchmark_name, parameters, framework, and platform to be defined.

```python
    context = BenchmarkRegistry.create_benchmark_context(
        benchmark_name, parameters=xxx, framework=xxx, platform=xxx
    )

    benchmark = BenchmarkRegistry.launch_benchmark(context)
    if benchmark:
        logger.info(
            'benchmark: {}, return code: {}, result: {}'.format(
                benchmark.name, benchmark.return_code, benchmark.result
            )
        )
```

--------------------------------

### Run Unit Tests

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/developer-guides/development.md

Execute the unit tests for the project using the setup.py script.

```bash
python3 setup.py test
```

--------------------------------

### Using YAML Anchors and Aliases for Configuration

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/superbench-config.mdx

Demonstrates how to use YAML anchors and aliases to define common parameters once and reuse them across multiple benchmark configurations, reducing redundancy.

```yaml
superbench:
  var:
    common_param: &param
      num_warmup: 16
      num_steps: 128
      batch_size: 128
  benchmarks:
    model-benchmarks:foo:
      models:
        - resnet50
      parameters: *param
    model-benchmarks:bar:
      models:
        - vgg19
      parameters: *param
```

--------------------------------

### List Parameters for All SuperBench Benchmarks

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Use this command to list parameters for all benchmarks.

```bash
sb benchmark list-parameters
```

--------------------------------

### Format Code with Yapf

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/developer-guides/development.md

Use the setup.py script to format the code according to project standards using yapf.

```bash
python3 setup.py format
```

--------------------------------

### Get System Manufacturer

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/system-config.md

Retrieves the manufacturer of the system. This command is part of system information collection.

```bash
dmidecode -s system-manufacturer
```

--------------------------------

### List All SuperBench Benchmarks

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Use this command to list all available benchmarks in the SuperBench CLI.

```bash
sb benchmark list
```

--------------------------------

### SB CLI Command Structure for Run

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

This outlines the available arguments for the 'sb run' command, used for distributedly running SuperBench benchmarks. It includes options for configuration, Docker, and host management.

```bash
sb run [--config-file]
       [--config-override]
       [--docker-image]
       [--docker-password]
       [--docker-username]
       [--get-info]
       [--host-file]
       [--host-list]
       [--host-password]
       [--host-username]
       [--no-docker]
       [--output-dir]
       [--private-key]
```

--------------------------------

### Deploy SuperBench Environment Locally

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/run-superbench.md

Use the `sb deploy` command with a local inventory file to set up the SuperBench environment on managed nodes. This command handles accessing nodes, pulling container images, and preparing containers.

```bash
sb deploy -f local.ini
```

--------------------------------

### Run Benchmarks on Localhost

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Executes all benchmarks on the local GPU node. This is a basic command for initiating benchmark runs on a single machine.

```bash
sb run --host-list localhost
```

--------------------------------

### Run Data Diagnosis Command

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/data-diagnosis.md

Execute the SuperBench data diagnosis command with specified input and output files. Ensure SuperBench is installed and input files are prepared.

```bash
sb result diagnosis --data-file ./results-summary.jsonl --rule-file ./rule.yaml --baseline-file ./baseline.json --output-file-format excel --output-dir ${output-dir}
```

--------------------------------

### Run SuperBench Benchmarks

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/run-superbench.md

After deployment, execute SuperBench benchmarks using the `sb run` command, specifying the inventory file and the benchmark configuration YAML file.

```bash
sb run -f local.ini -c resnet.yaml
```

--------------------------------

### Define Application and NvDecoder Source Files

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cuda_decode_performance/CMakeLists.txt

Lists the source files for the main application and the NvDecoder components.

```cmake
set(APP_SOURCES
   ${CMAKE_CURRENT_SOURCE_DIR}/AppDecPerf.cpp
   )

set(NV_DEC_SOURCES
   ${NV_DEC_DIR}/NvDecoder.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/OptimizedNvDecoder.cpp
   )
```

--------------------------------

### Build SuperBench from Source

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/installation.mdx

Clone the SuperBench repository from GitHub and build the project locally. Ensure you checkout a specific tag for release versions.

```bash
git clone https://github.com/microsoft/superbenchmark
cd superbenchmark

python3 -m pip install .
make postinstall
```

--------------------------------

### Execute SuperBench Benchmarks Locally

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Executes SuperBench benchmarks locally. Configuration files and output directories can be specified.

```bash
sb exec [--config-file]
        [--config-override]
        [--output-dir]

```

--------------------------------

### CUDA Environment Configuration

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/kernel_launch_overhead/CMakeLists.txt

Configures the build for CUDA environments. It finds the CUDAToolkit, includes common CUDA settings, adds the executable, sets CUDA architectures, and installs the target.

```cmake
find_package(CUDAToolkit QUIET)

# Cuda environment
if(CUDAToolkit_FOUND)
    message(STATUS "Found CUDA: " ${CUDAToolkit_VERSION})

    include(../cuda_common.cmake)
    add_executable(kernel_launch_overhead kernel_launch.cu)
    set_property(TARGET kernel_launch_overhead PROPERTY CUDA_ARCHITECTURES ${NVCC_ARCHS_SUPPORTED})
    install(TARGETS kernel_launch_overhead RUNTIME DESTINATION bin)
else()
    # ROCm environment
    include(../rocm_common.cmake)
    find_package(hip QUIET)
    if(hip_FOUND)
        message(STATUS "Found HIP: " ${HIP_VERSION})

        # Convert cuda code to hip code in cpp
        execute_process(COMMAND hipify-perl -print-stats -o kernel_launch.cpp kernel_launch.cu WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/)

        # link hip device lib
        add_executable(kernel_launch_overhead kernel_launch.cpp)
        target_link_libraries(kernel_launch_overhead hip::device)
        # Install tergets
        install(TARGETS kernel_launch_overhead RUNTIME DESTINATION bin)
    else()
        message(FATAL_ERROR "No CUDA or ROCm environment found.")
    endif()
endif()
```

--------------------------------

### Run Kernel Launch Benchmarks Directly on Host

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Executes kernel launch benchmarks directly on the host without using Docker. This command allows for specific benchmark enablement and path configuration.

```bash
sb run --no-docker --host-list localhost --config-override \
  superbench.enable=kernel-launch superbench.env.SB_MICRO_PATH=/path/to/superbenchmark
```

--------------------------------

### Run SB Result Summary with HTML Output

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

This command generates a summary of benchmark results and outputs them in HTML format. Ensure you provide the correct data and rule files.

```bash
sb result summary --data-file outputs/results-summary.jsonl --rule-file rule.yaml --output-file-format html
```

--------------------------------

### Collect System Info on Multiple Remote Machines

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/system-config.md

Initiate system information collection across multiple remote machines. Ensure SuperBench is installed locally and deployed on remote machines, with a prepared host file.

```bash
sb run --get-info -f host.ini --output-dir ${output-dir} -C superbench.enable=none
```

--------------------------------

### ROCm/HIP Environment Configuration

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/kernel_launch_overhead/CMakeLists.txt

Configures the build for ROCm/HIP environments. It finds the hip package, converts CUDA code to HIP using hipify-perl, adds the executable, links the hip::device library, and installs the target.

```cmake
# ROCm environment
include(../rocm_common.cmake)
find_package(hip QUIET)
if(hip_FOUND)
    message(STATUS "Found HIP: " ${HIP_VERSION})

    # Convert cuda code to hip code in cpp
    execute_process(COMMAND hipify-perl -print-stats -o kernel_launch.cpp kernel_launch.cu WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/)

    # link hip device lib
    add_executable(kernel_launch_overhead kernel_launch.cpp)
    target_link_libraries(kernel_launch_overhead hip::device)
    # Install tergets
    install(TARGETS kernel_launch_overhead RUNTIME DESTINATION bin)
else()
    message(FATAL_ERROR "No CUDA or ROCm environment found.")
endif()
```

--------------------------------

### Basic CMake Configuration

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/gpu_copy_performance/CMakeLists.txt

Sets the minimum CMake version and project name. Languages supported are CXX.

```cmake
cmake_minimum_required(VERSION 3.18)

project(gpu_copy LANGUAGES CXX)
```

--------------------------------

### CUDNN Function Benchmark CMakeLists

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cudnn_function/CMakeLists.txt

This CMakeLists.txt file configures the build for the CUDNN function benchmark. It finds CUDA, sets up source and target names, and links against CUDA, cuDNN, and nlohmann_json libraries. It also installs the built targets.

```cmake
cmake_minimum_required(VERSION 3.18)
project(cudnn_benchmark LANGUAGES CXX)

find_package(CUDAToolkit QUIET)
if(CUDAToolkit_FOUND)
  include(../cuda_common.cmake)
  set(SRC "cudnn_helper.cpp" CACHE STRING "source file")
  set(TARGET_NAME "cudnn_function" CACHE STRING "target name")

  set(CUDA_NVCC_FLAGS "${CUDA_NVCC_FLAGS} ${NVCC_ARCHS_SUPPORTED}")
  add_library(${TARGET_NAME} SHARED ${SRC})
  link_directories( ${CUDAToolkit_LIBRARY_DIR} ${CUDAToolkit_TARGET_DIR})
  include_directories( ${CUDAToolkit_INCLUDE_DIRS})
  find_library(CUDNN_LIBRARY cudnn
      HINTS ${CUDAToolkit_ROOT_DIR}
      PATH_SUFFIXES lib lib64 cuda/lib cuda/lib64 lib/x64)

  include(FetchContent)
  FetchContent_Declare (json
    GIT_REPOSITORY https://github.com/ArthurSonzogni/nlohmann_json_cmake_fetchcontent
    GIT_TAG v3.7.3)
  FetchContent_GetProperties(json)
  if(NOT json_POPULATED)
    FetchContent_Populate(json)
    add_subdirectory(${json_SOURCE_DIR} ${json_BINARY_DIR} EXCLUDE_FROM_ALL)
  endif()

  add_executable(cudnn_benchmark cudnn_test.cpp)
  target_link_libraries(cudnn_benchmark ${TARGET_NAME} nlohmann_json::nlohmann_json CUDA::cudart ${CUDNN_LIBRARY})
  install(TARGETS cudnn_benchmark ${TARGET_NAME} RUNTIME DESTINATION bin LIBRARY DESTINATION lib)
endif()
```

--------------------------------

### Run Benchmarks Locally

Source: https://github.com/microsoft/superbenchmark/blob/main/website/blog/2021-06-24-introduce-superbench.md

Execute benchmarks using the 'sb run' command, specifying the worker node configuration and benchmark details. This command requires both a worker node configuration file and a benchmark configuration file.

```bash
sb run -f local.ini -c config.yaml
```

--------------------------------

### Create Symbolic Link for libnvcuvid.so

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cuda_decode_performance/CMakeLists.txt

Checks for the existence of libnvcuvid.so and creates a symbolic link if it's missing, using sudo.

```cmake
if ( NOT EXISTS "/usr/local/lib/libnvcuvid.so" )
   execute_process(
       COMMAND sudo ln -s /usr/lib/x86_64-linux-gnu/libnvcuvid.so.1 /usr/local/lib/libnvcuvid.so
       RESULT_VARIABLE result
   )
   if(result)
       message(FATAL_ERROR "Failed to create symbolic link for nvcuvid lib: ${result}")
   endif()
endif()
```

--------------------------------

### Basic CMake Configuration

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cpu_copy_performance/CMakeLists.txt

Sets the minimum CMake version and project name. Finds the CUDAToolkit package.

```cmake
cmake_minimum_required(VERSION 3.18)

project(cpu_copy LANGUAGES CXX)

find_package(CUDAToolkit QUIET)
```

--------------------------------

### Run SuperBench Directly Inside a Container (No Docker)

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/run-superbench.md

For environments like Kubernetes where `sb deploy` might not be suitable, run `sb run` directly inside a pre-created privileged container. Use the `--no-docker` argument and specify the host address.

```bash
sb run --no-docker -l localhost -c resnet.yaml
```

--------------------------------

### Generate Benchmark Summary Report

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Generates a summary report from benchmarking results. Requires data and rule files, with optional arguments for decimal places, output directory, and file format.

```bash
sb result summary --data-file
                  --rule-file
                  [--decimal-place-value]
                  [--output-dir]
                  [--output-file-format {md, excel, html}]
```

--------------------------------

### List Parameters for SuperBench Benchmarks by Name Pattern

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Use this command to list parameters for benchmarks matching a specific regular expression for their name.

```bash
sb benchmark list-parameters --name pytorch-[a-z]+
```

--------------------------------

### Create and Activate Python Virtual Environment

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/installation.mdx

It is recommended to use Python virtual environments (venv) for managing dependencies. This snippet shows how to create, activate, and later deactivate a virtual environment.

```bash
# create a new virtual environment
python3 -m venv ./venv
# activate the virtual environment
source ./venv/bin/activate

# exit the virtual environment later
# after you finish running superbench
deactivate
```

--------------------------------

### Lint Code with Mypy and Flake8

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/developer-guides/development.md

Check code style and type hints using mypy and flake8 via the setup.py script.

```bash
python3 setup.py lint
```

--------------------------------

### Collect System Info While Running Benchmarks

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Collects system information from all nodes in a host file while simultaneously running benchmarks. This command is useful for gathering diagnostic data during active benchmark sessions.

```bash
sb run --get-info --host-file ./host.ini
```

--------------------------------

### Deploy Default SuperBench Environment

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Deploys the default SuperBench Docker image to a local GPU node.

```bash
sb deploy --host-list localhost
```

--------------------------------

### SuperBench CLI Welcome Message

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Displays the welcome message and ASCII art for the SuperBench CLI.

```bash
$ sb

   _____                       ____                  _ 
  / ____|                     |  _ \                | |
 | (___  _   _ _ __   ___ _ __| |_) | ___ _ __   ___| |__
  \___ \| | | | '_ \ / _ \ '__|  _ < / _ \ '_ \ / __| '_ \
  ____) | |_| | |_) |  __/ |  | |_) |  __/ | | | (__| | | |
 |_____/ \__,_| .__/ \___|_|  |____/ \___|_| |_|\___|_| |_|
              | |
              |_|

Welcome to the SB CLI!

```

--------------------------------

### Run SB Result Summary with Markdown Output

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Use this command to generate a summary of benchmark results and output them in markdown format. It allows specifying the data file, rule file, output format, and decimal places for precision.

```bash
sb result summary --data-file outputs/results-summary.jsonl --rule-file rule.yaml --output-file-format md --decimal-place-value 2
```

--------------------------------

### Creating Executable and Linking Libraries

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cublas_function/CMakeLists.txt

Defines the main benchmark executable and links it against the cublas library, nlohmann_json, and CUDA runtime/cublas libraries.

```cmake
add_executable(cublas_benchmark cublas_test.cpp)
target_link_libraries(cublas_benchmark ${TARGET_NAME} nlohmann_json::nlohmann_json CUDA::cudart CUDA::cublas)
```

--------------------------------

### Copy Default Configuration

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/getting-started/configuration.md

Copies the default SuperBench configuration file to a new file for customization.

```bash
cp superbench/config/default.yaml resnet.yaml
```

--------------------------------

### Basic CMake Configuration for cublas_function

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cublas_function/CMakeLists.txt

Sets up the minimum CMake version, project name, and language. This is the foundational part of the build script.

```cmake
cmake_minimum_required(VERSION 3.18)
project(cublas_benchmark LANGUAGES CXX)
```

--------------------------------

### List SuperBench Benchmarks by Name Pattern

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Use this command to list benchmarks matching a specific regular expression for their name.

```bash
sb benchmark list --name [a-z]+-bw
```

--------------------------------

### Generate Baseline File with Diagnosis Rules

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Use this command to generate a baseline file using specified data, summary rule, and diagnosis rule files. This is useful for creating a new baseline from multiple machines' results.

```bash
sb result generate-baseline --data-file outputs/results-summary.jsonl --summary-rule-file summary-rule.yaml --diagnosis-rule-file diagnosis-rule.yaml
```

--------------------------------

### Generate Baseline for Deterministic Results

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/user-tutorial/benchmarks/model-benchmarks.md

After running benchmarks with deterministic flags, generate a baseline for comparison. This command uses the results file and a summary rule file.

```bash
sb result generate-baseline --data-file results.jsonl --summary-rule-file rules.yaml
```

--------------------------------

### Execute GPT2 Model Benchmark

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

Executes the GPT2 model benchmark using the SuperBench CLI. Use `--config-override` to specify benchmark configurations.

```bash
sb exec --config-override superbench.enable=["gpt2_models"]
```

--------------------------------

### Define Third-Party and NVCODEC Directories

Source: https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/micro_benchmarks/cuda_decode_performance/CMakeLists.txt

Sets variables for the locations of third-party video codec SDK samples, interfaces, utilities, and NvCodec directories.

```cmake
set(THIRD_PARTY_SAMPLE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../../../third_party/Video_Codec_SDK/Samples)
set(NVCODEC_PUBLIC_INTERFACE_DIR ${THIRD_PARTY_SAMPLE_DIR}/../Interface)
set(NVCODEC_UTILS_DIR ${THIRD_PARTY_SAMPLE_DIR}/Utils)
set(NV_CODEC_DIR ${THIRD_PARTY_SAMPLE_DIR}/NvCodec)
set(NV_DEC_DIR ${THIRD_PARTY_SAMPLE_DIR}/NvCodec/NvDecoder)
```

--------------------------------

### SB CLI Command Structure for generate-baseline

Source: https://github.com/microsoft/superbenchmark/blob/main/docs/cli.md

This shows the general structure for the 'sb result generate-baseline' command, including required and optional arguments. Use this as a template for generating baseline files.

```bash
sb result generate-baseline --data-file
                            --summary-rule-file
                            [--diagnosis-rule-file]
                            [--baseline-file]
                            [--decimal-place-value]
                            [--output-dir]
```