MultiKernelBench

https://github.com/wzzll123/multikernelbench

Admin

MultiKernelBench is a multi-platform benchmark for evaluating large language models on generating...

Tokens:784

Snippets:15

Trust Score:7

License:MIT

Update:1 month ago

Show doc for...

Tokens:

Raw

### Set up environment

Source: https://github.com/wzzll123/multikernelbench/blob/main/README.md

Commands to create a conda environment and install necessary dependencies for different hardware backends.

```bash
conda create --name multi-kernel-bench python=3.10
conda activate multi-kernel-bench
pip install -r requirements.txt

# For NPU users:
pip install torch-npu==2.7.1

# For Intel GPU (torch xpu) users:
pip install torch==2.7.0 --index-url https://download.pytorch.org/whl/xpu

# For Moore Threads GPU (MUSA) users:
# Install the torch_musa package matching your MUSA runtime.
```

--------------------------------

### Set API keys

Source: https://github.com/wzzll123/multikernelbench/blob/main/README.md

Environment variable exports for LLM service authentication.

```bash
export DEEPSEEK_API_KEY=<your deepseek api key>
export DASHSCOPE_API_KEY=<your aliyun api key>
export OPEN_ROUNTER_KEY=<your openrouter api key>
```

--------------------------------

### Evaluate kernels

Source: https://github.com/wzzll123/multikernelbench/blob/main/README.md

Command to run evaluation on generated kernels.

```bash
python evaluation.py --model deepseek-chat --language ascendc --strategy add_shot --categories activation
```

--------------------------------

### Generate kernels

Source: https://github.com/wzzll123/multikernelbench/blob/main/README.md

Command to trigger kernel generation using a specified LLM model and strategy.

```bash
python generate_and_write.py --model deepseek-chat --language ascendc --strategy add_shot --categories activation
```

--------------------------------

### Evaluate a single response

Source: https://github.com/wzzll123/multikernelbench/blob/main/README.md

Command to evaluate a specific generated response file directly.

```bash
python eval_single_runner.py \
  --input prompts/ascendc_direct_launch_model_relu.json \
  --op relu \
  --language ascendc_direct_launch \
  --result relu_result.json
```

Tokens:

Raw