SecEval

https://github.com/xuanwuai/seceval
Admin
SecEval is a comprehensive benchmark with over 2,000 multiple-choice questions across 9...
Tokens:8,086
Snippets:76
Trust Score:7.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Install necessary Python packages for SecEval evaluation using pip.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Evaluation Dependencies

Source: https://context7.com/xuanwuai/seceval/llms.txt

Install all required Python packages for the SecEval evaluation framework using pip. The `requirements.txt` file lists essential libraries such as transformers, langchain, and PyTorch.

```bash
pip install -r requirements.txt
```

```text
# requirements.txt contents:
# transformers==4.36.*
# langchain
# einops
# protobuf
# scipy
# sentencepiece
# torch
# flash_attn==2.3.*
# python-dotenv
# transformers_stream_generator
# tiktoken
```

--------------------------------

### Evaluate Chat-Optimized Models

Source: https://context7.com/xuanwuai/seceval/llms.txt

Use the '-c' flag to evaluate chat-optimized models using a conversation format with system instructions and few-shot examples. The example shows evaluation against an Azure endpoint.

```bash
python eval.py \
    -d questions.json \
    -B azure \
    -m gpt-4-turbo \
    -o ./results \
    -c \
    -b 5
```

--------------------------------

### Python API: Load SecEval Dataset

Source: https://context7.com/xuanwuai/seceval/llms.txt

Programmatically load the SecEval dataset from a JSON file using this Python function. It includes examples of inspecting and filtering the loaded dataset.

```python
import json
from typing import List, Dict, Any

def load_dataset(dataset_path: str) -> List[Dict[str, Any]]:
    """Load SecEval dataset from JSON file."""
    with open(dataset_path, "r") as f:
        dataset = json.load(f)
    return dataset

# Load and inspect the dataset
dataset = load_dataset("questions.json")
print(f"Total questions: {len(dataset)}")

# Filter by topic
web_security_questions = [q for q in dataset if "WebSecurity" in q["topics"]]
print(f"Web Security questions: {len(web_security_questions)}")

# Get question distribution by source
sources = {}
for q in dataset:
    sources[q["source"]] = sources.get(q["source"], 0) + 1
print(f"Questions by source: {sources}")
```

--------------------------------

### Azure OpenAI Backend Evaluation

Source: https://context7.com/xuanwuai/seceval/llms.txt

Evaluate models on Azure OpenAI service by setting API key and endpoint environment variables. This example shows how to run evaluations for GPT-4 Turbo and GPT-3.5 Turbo, enabling chat mode and specifying batch sizes.

```bash
# Set Azure OpenAI credentials
export OPENAI_API_KEY="your-azure-openai-api-key"
export OPENAI_API_ENDPOINT="https://your-resource.openai.azure.com/"
```

```bash
# Evaluate GPT-4 Turbo on SecEval dataset
python eval.py \
    -d questions.json \
    -B azure \
    -m gpt-4-turbo \
    -o ./results \
    -c \
    -b 5
```

```bash
# Evaluate GPT-3.5 Turbo
python eval.py \
    -d questions.json \
    -B azure \
    -m gpt-35-turbo \
    -o ./results \
    -c \
    -b 10
```

--------------------------------

### Local Hugging Face Model Evaluation

Source: https://context7.com/xuanwuai/seceval/llms.txt

Evaluate locally downloaded Hugging Face models by setting the `LOCAL_HF_MODEL_DIR` environment variable. This example demonstrates evaluating a local Llama-2 model with a batch size of 1.

```bash
# Set the local model directory
export LOCAL_HF_MODEL_DIR="/path/to/your/models"
```

```bash
# Evaluate a local Llama-2 model
python eval.py \
    -d questions.json \
    -B local_hf \
    -m Llama-2-7b-hf \
    -o ./results \
    -b 1
```

--------------------------------

### SecEval Results Output Format

Source: https://context7.com/xuanwuai/seceval/llms.txt

The evaluation produces a JSON file containing detailed scores by topic and individual question results. This example shows the structure of the output JSON.

```json
{
    "score_fraction": {
        "SystemSecurity": "780/1065",
        "ApplicationSecurity": "608/808",
        "PenTest": "380/475",
        "MemorySafety": "34/48",
        "NetworkSecurity": "174/230",
        "WebSecurity": "635/773",
        "Vulnerability": "254/334",
        "SoftwareSecurity": "170/232",
        "Cryptography": "9/14",
        "Overall": "1681/2126"
    },
    "score_float": {
        "SystemSecurity": 73.24,
        "ApplicationSecurity": 75.25,
        "PenTest": 80.00,
        "MemorySafety": 70.83,
        "NetworkSecurity": 75.65,
        "WebSecurity": 82.15,
        "Vulnerability": 76.05,
        "SoftwareSecurity": 73.28,
        "Cryptography": 64.29,
        "Overall": 79.07
    },
    "detail": [
        {
            "id": "question-uuid",
            "question": "...",
            "choices": ["A: ...", "B: ...", "C: ...", "D: ..."],
            "answer": "ABC",
            "llm_input": "...",
            "llm_output": "Answer: ABC",
            "llm_answer": "ABC",
            "score": 1
        }
    ]
}
```

--------------------------------

### SecEval Dataset JSON Format Example

Source: https://context7.com/xuanwuai/seceval/llms.txt

Each question in the SecEval dataset is structured in JSON, including the question text, choices, correct answer, security topics, and source reference. This format is used for evaluating LLMs on cybersecurity knowledge.

```json
{
    "id": "282cd7f6-86f9-4174-95c2-df26b2b6395e",
    "source": "android_sec_doc",
    "question": "You are tasked with designing a secure storage system for an Android device's hardware identifiers as part of an ID attestation implementation. Which of the following properties are essential for ensuring the system's integrity and security?",
    "choices": [
        "A: The storage must contain the original identifiers to enable the TEE to verify their authenticity during attestation.",
        "B: The storage should be tamper-evident to ensure any modification is detectable, rendering the attestation invalid.",
        "C: The `destroyAttestationIds()` method should be able to restore the identifier-derived data after a factory reset.",
        "D: RMA facilities must not have the ability to regenerate hardware identifier-derived data to prevent unauthorized attestation."
    ],
    "answer": "B",
    "topics": [
        "SystemSecurity",
        "Vulnerability"
    ],
    "keyword": "Tamper-evident"
}
```

--------------------------------

### SecEval Evaluation CLI Basic Usage

Source: https://context7.com/xuanwuai/seceval/llms.txt

The evaluation CLI allows running SecEval benchmarks with specified dataset, backend, model, and output directory. Key options include dataset file path, backend type, model name(s), and output directory.

```bash
# Basic usage syntax
python eval.py -d <dataset_file> -B <backend> -m <model_name> -o <output_dir>
```

```text
# Available options:
#   -d, --dataset_file    Path to the SecEval questions.json file (required)
#   -B, --backend         Model backend: remote_hf, azure, textgen, local_hf (required)
#   -m, --models          Model name(s) to evaluate (required, supports multiple)
#   -o, --output_dir      Output directory for results (default: /tmp)
#   -b, --batch_size      Batch size for inference (default: 1)
#   -c, --chat            Enable chat model evaluation mode
```

--------------------------------

### Python API: Initialize Azure OpenAI

Source: https://context7.com/xuanwuai/seceval/llms.txt

Initialize the Azure OpenAI backend for programmatic evaluation using LangChain.

```APIDOC
## Python API: Initialize Azure OpenAI

### Description
Initialize the Azure OpenAI backend for programmatic evaluation using LangChain.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **model_id** (str) - Required - The identifier for the Azure OpenAI model to use (e.g., 'gpt-4-turbo').

### Environment Variables Required
- **OPENAI_API_ENDPOINT**: The endpoint URL for your Azure OpenAI service.
- **OPENAI_API_KEY**: The API key for your Azure OpenAI service.
- **OPENAI_API_TYPE** (Optional): The type of OpenAI API, defaults to 'azure'.

### Request Example
```python
import os

os.environ["OPENAI_API_ENDPOINT"] = "https://your-resource.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "your-api-key"
llm = init_azure_openai_llm("gpt-4-turbo")
```

### Response
#### Success Response (AzureChatOpenAI)
- Returns an initialized `AzureChatOpenAI` object configured with the provided model ID and environment variables.

#### Response Example
```python
# llm object is now ready to be used for chat completions
```
```

--------------------------------

### Python API: Initialize Hugging Face Model

Source: https://context7.com/xuanwuai/seceval/llms.txt

Initialize a local Hugging Face model with GPU acceleration for evaluation.

```APIDOC
## Python API: Initialize Hugging Face Model

### Description
Initialize a local Hugging Face model with GPU acceleration for evaluation.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **model_id** (str) - Required - The identifier for the Hugging Face model (e.g., '/models/Yi-6B' for local or '01-ai/Yi-6B' for remote).

### Request Example
```python
# Initialize a local model
llm = init_hf_llm("/models/Yi-6B")

# Or with remote Hugging Face model
llm = init_hf_llm("01-ai/Yi-6B")
```

### Response
#### Success Response (HuggingFacePipeline)
- Returns an initialized `HuggingFacePipeline` object configured for text generation with the specified model.

#### Response Example
```python
# llm object is now ready to be used for text generation
```
```

--------------------------------

### Initialize Azure OpenAI LLM

Source: https://context7.com/xuanwuai/seceval/llms.txt

Initializes the Azure OpenAI backend for programmatic evaluation using LangChain. Ensure OPENAI_API_ENDPOINT and OPENAI_API_KEY environment variables are set.

```python
import os
from langchain.chat_models import AzureChatOpenAI

def init_azure_openai_llm(model_id: str) -> AzureChatOpenAI:
    """Initialize Azure OpenAI chat model."""
    if os.environ.get("OPENAI_API_ENDPOINT") is None:
        raise RuntimeError("Please set OPENAI_API_ENDPOINT")
    if os.environ.get("OPENAI_API_KEY") is None:
        raise RuntimeError("Please set OPENAI_API_KEY")

    azure_params = {
        "model": model_id,
        "openai_api_base": os.environ["OPENAI_API_ENDPOINT"],
        "openai_api_key": os.environ["OPENAI_API_KEY"],
        "openai_api_type": os.environ.get("OPENAI_API_TYPE", "azure"),
        "openai_api_version": "2023-07-01-preview",
    }
    return AzureChatOpenAI(**azure_params)

# Initialize GPT-4 for evaluation
os.environ["OPENAI_API_ENDPOINT"] = "https://your-resource.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "your-api-key"
llm = init_azure_openai_llm("gpt-4-turbo")
```

--------------------------------

### Download SecEval Dataset

Source: https://context7.com/xuanwuai/seceval/llms.txt

Download the SecEval dataset JSON file using wget or load it directly into a Python environment using the Hugging Face datasets library.

```bash
# Download the dataset JSON file
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/questions.json
```

```python
# Or load directly using Python with Hugging Face datasets library
from datasets import load_dataset
dataset = load_dataset("XuanwuAI/SecEval")
```

--------------------------------

### Initialize Hugging Face LLM

Source: https://context7.com/xuanwuai/seceval/llms.txt

Initializes a local or remote Hugging Face model with GPU acceleration for evaluation. Uses `torch.bfloat16` for model precision if the GPU supports it.

```python
import torch
from langchain.llms.huggingface_pipeline import HuggingFacePipeline

def init_hf_llm(model_id: str) -> HuggingFacePipeline:
    """Initialize Hugging Face model pipeline."""
    flash_attn_enable = torch.cuda.get_device_capability()[0] >= 8

    llm = HuggingFacePipeline.from_model_id(
        model_id=model_id,
        task="text-generation",
        pipeline_kwargs={"max_new_tokens": 5},
        device=0,
        model_kwargs={
            "trust_remote_code": True,
            "torch_dtype": torch.bfloat16
        },
    )
    return llm

# Initialize a local model
llm = init_hf_llm("/models/Yi-6B")

# Or with remote Hugging Face model
llm = init_hf_llm("01-ai/Yi-6B")
```

--------------------------------

### Download SecEval Dataset

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Download the SecEval dataset in JSON format using wget.

```bash
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/questions.json
```

--------------------------------

### Configure Azure OpenAI Model Backend

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Set environment variables for API key and endpoint for Azure OpenAI models.

```bash
export OPENAI_API_KEY=<your openai api key>
export OPENAI_API_ENDPOINT=<your openai api endpoint>
```

--------------------------------

### Evaluate Local Models Sequentially

Source: https://context7.com/xuanwuai/seceval/llms.txt

Use this command to evaluate multiple local models in sequence. Specify the dataset, backend, model names, and output directory.

```bash
python eval.py \
    -d questions.json \
    -B local_hf \
    -m Yi-6B Mistral-7B-v0.1 Qwen-7B \
    -o ./results
```

--------------------------------

### Configure Local Huggingface Model Backend

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Set the environment variable for the local directory containing Huggingface models.

```bash
export HF_LOCAL_MODEL_DIR=<path to local huggingface model>
```

--------------------------------

### Configure text-generation-webui Backend

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Set the environment variable for the API URL of the text-generation-webui service.

```bash
export TEXTGEN_MODEL_URL=<your textgenerationwebui api url>
```

--------------------------------

### SecEval Evaluation CLI Usage

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Command-line interface for running SecEval evaluations. Specify dataset, backend, models, and output directory.

```bash
usage: eval.py [-h] [-o OUTPUT_DIR] -d DATASET_FILE [-c] [-b BATCH_SIZE] -B {remote_hf,azure,textgen,local_hf} -m MODELS [MODELS ...]

SecEval Evaluation CLI

options:
  -h, --help            show this help message and exit
  -o OUTPUT_DIR, --output_dir OUTPUT_DIR
                        Specify the output directory.
  -d DATASET_FILE, --dataset_file DATASET_FILE
                        Specify the dataset file to evaluate on.
  -c, --chat            Evaluate on chat model.
  -b BATCH_SIZE, --batch_size BATCH_SIZE
                        Specify the batch size.
  -B {remote_hf,azure,textgen,local_hf}, --backend {remote_hf,azure,textgen,local_hf}
                        Specify the llm type. remote_hf: remote huggingface model backed, azure: azure openai model, textgen: textgen backend, local_hf: local huggingface model backed
  -m MODELS [MODELS ...], --models MODELS [MODELS ...]
                        Specify the models.
```

--------------------------------

### GPT-3 2-Shot QCM→A Evaluation Results

Source: https://github.com/xuanwuai/seceval/blob/main/docs/javascript/print_result.ipynb

Presents the evaluation scores for GPT-3 using a 2-shot approach for the QCM→A task. The output is formatted as HTML table data.

```python
# GPT-3 2-shot QCM→A
result = "74.64 69.74 76.00 74.44 67.28 77.42 76.80 68.89 73.97"
print_result(result)
```

--------------------------------

### GPT-3 Zero-Shot QCM→A Evaluation Results

Source: https://github.com/xuanwuai/seceval/blob/main/docs/javascript/print_result.ipynb

Displays the evaluation results for GPT-3 in a zero-shot setting for the QCM→A task. The scores are formatted for HTML table display.

```python
# GPT-3 zero-shot QCM→A
result = "75.04 66.59 78.00 74.24 65.74 79.58 76.36 69.87 74.04"
print_result(result)
```

--------------------------------

### Evaluate Models via Text-Generation-WebUI

Source: https://context7.com/xuanwuai/seceval/llms.txt

Evaluate models deployed via text-generation-webui. Configure the API endpoint using the TEXTGEN_MODEL_URL environment variable and specify the 'textgen' backend.

```bash
export TEXTGEN_MODEL_URL="http://localhost:5000/api/v1/generate"

python eval.py \
    -d questions.json \
    -B textgen \
    -m "local-model" \
    -o ./results \
    -b 1
```

--------------------------------

### GPT-3 (CoT) 2-Shot QCM→ALE Evaluation Results

Source: https://github.com/xuanwuai/seceval/blob/main/docs/javascript/print_result.ipynb

Displays evaluation scores for GPT-3 using Chain-of-Thought (CoT) and a 2-shot approach for the QCM→ALE task. The results are formatted into HTML table data.

```python
# GPT-3 (CoT) 2-shot QCM→ALE
result = "75.44 70.87 78.09 74.68 67.43 79.93 78.23 69.68 75.17"
print_result(result)
```

--------------------------------

### Run Evaluation on Base Model

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Execute the evaluation script for a base model, specifying dataset, backend, model name, and output directory.

```bash
python eval.py -d <path to dataset file> -B <backend> -m <model name> -o <output directory>
```

--------------------------------

### Evaluate Hugging Face Hub Models

Source: https://context7.com/xuanwuai/seceval/llms.txt

Evaluate models hosted on Hugging Face Hub by specifying the model identifier directly. This command uses the 'remote_hf' backend.

```bash
python eval.py \
    -d questions.json \
    -B remote_hf \
    -m "01-ai/Yi-6B" \
    -o ./results
```

```bash
python eval.py \
    -d questions.json \
    -B remote_hf \
    -m "microsoft/Orca-2-7b" \
    -o ./results \
    -b 2
```

--------------------------------

### Run Evaluation on Chat Model

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

Execute the evaluation script for a chat model, including the -c flag for chat mode.

```bash
python eval.py -d <path to dataset file> -B <backend> -m <model name> -o <output directory> -c
```

--------------------------------

### GPT-3 (CoT) 2-Shot QCM→AE Evaluation Results

Source: https://github.com/xuanwuai/seceval/blob/main/docs/javascript/print_result.ipynb

Shows the evaluation results for GPT-3 with Chain-of-Thought (CoT) and a 2-shot setting for the QCM→AE task. Scores are formatted for HTML table display.

```python
# GPT-3 (CoT) 2-shot QCM→AE
result = "76.60 65.92 77.55 75.51 66.09 79.58 78.49 67.63 74.61"
print_result(result)
```

--------------------------------

### SecEval Topic Distribution and Data Sources

Source: https://context7.com/xuanwuai/seceval/llms.txt

Defines the distribution of cybersecurity topics covered in the SecEval dataset and lists the data sources used for its creation.

```python
# SecEval Topic Distribution
topic_distribution = {
    "SystemSecurity": 1065,      # OS-level security, platform security
    "ApplicationSecurity": 808,  # App vulnerabilities, secure coding
    "PenTest": 475,              # Penetration testing, MITRE ATT&CK
    "WebSecurity": 773,          # Web vulnerabilities, OWASP
    "NetworkSecurity": 230,      # Network protocols, attacks
    "Vulnerability": 334,        # CWE, vulnerability types
    "SoftwareSecurity": 232,     # SDLC, secure development
    "MemorySafety": 48,          # Buffer overflows, memory corruption
    "Cryptography": 14,          # Encryption, cryptographic protocols
}

# Total: 2126 questions (some questions have multiple topics)

# Data sources include:
sources = [
    "cs161_textbook",      # UC Berkeley Computer Security course
    "mit_6858",            # MIT Computer Systems Security
    "android_sec_doc",     # Android Security Documentation
    "apple_platform_sec",  # Apple Platform Security
    "windows_sec_doc",     # Windows Security Documentation
    "owasp_wstg",          # OWASP Web Security Testing Guide
    "owasp_mastg",         # OWASP Mobile App Security Testing Guide
    "cwe",                 # Common Weakness Enumeration
    "mitre_attack",        # MITRE ATT&CK Framework
    "mitre_d3fend",        # MITRE D3FEND Framework
    "mozilla_web_sec",     # Mozilla Web Security Guidelines
]
```

--------------------------------

### Random Chance Evaluation Results

Source: https://github.com/xuanwuai/seceval/blob/main/docs/javascript/print_result.ipynb

Demonstrates the output format for random chance performance on the Seceval task. The scores are processed by the print_result function.

```python
# Random chance
result = "40.28 46.13 29.25 47.45 40.08 33.66 39.35 40.67 39.83"
print_result(result)
```

--------------------------------

### Run Batch Inference with Async Processing

Source: https://context7.com/xuanwuai/seceval/llms.txt

Execute batch inference on a dataset using async processing for efficient evaluation. This function handles both chat and non-chat modes, processes LLM outputs, and scores the answers.

```python
import asyncio
import re
from langchain.schema.messages import SystemMessage, HumanMessage, AIMessage
from langchain.schema.language_model import BaseLanguageModel
from typing import List, Dict, Any

instruction = "Below are multiple-choice questions concerning cybersecurity. Please select the correct answers and respond with the letters ABCD only."

chat_few_shot = [
    HumanMessage(content="Question: Which mitigation prevent stack overflow bug? A: Stack Canary. B: ALSR. C: CFI. D: Code Signing."),
    AIMessage(content="Answer: ABC"),
]

async def batch_inference_dataset(
    llm: BaseLanguageModel,
    batch: List[Dict[str, Any]],
    chat: bool = False
) -> List[Dict[str, Any]]:
    """Run batch inference on dataset questions."""
    llm_inputs = []

    for dataset_row in batch:
        question_text = "Question: " + dataset_row["question"] + " ".join(dataset_row["choices"])
        question_text = question_text.replace("\n", " ")

        if chat:
            llm_input = (
                [SystemMessage(content=instruction)]
                + chat_few_shot
                + [HumanMessage(content=question_text)]
            )
        else:
            llm_input = instruction + question_text + "\n"
        llm_inputs.append(llm_input)

    llm_outputs = await llm.abatch(llm_inputs)

    for idx, llm_output in enumerate(llm_outputs):
        if type(llm_output) == AIMessage:
            llm_output = llm_output.content
        if "Answer:" in llm_output:
            llm_output = llm_output.replace("Answer:", "")

        batch[idx]["llm_output"] = llm_output
        batch[idx]["llm_answer"] = "".join(sorted(list(set(re.findall(r"[A-D]", llm_output))))))
        batch[idx]["score"] = int(batch[idx]["llm_answer"].lower() == batch[idx]["answer"].lower())

    return batch

# Run evaluation
loop = asyncio.get_event_loop()
results = loop.run_until_complete(batch_inference_dataset(llm, dataset[:10], chat=True))
```

--------------------------------

### Human Performance Evaluation Results

Source: https://github.com/xuanwuai/seceval/blob/main/docs/javascript/print_result.ipynb

Shows the evaluation scores representing human performance on the Seceval task. These scores are formatted into HTML table cells.

```python
# Human
result = "90.23 84.97 87.48 89.60 87.50 88.10 91.59 82.42 88.40"
print_result(result)
```

--------------------------------

### Calculate Scores by Topic

Source: https://context7.com/xuanwuai/seceval/llms.txt

Calculates evaluation scores broken down by security topic from a dataset of inference results. Requires a list of dictionaries, where each dictionary contains 'topics' (a list of strings) and 'score' (an integer).

```python
from typing import List, Dict, Any, Tuple

def count_score_by_topic(dataset: List[Dict[str, Any]]) -> Tuple[Dict, Dict]:
    """Calculate scores by topic from evaluated dataset."""
    score_by_topic = {}
    total_score_by_topic = {}
    score = 0

    for dataset_row in dataset:
        for topic in dataset_row["topics"]:
            if topic not in score_by_topic:
                score_by_topic[topic] = 0
                total_score_by_topic[topic] = 0
            score_by_topic[topic] += dataset_row["score"]
            total_score_by_topic[topic] += 1
        score += dataset_row["score"]

    # Calculate fractions and percentages
    score_fraction = {
        k: f"{v}/{total_score_by_topic[k]}"
        for k, v in score_by_topic.items()
    }
    score_float = {
        k: round(100 * float(v) / float(total_score_by_topic[k]), 4)
        for k, v in score_by_topic.items()
    }
    score_float["Overall"] = round(100 * float(score) / float(len(dataset)), 4)
    score_fraction["Overall"] = f"{score}/{len(dataset)}"

    return score_fraction, score_float

# Example usage with evaluated results
results = [
    {"topics": ["WebSecurity"], "score": 1},
    {"topics": ["WebSecurity", "PenTest"], "score": 0},
    {"topics": ["SystemSecurity"], "score": 1},
]
fractions, percentages = count_score_by_topic(results)
print(f"Scores: {percentages}")
# Output: {'WebSecurity': 50.0, 'PenTest': 0.0, 'SystemSecurity': 100.0, 'Overall': 66.67}
```

--------------------------------

### Python API: Score Calculation

Source: https://context7.com/xuanwuai/seceval/llms.txt

Calculate evaluation scores broken down by security topic from the inference results.

```APIDOC
## Python API: Score Calculation

### Description
Calculate evaluation scores broken down by security topic from the inference results.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **dataset** (List[Dict[str, Any]]) - Required - A list of dictionaries, where each dictionary represents a dataset row and must contain 'topics' (List[str]) and 'score' (int or float).

### Request Example
```python
results = [
    {"topics": ["WebSecurity"], "score": 1},
    {"topics": ["WebSecurity", "PenTest"], "score": 0},
    {"topics": ["SystemSecurity"], "score": 1},
]
```

### Response
#### Success Response (Tuple[Dict, Dict])
- **score_fraction** (Dict) - A dictionary where keys are security topics and values are strings representing the fraction of scores (e.g., '1/2').
- **score_float** (Dict) - A dictionary where keys are security topics and values are floats representing the percentage of scores (e.g., 50.0).

#### Response Example
```python
# Assuming the input 'results' from the request example
# fractions, percentages = count_score_by_topic(results)
# print(f"Scores: {percentages}")
# Output: {'WebSecurity': 50.0, 'PenTest': 0.0, 'SystemSecurity': 100.0, 'Overall': 66.67}
```
```

--------------------------------

### Format Evaluation Results to HTML Table Cells

Source: https://github.com/xuanwuai/seceval/blob/main/docs/javascript/print_result.ipynb

This Python function takes a space-separated string of numbers and prints each number enclosed in HTML table data (<td>) tags. It's used to format the raw evaluation scores for display.

```python
def print_result(result):
    nums = result.split(" ")
    for num in nums:
        print(f"<td>{num}</td>")
```

--------------------------------

### SecEval Results JSON Structure

Source: https://github.com/xuanwuai/seceval/blob/main/eval/README.md

The structure of the JSON object containing evaluation results, including score fractions, float scores, and detailed results.

```json
{
    "score_fraction": {
        "topic_name": "topic_score_fraction",
    },
    "score_float": {
        "topic_name": "topic_score_float",
    },
    "detail": ["detail1", "detail2",]
}
```
Tokens:
Raw