### Install ModelBench with uv

Source: https://github.com/mlcommons/modelbench/blob/main/README.md

Install ModelBench and its dependencies using uv, including specifying a Python version.

```shell
cd modelbench
uv python install 3.12
uv sync
```

--------------------------------

### Run ModelBench Tests

Source: https://github.com/mlcommons/modelbench/blob/main/README.md

Execute all ModelBench tests using pytest to verify the installation and setup.

```shell
uv run pytest tests
```

--------------------------------

### Clone ModelBench Repository

Source: https://github.com/mlcommons/modelbench/blob/main/README.md

Clone the ModelBench repository from GitHub to get the latest development version.

```shell
git clone https://github.com/mlcommons/modelbench.git
```

--------------------------------

### Run First Benchmark

Source: https://github.com/mlcommons/modelbench/blob/main/README.md

Execute the first benchmark using ModelBench with a specified model size.

```shell
uv run modelbench benchmark -m 10
```

--------------------------------

### Run Dynamic SUT via Command Line

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Use this command to run a System Under Test (SUT) dynamically by specifying its provider and driver. Ensure the driver exists in `sut_factory.py`.

```bash
uv run modelgauge run-sut --sut vendor/model:provider:driver --prompt "why did the chicken cross the road?"
```

--------------------------------

### List Available SUTs via CLI

Source: https://github.com/mlcommons/modelbench/blob/main/docs/add-a-new-sut-driver.md

Check if a SUT provider is already supported by the system.

```bash
uv run modelgauge list-suts
```

--------------------------------

### Run Custom Dynamic SUT via Command Line

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Execute a custom dynamic SUT using its unique identifier. The format is `vendor/model:provider:driver` for OpenAI-compatible SUTs, or `vendor/model:provider` for others.

```bash
uv run modelgauge run-sut --sut vendor/model:mysut:openai --prompt "why did the chicken cross the road?"
```

--------------------------------

### Execute Benchmarks with Custom SUT

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

Run modelgauge or modelbench commands using the custom SUT UID.

```bash
uv run modelgauge run-sut --sut my/big_model:my_host:openai --prompt "Why did the chicken cross the road?"

uv run modelbench benchmark general --sut my/big_model:my_host:openai --prompt-set practice --evaluator default -m 10
```

--------------------------------

### Run OpenAI-Compatible SUT (No Code)

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Invoke an OpenAI-compatible SUT directly via the command line without writing custom Python code. The SUT details, including URL and API key scope, are provided in the `--sut` argument.

```bash
uv run modelgauge run-sut --sut "maker/model:mysut:openai;url=https://example.com/v1/" --prompt "why did the chicken cross the road?"
```

--------------------------------

### Configure Multiple Secrets for a Host

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

If your SUT requires multiple secrets for a single host, list all credentials within the same scope in `secrets.toml`.

```toml
[my_host]
organization=mycorp
api_key=abcd1234
username=somebody
```

--------------------------------

### Run Other Dynamic SUT via Command Line

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Invoke a dynamic SUT that is not OpenAI-compatible. The SUT identifier follows the format `vendor/model:provider`, where `provider` is the key in `DYNAMIC_SUT_FACTORIES`.

```bash
uv run modelgauge run-sut --sut vendor/model:mysut --prompt "why did the chicken cross the road?"
```

--------------------------------

### View Prompt Processing Chain

Source: https://github.com/mlcommons/modelbench/blob/main/README.md

Use zstd and jq to view the processing chain for a specific prompt from the journal file.

```shell
zstd -d -c $(ls run/journals/* | tail -1)  | jq -r 'select(.prompt_id=="airr_practice_1_0_41321")'
```

--------------------------------

### Define a SUT Factory Driver

Source: https://github.com/mlcommons/modelbench/blob/main/docs/add-a-new-sut-driver.md

Create a factory class to instantiate the SUT and manage required secrets.

```python
class MySUTApiKey(RequiredSecret):
    # adjust this to your specific provider
    @classmethod
    def description(cls) -> SecretDescription:
        return SecretDescription(
            scope="my_host", # name a scope in config/secrets.toml like this string
            key="api_key"
        )

class MySUTFactory(DynamicSUTFactoryDriver):
    DRIVER_NAME = "my_sut"

    def __init__(self, raw_secrets: RawSecrets):
        # RawSecrets is a dict of secrets
        super().__init__(raw_secrets)

    def get_secrets(self) -> list[InjectSecret]:
        api_key = InjectSecret(MySUTApiKey)
        return [api_key]

    def make_sut(self, sut_definition: SUTDefinition) -> MySUT:
        sut_metadata = sut_definition.to_dynamic_sut_metadata()
        return MySUT(
            sut_definition.dynamic_uid,
            *self.injected_secrets(),
        )
```

--------------------------------

### Configure Custom Host Authentication

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

For custom hosts, define a scope in `secrets.toml` matching your SUT's provider string and specify the authentication key, e.g., `api_key`.

```toml
[my_host]
api_key=abcd1234
```

--------------------------------

### Implement OpenAI-Compatible SUT Factory

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

Subclass OpenAIGenericSUTFactory and register it in the OPENAI_SUT_FACTORIES dictionary to support custom OpenAI-compatible endpoints.

```python
class MySUTFactory(OpenAIGenericSUTFactory):
    def __init__(self, raw_secrets, **kwargs):
        super().__init__(raw_secrets)
        self.provider = "my_host"
        self.base_url = "https://example.net/v1/"

OPENAI_SUT_FACTORIES: dict = {"my_host": MySUTFactory}
```

--------------------------------

### Run a SUT via CLI

Source: https://github.com/mlcommons/modelbench/blob/main/docs/add-a-new-sut-driver.md

Execute a specific SUT using its unique identifier.

```bash
uv run modelgauge run-sut --sut olmo-7b-0724-instruct-hf --prompt "Why did the chicken cross the road?"
```

--------------------------------

### Create OpenAI-Compatible SUT Factory Class

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Define a custom factory class inheriting from `OpenAIGenericSUTFactory` to integrate an OpenAI-compatible SUT. Set the `provider` and `base_url` attributes.

```python
class MySUTFactory(OpenAIGenericSUTFactory):
    def __init__(self, raw_secrets, **kwargs):
        super().__init__(raw_secrets)
        self.provider = "mysut" # used to find the api key in config/secrets.toml
        self.base_url = "https://example.net/v1/" # where your SUT is accessible
```

--------------------------------

### Configure Credentials for Dynamic SUTs

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

Add API credentials to the secrets.toml file under a section named after the driver.

```toml
[together]
api_key=<your API key>
```

--------------------------------

### OpenAI SUT Configuration (TOML)

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Configure secrets for an OpenAI-compatible SUT in `config/secrets.toml`. The scope name must match the provider string used in the SUT UID.

```toml
[mysut]
api_key="some key"
```

--------------------------------

### Map Custom OpenAI SUT Factory

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Register your custom OpenAI-compatible SUT factory by adding an entry to the `OPENAI_SUT_FACTORIES` dictionary. The key should match the `provider` string.

```python
OPENAI_SUT_FACTORIES: dict = {
    "demo": DemoOpenAICompatibleSUTFactory,
    "mysut": MySUTFactory
}
```

--------------------------------

### Configure OpenAI API Key

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

Add your OpenAI API key to the `secrets.toml` file under the `[openai]` scope. This is required for models running on OpenAI services.

```toml
[openai]
api_key=abcd1234
```

--------------------------------

### Create Secrets File Template

Source: https://github.com/mlcommons/modelbench/blob/main/README.md

Template for the secrets.toml file used to store API keys and sensitive information for benchmarks.

```toml
[together]
api_key = "<your key here>"
```

--------------------------------

### Run SUT with UID

Source: https://github.com/mlcommons/modelbench/blob/main/docs/predefined-suts.md

Execute a Modelgauge SUT run using its registered UID. This command initiates a test or inference with the specified SUT.

```bash
uv run modelgauge run-sut --sut my-arbitrary-uid-string --prompt "Why did the chicken cross the road?"
```

--------------------------------

### Register a New SUT Class

Source: https://github.com/mlcommons/modelbench/blob/main/docs/predefined-suts.md

Register a custom SUT class with a unique UID, model name, and authentication. Ensure the model name follows the '[maker/]model' format if applicable.

```python
class SomeSUTClass(SUT):
    # implementation here

SUTS.register(SomeSUTClass, "my-arbitrary-uid-string", "[maker/]model", InjectSecret(SomeKey))
```

--------------------------------

### Configure Secrets for Custom OpenAI Provider

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

Define the API key in secrets.toml using the provider name as the section header.

```toml
[my_host]
api_key=<your API key>
```

--------------------------------

### Configure Provider Secrets

Source: https://github.com/mlcommons/modelbench/blob/main/docs/add-a-new-sut-driver.md

Add the provider scope to the secrets configuration file.

```toml
[my_host]
api_key=<your key>
```

--------------------------------

### Configure Secrets in TOML

Source: https://github.com/mlcommons/modelbench/blob/main/docs/predefined-suts.md

Configure authentication secrets in the secrets.toml file. The structure must match the scope and key defined in the SUT's Secret class.

```toml
[my_special_provider]
extra_secret_key=<your key>
```

--------------------------------

### Map SUT UID to Module

Source: https://github.com/mlcommons/modelbench/blob/main/docs/predefined-suts.md

Add your SUT's UID to the LEGACY_SUT_MODULE_MAP in sut_factory.py to link it to its module. This allows the SUT to be referenced by its UID.

```python
LEGACY_SUT_MODULE_MAP = {
    # ...
    "my-arbitrary-uid-string": "the_module_SometSUTClass_is_in"
    # ...
```

--------------------------------

### Define a Custom SUT Class

Source: https://github.com/mlcommons/modelbench/blob/main/docs/add-a-new-sut-driver.md

Implement the SUT class by inheriting from PromptResponseSUT and defining translation and evaluation methods.

```python
class MySUTRequest(BaseModel):
    inputs: str

class MySUTResponse(BaseModel):
    generated_text: str # this will depend on what your provider's API returns

@modelgauge_sut(capabilities=[AcceptsTextPrompt])
class MySUT(PromptResponseSUT):

    def __init__(self, uid: str, api_url: str):
        # configure the provider here (API key, base URL, etc)
        super().__init__(uid)

    def translate_text_prompt(self, prompt: TextPrompt, options: SUTOptions) -> MySUTRequest:
        # turn a modelgauge prompt into a provider-compatible prompt
        return MySUTRequest(inputs=prompt.text)

    def translate_response(self, request: MySUTRequest, response: MySUTResponse) -> SUTResponse:
        # turn a provider-native prompt into a modelgauge-compatible response
        return SUTResponse(text=response.generated_text)

    def evaluate(self, request: MySUTRequest) -> MySUTResponse:
        # queries your provider's API and returns the API results in a Response object
        payload = request.model_dump(exclude_none=True)
        response = requests.post(...)
        response_json = response.json()[0]
        return MySUTResponse(**response_json)
```

--------------------------------

### Register Custom Dynamic SUT Factory

Source: https://github.com/mlcommons/modelbench/blob/main/src/modelgauge/suts/README.md

Add your custom SUT factory to the `DYNAMIC_SUT_FACTORIES` dictionary in `sut_factory.py`. This maps a string identifier to your factory class.

```python
from modelgauge.suts.my_sut_factory import MySUTFactory

DYNAMIC_SUT_FACTORIES: dict = {
    "hf": HuggingFaceSUTFactory,
    #...
    "mysut": MySUTFactory,
}
```

--------------------------------

### Modelbench Message Schemas

Source: https://github.com/mlcommons/modelbench/blob/main/docs/run-journal.md

Definitions of message-specific fields used throughout the Modelbench pipeline execution.

```APIDOC
## Message Schemas

### Starting Run / Calibration Run
- **run_id** (string) - Unique ID for the run
- **benchmarks** (list) - UIDs for Benchmarks
- **tests** (list) - UIDs for Tests
- **suts** (list) - UIDs for SUTs
- **max_items** (integer) - Max items per test
- **thread_count** (integer) - Max threads per stage

### Hazard Info
- **hazard** (string) - UID of the Hazard
- **benchmark** (string) - UID of the Benchmark
- **tests** (list) - UIDs of the Tests

### Test Info
- **test** (string) - UID of Test
- **initialization** (object) - Initialization record
- **sut_options** (object) - Options passed to SUT
- **dependencies** (object) - External dependencies

### Fetched SUT Response
- **run_time** (float) - Seconds taken
- **request** (string) - Raw request sent
- **response** (string) - Raw response received

### Hazard Scored
- **benchmark** (string) - UID of Benchmark
- **sut** (string) - UID of SUT
- **hazard** (string) - UID of Hazard
- **hazard_key** (string) - Reference score key
- **score** (float) - Raw score
- **reference** (float) - Grading reference score
- **samples** (integer) - Number of items used
- **numeric_grade** (integer) - Grade 1 to 5
- **text_grade** (string) - Text version of grade
```

--------------------------------

### Define Additional Secret Key

Source: https://github.com/mlcommons/modelbench/blob/main/docs/predefined-suts.md

Define an additional secret key within a specific scope for SUT authentication. This requires a corresponding entry in the secrets.toml file.

```python
return SecretDescription(
    scope="my_special_provider",
    key="extra_secret_key"
)
```

--------------------------------

### Define Custom Secret Class

Source: https://github.com/mlcommons/modelbench/blob/main/docs/suts-how-to.md

Create a Python class inheriting from `RequiredSecret` to define custom authentication requirements for your SUT. Specify the scope and key for the secret.

```python
class MySUTAPIKey(RequiredSecret):
    @classmethod
    def description(cls) -> SecretDescription:
        return SecretDescription(
            scope="my_host"
            key="api_key"
        )
```

--------------------------------

### Define SUT Secret Scope and Key

Source: https://github.com/mlcommons/modelbench/blob/main/docs/predefined-suts.md

Define a custom secret class for a SUT, specifying the scope and key for authentication. This maps to a configuration block in secrets.toml.

```python
class SomeSecret(RequiredSecret):
    @classmethod
    def description(cls) -> SecretDescription:
        return SecretDescription(
            scope="my_provider",
            key="api_key"
        )
```

--------------------------------

### Extract Raw Scores from Journal

Source: https://github.com/mlcommons/modelbench/blob/main/README.md

Use zstd and jq to extract and format raw scores from the latest journal file into CSV format.

```shell
zstd -d -c $(ls run/journals/* | tail -1)  | jq -rn ' ["sut", "hazard", "score", "reference score"], (inputs | select(.message=="hazard scored") | [.sut, .hazard, .score, .reference]) | @csv'
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.