EvalHub

https://eval-hub.github.io/docs

Admin

EvalHub orchestrates reproducible LLM evaluations on Kubernetes — broad benchmark coverage, curated...

Tokens:421

Snippets:8

Trust Score:9.7

Update:3 months ago

Show doc for...

Tokens:

Raw

### Install EvalHub CLI

Source: https://eval-hub.github.io

Install the EvalHub SDK with CLI support using pip. This is the first step to using EvalHub.

```bash
pip install "eval-hub-sdk[cli]"
```

--------------------------------

### Evaluation Results Example

Source: https://eval-hub.github.io

Example output showing the status and key metrics of completed evaluation jobs.

```bash
# Job submitted: eval-a1b2c3d4
# ✓ completed  mmlu       acc       0.723
# ✓ completed  hellaswag  acc_norm  0.801
```

--------------------------------

### Configure EvalHub Connection

Source: https://eval-hub.github.io

Set the base URL and authentication token for connecting to your EvalHub cluster. This requires `kubectl` to obtain a token.

```bash
evalhub config set base_url https://evalhub.apps.cluster.example.com
evalhub config set token $(kubectl create token evalhub-sa -n evalhub)
```

--------------------------------

### Run an LLM Evaluation

Source: https://eval-hub.github.io

Execute an LLM evaluation using the EvalHub CLI. Specify the model URL, model name, and benchmark to run. The `--wait` flag ensures the command blocks until the job is complete.

```bash
evalhub eval run \
--model-url  https://llama3.example.com/v1 \
--model-name meta-llama/Llama-3.2-8B-Instruct \
--benchmark  mmlu --wait
```

Tokens:

Raw