Verifiers

https://verifiers.readthedocs.io/en/latest/
Admin
Verifiers is a Python library for creating environments to train and evaluate Large Language Models...
Tokens:1,082
Snippets:17
Trust Score:8.3
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install and Configure Prime CLI

Source: https://verifiers.readthedocs.io/en/latest

Installs the uv package manager and the prime CLI tool, then logs into the Prime Intellect platform. Ensure uv is installed first.

```bash
# install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# install the prime CLI
uv tool install prime
# log in to the Prime Intellect platform
prime login
```

--------------------------------

### Install Environment from Hub

Source: https://verifiers.readthedocs.io/en/latest

Installs a pre-built environment directly from the Environments Hub. This is a convenient way to use established environments without local setup.

```bash
prime env install primeintellect/math-python
```

--------------------------------

### Install Local Environment Module

Source: https://verifiers.readthedocs.io/en/latest

Installs a custom environment module from a local directory into your project. This makes the environment available for use in evaluations and training.

```bash
prime env install my-env # installs from ./environments/my_env
```

--------------------------------

### Initialize New Environment Template

Source: https://verifiers.readthedocs.io/en/latest

Creates a new, basic environment template module within the './environments/' directory. This is the starting point for building custom environments.

```bash
prime env init my-env # creates a new template in ./environments/my_env
```

--------------------------------

### Set Up New Development Workspace

Source: https://verifiers.readthedocs.io/en/latest

Initializes a new workspace for developing environments. This command sets up a Python project, installs the 'verifiers' library, and creates a recommended workspace structure with starter configuration files.

```bash
# ~/dev/my-lab
prime lab setup
```

--------------------------------

### Run Evaluation from Environments Hub

Source: https://verifiers.readthedocs.io/en/latest

Executes an evaluation using an environment directly from the Environments Hub. This bypasses the need to install the environment locally first.

```bash
prime eval run primeintellect/math-python
```

--------------------------------

### Publish Environment to Hub

Source: https://verifiers.readthedocs.io/en/latest

Uploads a local environment to the Environments Hub, making it accessible for others to install and use. This command is equivalent to using the --path flag with the local directory.

```bash
prime env push my-env # equivalent to --path ./environments/my_env
```

--------------------------------

### Define Environment Load Function

Source: https://verifiers.readthedocs.io/en/latest

Example implementation of an environment module's main file. It defines a `load_environment` function that initializes and returns a `vf.Environment` instance, configured with a dataset and a custom rubric for scoring completions.

```python
# my_env.py
import verifiers as vf

def load_environment(dataset_name: str = 'gsm8k') -> vf.Environment:
    dataset = vf.load_example_dataset(dataset_name) # 'question'
    async def correct_answer(completion, answer) -> float:
        completion_ans = completion[-1]['content']
        return 1.0 if completion_ans == answer else 0.0
    rubric = Rubric(funcs=[correct_answer])
    env = vf.SingleTurnEnv(dataset=dataset, rubric=rubric)
    return env
```

--------------------------------

### Add Verifiers to Existing Project

Source: https://verifiers.readthedocs.io/en/latest

Adds the 'verifiers' library to an existing Python project and sets up the workspace structure without reinstalling dependencies.

```bash
uv add verifiers && prime lab setup --skip-install
```

--------------------------------

### Run Local Evaluation

Source: https://verifiers.readthedocs.io/en/latest

Executes a local evaluation of a specified model using a configured environment. Evaluation results are saved locally. Ensure your API endpoints are configured in './configs/endpoints.toml' if not using default Prime Inference.

```bash
prime eval run my-env -m openai/gpt-5-nano # run and save eval results locally
```

--------------------------------

### View Local Evaluation Results

Source: https://verifiers.readthedocs.io/en/latest

Launches a terminal UI to browse and analyze local evaluation results. The TUI allows navigation through environments, models, and individual run details, with options to expand history and copy data.

```bash
prime eval tui
```
Tokens:
Raw