Open Model Evaluation

https://github.com/zhengyanzhao1997/skillrouter
Admin
A project for evaluating open models, including data handling, metrics, and evaluation scripts.
Tokens:2,133
Snippets:35
Trust Score:7.9
License:MIT
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

Installs the necessary Python packages for SkillRouter. Assumes a CUDA-capable local machine.

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

--------------------------------

### Prediction Format Example

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/evaluation/README.md

This JSON object demonstrates the expected format for prediction files, mapping task IDs to lists of skill IDs.

```json
{
  "task_id_1": ["skill_id_a", "skill_id_b", "skill_id_c"],
  "task_id_2": ["skill_id_x", "skill_id_y", "skill_id_z"]
}
```

--------------------------------

### Run One-Command Evaluation (Make)

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

An alternative command to run the one-command evaluation using 'make'.

```bash
make eval-open-models
```

--------------------------------

### Repository Layout

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

Illustrates the directory structure of the SkillRouter project.

```text
.
├── README.md
├── assets/
│   └── readme/
├── data/
│   └── eval_core/
├── evaluation/
├── manifests/
├── scripts/
└── src/

```

--------------------------------

### Download Evaluation Data

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/data/README.md

Executes a script to download the full benchmark into the data/eval_core directory. Ensure you have the necessary permissions and disk space.

```bash
bash scripts/download_eval_data.sh
```

--------------------------------

### Run One-Command Evaluation

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

Executes the released 0.6B embedding and reranker models over the easy and hard benchmark tiers. Outputs are saved to specific directories.

```bash
bash scripts/evaluate_open_models.sh
```

--------------------------------

### Evaluate Open Models with Local Checkpoints

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

Runs the evaluation using local checkpoints for the embedding and reranker models by setting environment variables.

```bash
SKILLROUTER_EMB_MODEL_OR_PATH=/path/to/SkillRouter-Embedding-0.6B \
SKILLROUTER_RERANK_MODEL_OR_PATH=/path/to/SkillRouter-Reranker-0.6B \
bash scripts/evaluate_open_models.sh
```

--------------------------------

### End-To-End Pipeline Evaluation

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/evaluation/README.md

This bash script command initiates an end-to-end evaluation of public models across specified tiers (e.g., 'easy' and 'hard').

```bash
bash scripts/evaluate_open_models.sh --tiers easy hard
```

--------------------------------

### Evaluate Prediction File

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/data/eval_core/README.md

This command evaluates a prediction file against a specified tier of the benchmark. Ensure the predictions file path and tier are correct.

```bash
bash scripts/evaluate_predictions.sh \
  --predictions outputs/custom_eval/retrieval/easy.json \
  --tier easy
```

--------------------------------

### Run Custom Retrieval + Rerank Pipeline

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

Executes a custom retrieval and rerank pipeline using specified encoder and reranker models. Reranks top-20 candidates with 'flat-full' prompts by default.

```python
python3 -m src.run_open_model_eval \
  --data_root data/eval_core \
  --encoder_model_or_path /path/to/your/encoder \
  --reranker_model_or_path /path/to/your/reranker \
  --tiers easy hard \
  --output_dir outputs/custom_pipeline_eval
```

--------------------------------

### Citation for SkillRouter Paper

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

Provides the BibTeX entry for citing the SkillRouter paper.

```bibtex
@misc{zheng2026skillrouterskillroutingllm,
      title={SkillRouter: Skill Routing for LLM Agents at Scale},
      author={YanZhao Zheng and ZhenTao Zhang and Chao Ma and YuanQiang Yu and JiHuai Zhu and Yong Wu and Tianze Xu and Baohua Dong and Hangcheng Zhu and Ruohui Huang and Gang Yu},
      year={2026},
      eprint={2603.22455},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.22455},
}

```

--------------------------------

### Export Retrieval Predictions

Source: https://github.com/zhengyanzhao1997/skillrouter/blob/main/README.md

Exports retrieval predictions using a specified encoder model and data root. Keeps top-50 candidates by default.

```python
python3 -m src.export_retrieval \
  --encoder_model_or_path /path/to/your/encoder \
  --data_root data/eval_core \
  --output_dir outputs/custom_eval \
  --tiers easy hard
```
Tokens:
Raw