FollowBench

https://github.com/yjiangcm/followbench
Admin
FollowBench is a multi-level, fine-grained constraint-following benchmark designed to systemically...
Tokens:847
Snippets:10
Trust Score:8.5
License:Apache-2.0
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install FollowBench Dependencies

Source: https://github.com/yjiangcm/followbench/blob/master/README.md

Sets up a conda environment and installs the required PyTorch and project dependencies.

```bash
conda create -n followbench python=3.10
conda activate followbench
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt
```

--------------------------------

### Cite FollowBench via BibTeX

Source: https://github.com/yjiangcm/followbench/blob/master/README.md

Use this BibTeX entry to cite the FollowBench paper in academic publications.

```bibtex
@inproceedings{jiang-etal-2024-followbench,
    title = "{F}ollow{B}ench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models",
    author = "Jiang, Yuxin  and
      Wang, Yufei  and
      Zeng, Xingshan  and
      Zhong, Wanjun  and
      Li, Liangyou  and
      Mi, Fei  and
      Shang, Lifeng  and
      Jiang, Xin  and
      Liu, Qun  and
      Wang, Wei",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.257",
    pages = "4667--4688",
}
```

--------------------------------

### Run Model Inference

Source: https://github.com/yjiangcm/followbench/blob/master/README.md

Executes model inference using the vLLM framework.

```bash
cd FollowBench/
python code/model_inference_vllm.py --model-path <model_name_or_path>
```

--------------------------------

### Perform LLM-based Evaluation

Source: https://github.com/yjiangcm/followbench/blob/master/README.md

Runs the LLM-based evaluation script using a specified model path and API key.

```bash
cd FollowBench/
python code/llm_eval.py --model_path <model_name_or_path> --api_key <your_own_gpt4_api_key>
```

--------------------------------

### Merge Evaluation Results

Source: https://github.com/yjiangcm/followbench/blob/master/README.md

Combines rule-based and LLM-based evaluation results and saves them to the evaluation_result folder.

```bash
cd FollowBench/
python code/eval.py --model_paths <a_list_of_evaluated_models>
```
Tokens:
Raw