AlignScore

https://github.com/yuh-zha/alignscore
Admin
AlignScore is an automatic metric for evaluating factual consistency between text pairs, trained on...
Tokens:5,389
Snippets:29
Trust Score:6.7
License:MIT
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Benchmark Dependencies

Source: https://github.com/yuh-zha/alignscore/blob/main/README.md

Installs additional dependencies required for running the benchmark.

```bash
pip install -r requirements.txt
```

--------------------------------

### Evaluating Factual Consistency

Source: https://github.com/yuh-zha/alignscore/blob/main/README.md

Example of how to evaluate factual consistency using the AlignScore metric.

```python
from alignscore import AlignScore

scorer = AlignScore(model='roberta-base', batch_size=32, device='cuda:0', ckpt_path='/path/to/checkpoint', evaluation_mode='nli_sp')
score = scorer.score(contexts=['hello world.'], claims=['hello world.'])
```

--------------------------------

### Package Dependencies

Source: https://github.com/yuh-zha/alignscore/blob/main/requirements.txt

This file lists the Python packages and their version constraints required for the alignscore project.

```text
alignscore>=0.1
ctc_score==0.1.3
BLEURT @ git+https://github.com/google-research/bleurt@cebe7e6f996b40910cfaa520a63db47807e3bf5c
bert_score==0.3.11
rake_nltk==1.0.6
summa==1.2.0
benepar==0.2.0
summac==0.0.3
tabulate>=0.9.0,<1
```

--------------------------------

### Training AlignScore

Source: https://github.com/yuh-zha/alignscore/blob/main/README.md

Command to train an alignment model from scratch or on custom data.

```bash
python train.py --seed 2022 --batch-size 32 \
--num-epoch 3 --devices 0 1 2 3 \
--model-name roberta-large -- ckpt-save-path ./ckpt/ \
--data-path ./data/training_sets/ \
--max-samples-per-dataset 500000
```

--------------------------------

### AlignScore Evaluation Arguments

Source: https://github.com/yuh-zha/alignscore/blob/main/README.md

Arguments for evaluating AlignScore during benchmarking.

```bash
--alignscore
--alignscore-model roberta-base or roberta-large
--alignscore-ckpt path/to/checkpoint
--alignscore-eval-mode nli_sp (default)
--device cuda:0 (default)
--tasks SummEval, QAGS-CNNDM, ...
```

--------------------------------

### Citation for AlignScore

Source: https://github.com/yuh-zha/alignscore/blob/main/README.md

BibTeX entry for citing the AlignScore metric and repository.

```bibtex
@inproceedings{zha-etal-2023-alignscore,
    title = "{A}lign{S}core: Evaluating Factual Consistency with A Unified Alignment Function",
    author = "Zha, Yuheng  and
      Yang, Yichi  and
      Li, Ruichen  and
      Hu, Zhiting",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.634",
    pages = "11328--11348",
    abstract = "Many text generation applications require the generated text to be factually consistent with input information. Automatic evaluation of factual consistency is challenging. Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data. Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks. In this paper, we propose AlignScore, a new holistic metric that applies to a variety of factual inconsistency scenarios as above. AlignScore is based on a general function of information alignment between two arbitrary text pieces. Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization). We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training. AlignScore achieves substantial improvement over a wide range of previous metrics. Moreover, AlignScore (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger."
}
```
Tokens:
Raw