### Install QuestEval with Conda Source: https://github.com/thomasscialom/questeval/blob/main/README.md Set up a Python 3.9 environment using Conda and activate it. Ensure PyTorch is installed with the correct CUDA version before proceeding. ```bash $ conda create --name questeval python=3.9 $ conda activate questeval ``` ```bash (questeval) $ conda install pytorch cudatoolkit=10.1 -c pytorch ``` ```bash (questeval) $ conda install pip (questeval) $ pip install -e . ``` -------------------------------- ### Install QuestEval and Dependencies Source: https://context7.com/thomasscialom/questeval/llms.txt Installs QuestEval using conda and pip. Ensure PyTorch is installed first, matching your CUDA version. ```bash conda create --name questeval python=3.9 conda activate questeval # Install PyTorch first (match your CUDA version) conda install pytorch cudatoolkit=10.1 -c pytorch conda install pip pip install -e . ``` -------------------------------- ### QuestEval Corpus Scoring Source: https://context7.com/thomasscialom/questeval/llms.txt Scores a list of hypothesis texts against sources, references, or both. Returns corpus-level average and per-example scores. Requires at least one of `sources` or `list_references`. Processes examples in batches and caches intermediate logs. ```APIDOC ## QuestEval.corpus_questeval Scores a corpus of hypothesis texts against sources, references, or both. Returns a corpus-level average and per-example scores. At least one of `sources` or `list_references` must be provided. Internally processes examples in batches of 512 and caches intermediate QA/QG logs to disk. ### Method POST ### Endpoint `/corpus_questeval` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **hypothesis** (list[str]) - Required - A list of hypothesis strings to evaluate. - **sources** (list[str]) - Optional - A list of source texts corresponding to the hypotheses. - **list_references** (list[list[str]]) - Optional - A list of lists, where each inner list contains reference texts for the corresponding hypothesis. ### Request Example ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(no_cuda=True) # --- With both sources and references --- source_1 = "Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award to the value of 35000." prediction_1 = "Since 2000, the winner of the Kate Greenaway medal has also been given to the Colin Mears award of the Kate Greenaway medal." references_1 = [ "Since 2000, the recipient of the Kate Greenaway Medal will also receive the Colin Mears Award which worth 5000 pounds", "Since 2000, the recipient of the Kate Greenaway Medal has also been given the Colin Mears Award." ] source_2 = "He is also a member of another Jungiery boyband 183 Club." prediction_2 = "He also has another Jungiery Boyband 183 club." references_2 = [ "He's also a member of another Jungiery boyband, 183 Club.", "He belonged to the Jungiery boyband 183 Club." ] score = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], sources=[source_1, source_2], list_references=[references_1, references_2] ) print(score) # {'corpus_score': 0.6115364039438322, 'ex_level_scores': [0.5698804143875364, 0.6531923935001279]} # --- Reference-less mode (source only) --- score_ref_less = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], sources=[source_1, source_2] ) print(score_ref_less) # {'corpus_score': 0.5538727587335324, 'ex_level_scores': [0.5382940950847808, 0.569451422382284]} # --- Reference-only mode --- score_ref_only = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], list_references=[references_1, references_2] ) print(score_ref_only) # {'corpus_score': ..., 'ex_level_scores': [...]} ``` ### Response #### Success Response (200) - **corpus_score** (float) - The average score across all examples in the corpus. - **ex_level_scores** (list[float]) - A list of scores for each individual example. #### Response Example ```json { "corpus_score": 0.6115364039438322, "ex_level_scores": [ 0.5698804143875364, 0.6531923935001279 ] } ``` ``` -------------------------------- ### QuestEval Initialization Source: https://context7.com/thomasscialom/questeval/llms.txt Instantiates the QuestEval metric, loading required QA/QG models. Parameters control task type, scoring components, device, and caching. Pre-trained models are downloaded and cached automatically. ```APIDOC ## QuestEval.__init__ Instantiates the QuestEval metric, loading all required QA/QG models from Hugging Face. Key parameters control the task type, scoring components, device, and caching behaviour. Pre-trained models are automatically downloaded on first use and cached locally. ```python from questeval.questeval_metric import QuestEval # Default (text2text, English, GPU if available, cache enabled) questeval = QuestEval() # CPU-only, no cache questeval = QuestEval(no_cuda=True, use_cache=False) # Summarization with importance Weighter questeval = QuestEval(task="summarization", do_weighter=True, no_cuda=True) # Data-to-text (structured table inputs) questeval = QuestEval(task="data2text", no_cuda=True) # Custom scoring components and answer types questeval = QuestEval( task="text2text", language="en", answer_types=('NER', 'NOUN'), # entity types used as answer candidates list_scores=('answerability', 'bertscore', 'f1'), # similarity metrics do_consistency=False, # round-trip consistency check do_weighter=False, # importance weighter (summarization) qg_batch_size=36, # batch size for question generation clf_batch_size=48, # batch size for QA / classification limit_sent=5, # max sentences per text for QG no_cuda=True, use_cache=True ) ``` ``` -------------------------------- ### Initialize QuestEval Metric Source: https://context7.com/thomasscialom/questeval/llms.txt Instantiates the QuestEval metric. Default settings use text2text, English, GPU if available, and caching. Customize task type, components, device, and batch sizes as needed. ```python from questeval.questeval_metric import QuestEval # Default (text2text, English, GPU if available, cache enabled) questeval = QuestEval() # CPU-only, no cache questeval = QuestEval(no_cuda=True, use_cache=False) # Summarization with importance Weighter questeval = QuestEval(task="summarization", do_weighter=True, no_cuda=True) # Data-to-text (structured table inputs) questeval = QuestEval(task="data2text", no_cuda=True) # Custom scoring components and answer types questeval = QuestEval( task="text2text", language="en", answer_types=('NER', 'NOUN'), # entity types used as answer candidates list_scores=('answerability', 'bertscore', 'f1'), # similarity metrics do_consistency=False, # round-trip consistency check do_weighter=False, # importance weighter (summarization) qg_batch_size=36, # batch size for question generation clf_batch_size=48, # batch size for QA / classification limit_sent=5, # max sentences per text for QG no_cuda=True, use_cache=True ) ``` -------------------------------- ### Instantiate and Use API_T2T for T5 Inference Source: https://context7.com/thomasscialom/questeval/llms.txt Instantiate the API_T2T wrapper for batched T5 inference. Use this for custom pipelines requiring generated text and answerability scores. Ensure inputs are formatted as 'question context'. ```python from questeval.utils import API_T2T model = API_T2T( pretrained_model_name_or_path="ThomasNLG/t5-qa_squad2neg-en", keep_score_idx=73, # token index for "unanswerable" in T5 vocabulary max_source_length=512, model_batch_size=48, device="cpu" ) # Input format: "question context" inputs = [ "What award has been given since 2000? Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award.", "Who discovered 1950 DA? 1950 DA was discovered by Carl A. Wirtanen." ] answerability_scores, answer_texts = model.predict(inputs) print(answer_texts) # ['Colin Mears award', 'Carl A. Wirtanen'] print(answerability_scores) # [0.97..., 0.99...] # probability the question IS answerable ``` -------------------------------- ### Swap QA/QG models at runtime with QuestEval.set_model Source: https://context7.com/thomasscialom/questeval/llms.txt Replace QA, QG, or Weighter models after initialization without reloading the entire metric. Accepts Hugging Face model names or local file paths for T5-based architectures. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(no_cuda=True) # Replace hypothesis QG model with a fine-tuned local checkpoint questeval.set_model(key='hyp', task='QG', model_name='/path/to/my-finetuned-qg-model') # Replace hypothesis QA model with a Hugging Face model questeval.set_model(key='hyp', task='QA', model_name='ThomasNLG/t5-qa_squad2neg-en') # Replace reference QA model questeval.set_model(key='ref', task='QA', model_name='ThomasNLG/t5-qa_squad2neg-en') # Score with the custom models score = questeval.corpus_questeval( hypothesis=["The winner received the award since 2000."], sources=["Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award."] ) print(score) # {'corpus_score': ..., 'ex_level_scores': [...]} ``` -------------------------------- ### Summarization with QuestEval Weighter Source: https://context7.com/thomasscialom/questeval/llms.txt Use QuestEval for summarization tasks with an optional weighter to prioritize important facts. The weighter can be dynamically disabled. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(task="summarization", do_weighter=True, no_cuda=True) source = ( "(CNN) -- An American woman died aboard a cruise ship that docked at Rio de Janeiro on Tuesday, " "the same ship on which 86 passengers previously fell ill, according to Agencia Brasil. " "The ship's doctors told police that the woman was elderly and suffered from diabetes and hypertension." ) hypothesis = "The elderly woman suffered from diabetes and hypertension, ship's doctors say. Previously, 86 passengers had fallen ill on the ship." reference = ["The elderly woman suffered from diabetes and hypertension, ship's doctors say. Previously, 86 passengers had fallen ill on the ship, Agencia Brasil says."] score = questeval.corpus_questeval( hypothesis=[hypothesis], sources=[source], list_references=[reference] ) print(score) # {'corpus_score': 0.8672..., 'ex_level_scores': [0.8672...])} # Disable the weighter dynamically without reloading models questeval.do_weighter = False score_no_weight = questeval.corpus_questeval( hypothesis=[hypothesis], sources=[source], list_references=[reference] ) print(score_no_weight) # {'corpus_score': 0.8672..., 'ex_level_scores': [0.8672...])} ``` -------------------------------- ### Evaluate Text Similarity with QuestEval Source: https://github.com/thomasscialom/questeval/blob/main/README.md Instantiate QuestEval and use corpus_questeval to measure content similarity between hypotheses and sources, with optional references. Caching is enabled by default for faster subsequent runs. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(no_cuda=True) source_1 = "Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award to the value of 35000." prediction_1 = "Since 2000, the winner of the Kate Greenaway medal has also been given to the Colin Mears award of the Kate Greenaway medal." references_1 = [ "Since 2000, the recipient of the Kate Greenaway Medal will also receive the Colin Mears Awad which worth 5000 pounds", "Since 2000, the recipient of the Kate Greenaway Medal has also been given the Colin Mears Award." ] source_2 = "He is also a member of another Jungiery boyband 183 Club." prediction_2 = "He also has another Jungiery Boyband 183 club." references_2 = [ "He's also a member of another Jungiery boyband, 183 Club.", "He belonged to the Jungiery boyband 183 Club." ] score = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], sources=[source_1, source_2], list_references=[references_1, references_2] ) print(score) ``` ```python score = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], sources=[source_1, source_2] ) print(score) ``` -------------------------------- ### Generate QuestEval Reproducibility Hash Source: https://github.com/thomasscialom/questeval/blob/main/README.md Generate a hash string that encapsulates the QuestEval version, models used, and scoring configurations for reproducible results. ```python questeval.__hash__() ``` -------------------------------- ### Data-to-Text Task with QuestEval Source: https://context7.com/thomasscialom/questeval/llms.txt QuestEval supports data-to-text tasks using structured tables (GEM/WebNLG format). A preprocessor converts triples to tokens, or a custom callable can be provided. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(task="data2text", no_cuda=True) # Source: list of triples (GEM/WebNLG format) source_table = [["(29075)_1950_da | discoverer | carl_a._wirtanen"]] hypothesis = ["1950 da was discovered by carl."] reference = [["( 29075 ) 1950 da was discovered by carl a wirtanen ."]] # Source + reference score = questeval.corpus_questeval( hypothesis=hypothesis, sources=source_table, list_references=reference ) print(score) # {'corpus_score': 0.5690..., 'ex_level_scores': [0.5690...])} # Source only (referenceless) score_src = questeval.corpus_questeval( hypothesis=hypothesis, sources=source_table ) print(score_src) # {'corpus_score': 0.4912..., 'ex_level_scores': [0.4912...])} ``` -------------------------------- ### Access QuestEval Logs Source: https://github.com/thomasscialom/questeval/blob/main/README.md Open logs from text to access detailed information about generated questions and question-answering outputs. This allows inspection of the internal processes. ```python log = questeval.open_log_from_text(source_1) print(log['asked'].keys()) ``` -------------------------------- ### QuestEval.set_model Source: https://context7.com/thomasscialom/questeval/llms.txt Swap QA/QG models at runtime without reloading the entire metric. Accepts Hugging Face model names or local file paths for T5-based architectures. ```APIDOC ## `QuestEval.set_model` — Swap QA/QG models at runtime Replaces any QA, QG, or Weighter model after initialization without reloading the entire metric. Accepts a Hugging Face model name (auto-downloaded) or a local file path. The model must be a T5-based architecture; the `predict()` interface is used internally. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(no_cuda=True) # Replace hypothesis QG model with a fine-tuned local checkpoint questeval.set_model(key='hyp', task='QG', model_name='/path/to/my-finetuned-qg-model') # Replace hypothesis QA model with a Hugging Face model questeval.set_model(key='hyp', task='QA', model_name='ThomasNLG/t5-qa_squad2neg-en') # Replace reference QA model questeval.set_model(key='ref', task='QA', model_name='ThomasNLG/t5-qa_squad2neg-en') # Score with the custom models score = questeval.corpus_questeval( hypothesis=["The winner received the award since 2000."], sources=["Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award."] ) print(score) # {'corpus_score': ..., 'ex_level_scores': [...]} ``` ``` -------------------------------- ### linearize_e2e_input Source: https://context7.com/thomasscialom/questeval/llms.txt Linearize E2E table strings into a space-separated token format for QuestEval's data-to-text QA/QG models. ```APIDOC ## `linearize_e2e_input` — Linearize E2E table strings A standalone utility that converts an E2E Meaning Representation string (key-value pairs with bracket notation) into the space-separated token format expected by QuestEval's data-to-text QA/QG models. ```python from questeval.utils import linearize_e2e_input mr = 'name[The Eagle], eatType[coffee shop], food[French], priceRange[£20-25], customerRating[3 out of 5]' linearized = linearize_e2e_input(mr, format='gem') print(linearized) # 'name [ The Eagle ] , eatType [ coffee shop ] , food [ French ] , priceRange [ £20-25 ] , customerRating [ 3 out of 5 ]' # Then pass to QuestEval as a custom src_preproc_pipe if needed: questeval_d2t = QuestEval(task="data2text", src_preproc_pipe=lambda s: linearize_e2e_input(s)) ``` ``` -------------------------------- ### Linearize E2E table strings with linearize_e2e_input Source: https://context7.com/thomasscialom/questeval/llms.txt Converts E2E Meaning Representation strings (key-value pairs with bracket notation) into a space-separated token format for QuestEval's data-to-text QA/QG models. Can be used as a custom src_preproc_pipe. ```python from questeval.utils import linearize_e2e_input mr = 'name[The Eagle], eatType[coffee shop], food[French], priceRange[£20-25], customerRating[3 out of 5]' linearized = linearize_e2e_input(mr, format='gem') print(linearized) # 'name [ The Eagle ] , eatType [ coffee shop ] , food [ French ] , priceRange [ £20-25 ] , customerRating [ 3 out of 5 ]' # Then pass to QuestEval as a custom src_preproc_pipe if needed: questeval_d2t = QuestEval(task="data2text", src_preproc_pipe=lambda s: linearize_e2e_input(s)) ``` -------------------------------- ### LinearizeWebnlgInput Source: https://context7.com/thomasscialom/questeval/llms.txt Linearize WebNLG triple lists into a grouped, linearized token sequence, handling entity grouping and normalization. ```APIDOC ## `LinearizeWebnlgInput` — Linearize WebNLG triple lists A callable class (used automatically when `task="data2text"`) that converts a list of RDF-style triple strings in GEM/WebNLG format into a grouped, linearized token sequence. Handles entity grouping, underscore-to-space conversion, Unicode normalization, and optional lowercasing. ```python import spacy from questeval.utils import LinearizeWebnlgInput nlp = spacy.load('en_core_web_sm') linearizer = LinearizeWebnlgInput(spacy_pipeline=nlp, lowercase=False, format='gem') triples = [ "(15788)_1993_SB | discoverer | Donal_O'Ceallaigh", "(15788)_1993_SB | epoch | 2006-03-06", "(15788)_1993_SB | apoapsis | 2.801_AU" ] result = linearizer(triples) print(result) # 'entity [ (15788) 1993 SB ] , discoverer [ Donal O\'Ceallaigh ] , epoch [ 2006-03-06 ] , apoapsis [ 2.801 AU ]' ``` ``` -------------------------------- ### QuestEval Reproducibility Hash Source: https://context7.com/thomasscialom/questeval/llms.txt Generate a canonical string identifier using `__hash__` that encodes the metric's full configuration, including version, task, language, and model identifiers, for logging. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(no_cuda=True) print(questeval.__hash__()) # "QuestEval_version=0.2.4_task=text2text_lang=en_preproc=None_consist=False" ``` -------------------------------- ### Score Corpus with QuestEval Source: https://context7.com/thomasscialom/questeval/llms.txt Uses the `corpus_questeval` API to score hypothesis texts against sources, references, or both. Supports reference-less and reference-only modes. Results include a corpus-level average and per-example scores. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(no_cuda=True) # --- With both sources and references --- source_1 = "Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award to the value of 35000." prediction_1 = "Since 2000, the winner of the Kate Greenaway medal has also been given to the Colin Mears award of the Kate Greenaway medal." references_1 = [ "Since 2000, the recipient of the Kate Greenaway Medal will also receive the Colin Mears Award which worth 5000 pounds", "Since 2000, the recipient of the Kate Greenaway Medal has also been given the Colin Mears Award." ] source_2 = "He is also a member of another Jungiery boyband 183 Club." prediction_2 = "He also has another Jungiery Boyband 183 club." references_2 = [ "He's also a member of another Jungiery boyband, 183 Club.", "He belonged to the Jungiery boyband 183 Club." ] score = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], sources=[source_1, source_2], list_references=[references_1, references_2] ) print(score) # {'corpus_score': 0.6115364039438322, 'ex_level_scores': [0.5698804143875364, 0.6531923935001279]} # --- Reference-less mode (source only) --- score_ref_less = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], sources=[source_1, source_2] ) print(score_ref_less) # {'corpus_score': 0.5538727587335324, 'ex_level_scores': [0.5382940950847808, 0.569451422382284]} # --- Reference-only mode --- score_ref_only = questeval.corpus_questeval( hypothesis=[prediction_1, prediction_2], list_references=[references_1, references_2] ) print(score_ref_only) # {'corpus_score': ..., 'ex_level_scores': [...]} ``` -------------------------------- ### Linearize WebNLG triple lists with LinearizeWebnlgInput Source: https://context7.com/thomasscialom/questeval/llms.txt Converts RDF-style triple strings in GEM/WebNLG format into a grouped, linearized token sequence. Handles entity grouping, underscore-to-space conversion, Unicode normalization, and optional lowercasing. Used automatically when task='data2text'. ```python import spacy from questeval.utils import LinearizeWebnlgInput nlp = spacy.load('en_core_web_sm') linearizer = LinearizeWebnlgInput(spacy_pipeline=nlp, lowercase=False, format='gem') triples = [ "(15788)_1993_SB | discoverer | Donal_O'Ceallaigh", "(15788)_1993_SB | epoch | 2006-03-06", "(15788)_1993_SB | apoapsis | 2.801_AU" ] result = linearizer(triples) print(result) # 'entity [ (15788) 1993 SB ] , discoverer [ Donal O\'Ceallaigh ] , epoch [ 2006-03-06 ] , apoapsis [ 2.801 AU ]' ``` -------------------------------- ### Inspect QuestEval Cached Logs Source: https://context7.com/thomasscialom/questeval/llms.txt Use `open_log_from_text` to retrieve cached JSON logs for processed texts. This exposes generated questions, answers, and scores for debugging. ```python from questeval.questeval_metric import QuestEval questeval = QuestEval(no_cuda=True) source = "Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award to the value of 35000." prediction = "Since 2000, the winner of the Kate Greenaway medal has also been given to the Colin Mears award of the Kate Greenaway medal." # Run scoring first to populate the cache questeval.corpus_questeval(hypothesis=[prediction], sources=[source]) # Inspect the source log log = questeval.open_log_from_text(source) # Questions generated from the source text print(log['asked'].keys()) # dict_keys(['Since 2000, the winner...', 'What medal has been given to the winner...', ...]) # Answer candidates extracted by NER print(log['self']['NER']['answers']) # ['2000', 'the Kate Greenaway', 'Colin Mears', '35000'] # Inspect a specific question's QA output for question, qa_data in log['asked'].items(): for model_key, result in qa_data.items(): print(f"Q: {question}") print(f" Predicted answer : {result['answer']}") print(f" Answerability : {result['answerability']:.4f}") for gold, scores in result['ground_truth'].items(): print(f" Gold: {gold} | F1: {scores.get('f1', 'N/A')} | BERTScore: {scores.get('bertscore', 'N/A')}") break break ``` -------------------------------- ### API_T2T - Low-level T5 inference wrapper Source: https://context7.com/thomasscialom/questeval/llms.txt The internal class that wraps any T5-based Hugging Face model for batched sequence-to-sequence inference. It returns both generated text and a scalar score extracted from the first decoder token's softmax distribution. This class can be instantiated directly for custom pipelines. ```APIDOC ## `API_T2T` — Low-level T5 inference wrapper The internal class that wraps any T5-based Hugging Face model for batched sequence-to-sequence inference. Returns both generated text and a scalar score extracted from the first decoder token's softmax distribution (used for answerability and weighter classification). Can be instantiated directly for custom pipelines. ```python from questeval.utils import API_T2T model = API_T2T( pretrained_model_name_or_path="ThomasNLG/t5-qa_squad2neg-en", keep_score_idx=73, # token index for "unanswerable" in T5 vocabulary max_source_length=512, model_batch_size=48, device="cpu" ) # Input format: "question context" inputs = [ "What award has been given since 2000? Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award.", "Who discovered 1950 DA? 1950 DA was discovered by Carl A. Wirtanen." ] answerability_scores, answer_texts = model.predict(inputs) print(answer_texts) # ['Colin Mears award', 'Carl A. Wirtanen'] print(answerability_scores) # [0.97..., 0.99...] # probability the question IS answerable ``` ``` -------------------------------- ### Calculate token-level F1 similarity with calculate_f1_squad Source: https://context7.com/thomasscialom/questeval/llms.txt Computes the token overlap F1 score between predicted and gold answers following SQuAD evaluation protocol (lowercasing, punctuation removal, article removal, whitespace normalization). ```python from questeval.utils import calculate_f1_squad # Exact match print(calculate_f1_squad("Kate Greenaway medal", "Kate Greenaway medal")) # 1.0 # Partial match print(calculate_f1_squad("Kate Greenaway medal", "Greenaway medal")) # 0.8 # No overlap print(calculate_f1_squad("Colin Mears award", "Kate Greenaway medal")) # 0.0 # Both unanswerable print(calculate_f1_squad("", "")) # 1.0 ``` -------------------------------- ### calculate_f1_squad Source: https://context7.com/thomasscialom/questeval/llms.txt Calculate token-level F1 similarity between predicted and gold answers following SQuAD evaluation protocol. ```APIDOC ## `calculate_f1_squad` — Token-level F1 similarity A utility function that computes the token overlap F1 score between a predicted answer and a gold answer, following the SQuAD evaluation protocol: lowercasing, punctuation removal, article removal, and whitespace normalization before token matching. ```python from questeval.utils import calculate_f1_squad # Exact match print(calculate_f1_squad("Kate Greenaway medal", "Kate Greenaway medal")) # 1.0 # Partial match print(calculate_f1_squad("Kate Greenaway medal", "Greenaway medal")) # 0.8 # No overlap print(calculate_f1_squad("Colin Mears award", "Kate Greenaway medal")) # 0.0 # Both unanswerable print(calculate_f1_squad("", "")) # 1.0 ``` ``` -------------------------------- ### calculate_BERTScore Source: https://context7.com/thomasscialom/questeval/llms.txt Compute semantic similarity using BERTScore, wrapping the Hugging Face `bertscore` metric. ```APIDOC ## `calculate_BERTScore` — Semantic similarity via BERTScore Wraps the Hugging Face `bertscore` metric to compute semantic F1 scores between lists of predicted and gold answers using `bert-base-multilingual-cased`. Returns a list of float F1 values, one per pair. Called internally during `corpus_questeval` when `'bertscore'` is in `list_scores`. ```python from datasets import load_metric from questeval.utils import calculate_BERTScore metric_BERTScore = load_metric("bertscore") predictions = ["Kate Greenaway medal", "the Colin Mears award", "35000 pounds"] references = ["Kate Greenaway Medal", "Colin Mears Award", "35,000 GBP"] scores = calculate_BERTScore(predictions, references, metric_BERTScore, device='cpu') print(scores) # [0.982..., 0.951..., 0.743...] ``` ``` -------------------------------- ### Calculate semantic similarity with calculate_BERTScore Source: https://context7.com/thomasscialom/questeval/llms.txt Wraps Hugging Face's `bertscore` to compute semantic F1 scores between lists of predicted and gold answers using `bert-base-multilingual-cased`. Returns a list of float F1 values per pair. Used internally when 'bertscore' is in `list_scores`. ```python from datasets import load_metric from questeval.utils import calculate_BERTScore metric_BERTScore = load_metric("bertscore") predictions = ["Kate Greenaway medal", "the Colin Mears award", "35000 pounds"] references = ["Kate Greenaway Medal", "Colin Mears Award", "35,000 GBP"] scores = calculate_BERTScore(predictions, references, metric_BERTScore, device='cpu') print(scores) # [0.982..., 0.951..., 0.743...] ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.