### Supervised fine-tuning LLaMA-3 with train_llama3.py Source: https://context7.com/tablebench/tablebench/llms.txt Implements a supervised fine-tuning loop using HuggingFace Trainer. It tokenizes data, masks instruction tokens from the loss, and saves the final model checkpoint. Requires distributed training setup. ```bash # Fine-tune LLaMA-3-8B on TableInstruct with 8×GPU FSDP CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" \ python -m torch.distributed.launch \ --nproc_per_node=8 --use_env \ train/train_llama3.py \ --model_name_or_path Llama-3-8B \ --data_path data/v2/filtered_results_v2/S_TableBench_Merge_instructions.jsonl \ --output_dir ckpt/llama-3-8b-v2/ \ --num_train_epochs 3 \ --per_device_train_batch_size 1 \ --gradient_accumulation_steps 16 \ --learning_rate 2e-5 \ --model_max_length 4096 \ --bf16 True \ --tf32 True \ --fsdp "full_shard auto_wrap" \ --fsdp_transformer_layer_cls_to_wrap LlamaDecoderLayer ``` -------------------------------- ### Example JSON Structure for LLM Predictions Source: https://github.com/tablebench/tablebench/blob/main/README.md Store your model's predictions in the 'prediction' field and include your model name in the 'model_name' field. This is the expected format for inference results. ```json [ { ... "model_name": "your-model-name-here", "prediction": "Final Answer: 1062" }, ... ] ``` -------------------------------- ### Run vLLM batch inference with infer.py Source: https://context7.com/tablebench/tablebench/llms.txt Loads a local LLM using vLLM for efficient batch inference. Processes .jsonl files from a data path, generates predictions with model-specific prompts, and saves output JSONL files. ```bash # Run inference with a LLaMA-3 checkpoint on TableBench test data python inference/infer.py \ --base_model ckpt/llama-3-1-8b-v2 \ --data_path data/v2/test \ --outdir outputs/ \ --temperature 0.0 \ --sample_n 1 # Each output record will contain: # {"id": "...", "qtype": "...", ..., "model_name": "llama-3-1-8b-v2", "prediction": "Final Answer: 1062"} ``` -------------------------------- ### Execute PoT Code and Capture Output Source: https://context7.com/tablebench/tablebench/llms.txt Wraps extracted Python code in a `__main__` guard, executes it with a timeout, captures stdout, and parses the output for "Final Answer: ...". Returns the parsed answer and a boolean indicating if execution succeeded without errors. ```python from parse_tablebench_instruction_response_script import parse_general_code_then_exec prediction = "" ```python data = [121, 322, 619] total = sum(data) print(f"Final Answer: {total}") ``` " parsed_answer, ecr_1 = parse_general_code_then_exec(prediction) print(parsed_answer) # "1062" print(ecr_1) # True # Code with a runtime error bad_pred = "```python\nprint(1/0)\n```" answer, ecr_1 = parse_general_code_then_exec(bad_pred) print(answer, ecr_1) # '' False ``` -------------------------------- ### Evaluate and Score Subtypes Independently Source: https://context7.com/tablebench/tablebench/llms.txt Iterates through all model/instruction/subtype combinations to apply appropriate metrics and save results. Supports EM, EM with error margins, ROUGE-L, and Pass@1/ECR@1 for visualizations. ```python from eval_tablebench_script import eval_by_subtype, build_categoried_llm_inference_results from metrics.qa_metrics import QAMetric metric_engine = QAMetric() categorized = build_categoried_llm_inference_results( ["eval_examples/evaluation_results/overall_sim_inference_results.json"], [] ) subtype_results = eval_by_subtype(categorized, metric_engine) # Example output structure: # { # "o3-mini-2025-01-31/DP": { # "NumericalReasoning_Aggregation": { # "EM": 95.24, "EM_with_error_10": 97.62, "ROUGE-L": 95.24, # "Parse@1": 100.0, "total": 42 # }, ``` -------------------------------- ### Evaluate LLM Results with TableBench Script Source: https://github.com/tablebench/tablebench/blob/main/README.md This script evaluates parsed LLM predictions using metrics like subtype and type accuracy. It merges results, builds categorized data, and saves CSV reports. Requires `QAMetric` and helper functions for merging, building, and saving results. ```python if __name__ == '__main__': # ==== Global settings ==== PROJECT_ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) EXP_DIR = 'eval_examples' PARSED_RUSULT_DIR = f'{PROJECT_ROOT_DIR}/{EXP_DIR}/parsed_results' EVAL_RESULT_DIR = f'{PROJECT_ROOT_DIR}/{EXP_DIR}/evaluation_results' # Init metric evaluation engine metric_eval_engine = QAMetric() # Merge parsed results to one file overall_sim_inference_results_path = merge_parsed_results_to_one_sim_file( PARSED_RUSULT_DIR, EVAL_RESULT_DIR) # Support add multiple inference result files candidate_eval_file_paths = [ overall_sim_inference_results_path ] # Support eval targeted models,if eval_models is empty, all models will be eval eval_models = [ # 'o3-mini-2025-01-31', ] # Load the candidate evaluation results categoried_llm_inference_results = build_categoried_llm_inference_results( candidate_eval_file_paths, eval_models) # Evaluate by subtype print('==== Evaluate by subtype ==== ') llm_eval_subtype_results = eval_by_subtype( categoried_llm_inference_results, metric_eval_engine) # sve_subtype_results to csv llm_eval_subtype_results_csv_path = f'{EVAL_RESULT_DIR}/llm_eval_subtype_results.csv' save_subtype_results_to_csv( llm_eval_subtype_results, llm_eval_subtype_results_csv_path) # Evaluate by type print('==== Evaluate by type ==== ') llm_eval_type_results = eval_by_type( categoried_llm_inference_results, metric_eval_engine) # save_type_results to csv llm_eval_type_results_csv_path = f'{EVAL_RESULT_DIR}/llm_eval_type_results.csv' save_type_results_to_csv( llm_eval_type_results, llm_eval_type_results_csv_path) ``` -------------------------------- ### Build Categorized LLM Inference Results Source: https://context7.com/tablebench/tablebench/llms.txt Organizes inference results from one or more files into a nested dictionary structure. This facilitates analysis by grouping results by model and task type. ```python from eval_tablebench_script import build_categoried_llm_inference_results candidate_paths = ["eval_examples/evaluation_results/overall_sim_inference_results.json"] eval_models = [] # empty = evaluate all models categorized = build_categoried_llm_inference_results(candidate_paths, eval_models) # Inspect available model/instruction combos for key in categorized: print(key) # o3-mini-2025-01-31/DP # Inspect task subtypes for a model for subtype, results in categorized["o3-mini-2025-01-31/DP"].items(): print(f"{subtype}: {len(results)} samples") # NumericalReasoning_Aggregation: 42 samples # FactChecking_MatchBased: 35 samples # ... ``` -------------------------------- ### Consolidate parsed results with merge_parsed_results_to_one_sim_file Source: https://context7.com/tablebench/tablebench/llms.txt Merges all .jsonl files from a specified directory into a single .json file, stripping large fields to manage file size. Output is saved to a designated evaluation results directory. ```python from eval_tablebench_script import merge_parsed_results_to_one_sim_file merged_path = merge_parsed_results_to_one_sim_file( parsed_results_dir="eval_examples/parsed_results", eval_results_dir="eval_examples/evaluation_results" ) print(merged_path) # eval_examples/evaluation_results/overall_sim_inference_results.json ``` -------------------------------- ### Compute All QA Metrics with QAMetric Source: https://context7.com/tablebench/tablebench/llms.txt A high-level wrapper to compute various text-QA metrics including EM, EM with error margins, and ROUGE-L. It normalizes inputs by lowercasing, removing punctuation, and articles. ```python from metrics.qa_metrics import QAMetric engine = QAMetric() references = ["1062", "68", "12 de octubre"] predictions = ["1062", "68", "12 De Octubre"] # case mismatch → normalized scores = engine.compute(references, predictions) print(scores) # { # 'EM': 100.0, # 'EM_with_error_2': 100.0, # 'EM_with_error_5': 100.0, # 'EM_with_error_10': 100.0, # 'ROUGE-L': 100.0 # } ``` -------------------------------- ### Parse LLM Predictions with TableBench Script Source: https://github.com/tablebench/tablebench/blob/main/README.md This script iterates through inference result files, parses the predictions, and saves the structured results. Ensure the necessary helper functions like `iter_file_from_dir`, `read_json_file`, `parse_inference_results`, and `write_json_to_file` are available. ```python if __name__ == '__main__': # ==== Global settings ==== PROJECT_ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) EXP_DIR = 'eval_examples' INFERENCE_RESULT_DIR = f'{PROJECT_ROOT_DIR}/{EXP_DIR}/inference_results' PARSED_RUSULT_DIR = f'{PROJECT_ROOT_DIR}/{EXP_DIR}/parsed_results' # ==== Load inference results ==== for inference_result_file in iter_file_from_dir(f'{INFERENCE_RESULT_DIR}', '.jsonl'): print(f'Parsing {inference_result_file}') # === Load inference results === inference_results = read_json_file(inference_result_file) if not isinstance(inference_results, list): inference_results = [inference_results] # === Parse inference results === parsed_results = parse_inference_results(inference_results) # === Save parsed results === write_json_to_file( f'{PARSED_RUSULT_DIR}/{os.path.basename(inference_result_file)}', parsed_results, is_json_line=True) print('Parsing completed.') ``` -------------------------------- ### JSON I/O utilities: read_json_file and write_json_to_file Source: https://context7.com/tablebench/tablebench/llms.txt Provides utilities for reading JSON and JSONL files into Python lists and writing Python lists back to files in either JSON or JSONL format. Automatically creates parent directories as needed. ```python from utils.file_util import read_json_file, write_json_to_file # Read JSONL inference file records = read_json_file("eval_examples/inference_results/o3-mini-2025-01-31=TableBench_DP=Example.jsonl") print(type(records), len(records)) # N # Write back as JSON (pretty-printed) write_json_to_file("output/results.json", records, is_json_line=False) # Write as JSONL write_json_to_file("output/results.jsonl", records, is_json_line=True) ``` -------------------------------- ### build_categoried_llm_inference_results Source: https://context7.com/tablebench/tablebench/llms.txt Groups parsed inference results into a nested dictionary structure, organized by model name and instruction type, then by question type and subtype. ```APIDOC ## `build_categoried_llm_inference_results` — Group parsed results by model and task type ### Description Reads one or more merged result files and organizes records into a nested dict keyed by `"model_name/instruction_type"` → `"qtype_qsubtype"` → list of result dicts. Supports filtering to a specific subset of models via `eval_models`. ### Usage ```python from eval_tablebench_script import build_categoried_llm_inference_results candidate_paths = ["eval_examples/evaluation_results/overall_sim_inference_results.json"] eval_models = [] # empty = evaluate all models categorized = build_categoried_llm_inference_results(candidate_paths, eval_models) # Inspect available model/instruction combos for key in categorized: print(key) # o3-mini-2025-01-31/DP # Inspect task subtypes for a model for subtype, results in categorized["o3-mini-2025-01-31/DP"].items(): print(f"{subtype}: {len(results)} samples") # NumericalReasoning_Aggregation: 42 samples # FactChecking_MatchBased: 35 samples # ... ``` ``` -------------------------------- ### Extract Python Code from PoT / Visualization Output Source: https://context7.com/tablebench/tablebench/llms.txt Finds the last ```python ... ``` fenced block in the model's output. This is used for both Program-of-Thought (PoT) and Visualization chart-generation paths. ```python from parse_tablebench_instruction_response_script import parse_python_code prediction = "" Here is my solution: ```python total = 121 + 322 + 619 print(f"Final Answer: {total}") ``` " code = parse_python_code(prediction) print(code) # total = 121 + 322 + 619 # print(f"Final Answer: {total}") ``` -------------------------------- ### Parse Inference Results with TableBench Source: https://context7.com/tablebench/tablebench/llms.txt Use this function to parse raw inference records from LLMs. It dispatches samples based on instruction and question types, enriching them with parsed predictions and scores. ```python from parse_tablebench_instruction_response_script import parse_inference_results inference_results = [ { "id": "abc123", "qtype": "NumericalReasoning", "qsubtype": "Aggregation", "instruction_type": "DP", "table": {"columns": ["year", "films"], "data": [["2012", "121"], ["2013", "322"], ["2014", "619"]}}, "answer": "1062", "prediction": "Final Answer: 1062", "model_name": "gpt-4o" } ] parsed = parse_inference_results(inference_results) print(parsed[0]["parsed_result"]) # {'parsed_prediction': '1062', 'Parse@1': True} ``` -------------------------------- ### Execute and Evaluate Chart-Generation Code Source: https://context7.com/tablebench/tablebench/llms.txt For Visualization tasks, this function extracts matplotlib code, executes it, and appends a test harness to evaluate rendered chart data against reference values. It returns booleans indicating if the chart data matches the reference and if the code executed without errors. ```python from parse_tablebench_instruction_response_script import parse_chart_code_then_exec sample = { "prediction": "" ```python import matplotlib.pyplot as plt y_references = [[10, 20, 30]] x = [1, 2, 3] plt.plot(x, [10, 20, 30]) plt.show() ``` ", "answer": "y_references = [[10, 20, 30]]", "chart_type": "line", "table": { "columns": ["x", "y"], "data": [[1, 10], [2, 20], [3, 30]] } } passed, ecr_1 = parse_chart_code_then_exec(sample) print(passed) # True (chart data matches reference) print(ecr_1) # True (code executed without error) ``` -------------------------------- ### parse_general_code_then_exec Source: https://context7.com/tablebench/tablebench/llms.txt Executes extracted Python code within a `__main__` guard, with a 15-second timeout. It captures standard output and then uses `parse_dp_prediction` to extract the final answer. Returns the parsed answer and a boolean indicating execution success. ```APIDOC ## parse_general_code_then_exec ### Description Wraps the extracted Python code in a `__main__` guard, executes it with a 15-second timeout (via `wrapt_timeout_decorator`), captures stdout, and applies `parse_dp_prediction` to extract "Final Answer: ...". Returns `(parsed_prediction, ecr_1)` where `ecr_1` indicates whether execution succeeded without errors. ### Parameters None ### Request Example ```python from parse_tablebench_instruction_response_script import parse_general_code_then_exec prediction = """ ```python data = [121, 322, 619] total = sum(data) print(f"Final Answer: {total}") ``` " parsed_answer, ecr_1 = parse_general_code_then_exec(prediction) print(parsed_answer) # "1062" print(ecr_1) # True # Code with a runtime error bad_pred = "```python\nprint(1/0)\n```" answer, ecr_1 = parse_general_code_then_exec(bad_pred) print(answer, ecr_1) # '' False ``` ### Response - **parsed_prediction** (string) - The final answer extracted from the code's output, or an empty string if execution fails or no answer is found. - **ecr_1** (boolean) - True if the code executed successfully without runtime errors, False otherwise. ``` -------------------------------- ### parse_inference_results Source: https://context7.com/tablebench/tablebench/llms.txt Batch-parses a list of raw inference records, dispatching each sample to the correct parser based on instruction type and question type. It enriches each sample with a parsed_result dictionary. ```APIDOC ## `parse_inference_results` — Batch-parse a list of raw inference records ### Description Entry point for the parsing stage. Dispatches each sample to the correct parser based on `instruction_type` (`DP`, `SCoT`, `TCoT`, `PoT`) and `qtype` (`Visualization` vs. all others). Enriches each sample with a `parsed_result` dict containing `parsed_prediction`, optional `ecr_1`, and `Parse@1`. ### Usage ```python from parse_tablebench_instruction_response_script import parse_inference_results inference_results = [ { "id": "abc123", "qtype": "NumericalReasoning", "qsubtype": "Aggregation", "instruction_type": "DP", "table": {"columns": ["year", "films"], "data": [["2012", "121"], ["2013", "322"], ["2014", "619"]}}, "answer": "1062", "prediction": "Final Answer: 1062", "model_name": "gpt-4o" } ] parsed = parse_inference_results(inference_results) print(parsed[0]["parsed_result"]) # {'parsed_prediction': '1062', 'Parse@1': True} ``` ``` -------------------------------- ### QAMetric.compute Source: https://context7.com/tablebench/tablebench/llms.txt Computes all text-based QA metrics, including EM, EM with error tolerance, and ROUGE-L, after normalizing predictions and references. ```APIDOC ## `QAMetric.compute` — Compute all text-QA metrics in one call ### Description High-level wrapper that normalizes predictions/references (lowercasing, punctuation removal, article removal), then returns a dict of EM, EM_with_error_2/5/10, and ROUGE-L scores (all scaled to 0–100). ### Usage ```python from metrics.qa_metrics import QAMetric engine = QAMetric() references = ["1062", "68", "12 de octubre"] predictions = ["1062", "68", "12 De Octubre"] # case mismatch → normalized scores = engine.compute(references, predictions) print(scores) # { # 'EM': 100.0, # 'EM_with_error_2': 100.0, # 'EM_with_error_5': 100.0, # 'EM_with_error_10': 100.0, # 'ROUGE-L': 100.0 # } ``` ``` -------------------------------- ### Compute EM with Tolerance Source: https://context7.com/tablebench/tablebench/llms.txt Extends `compute_em` by allowing a relative error margin specified by `error_range`. Useful for tasks where approximate numeric answers are acceptable. ```python from metrics.custom_em_metric import compute_em_with_tolerance references = ["100", "200"] predictions = ["105", "210"] # within 10% of references score = compute_em_with_tolerance(references, predictions, error_range=10) print(score) # 1.0 # 11% error → miss score = compute_em_with_tolerance(["100"], ["112"], error_range=10) print(score) # 0.0 ``` -------------------------------- ### Parse Final Answer from Direct Prompting Output Source: https://context7.com/tablebench/tablebench/llms.txt Parses the standardized "Final Answer: ..." string from DP, SCoT, and TCoT instruction types. Returns an empty string if the pattern is not found, which will result in Parse@1 = False downstream. ```python from parse_tablebench_instruction_response_script import parse_dp_prediction prediction = "Based on the table, the total is 1062.\nFinal Answer: 1062" answer = parse_dp_prediction(prediction) print(answer) # "1062" # Missing pattern → empty string, Parse@1 = False bad = parse_dp_prediction("The answer is approximately 1062.") print(repr(bad)) # '' ``` -------------------------------- ### Compute Exact Match (EM) Metric Source: https://context7.com/tablebench/tablebench/llms.txt Calculates the Exact Match score for a list of reference and prediction pairs. Handles multi-answer responses, percentage normalization, and decimal precision. ```python from metrics.custom_em_metric import compute_em references = ["1062", "69.75%", "True, False"] predictions = ["1062", "69.75%", "True, False"] print(compute_em(references, predictions)) # 1.0 # Partial match (only first of two answers correct) print(compute_em(["True, False"], ["True, True"])) # 0.5 # Numeric normalization handles decimal rounding print(compute_em(["3.14"], ["3.1"])) # 0.0 (precision mismatch) print(compute_em(["3.1"], ["3.1"])) # 1.0 ``` -------------------------------- ### eval_by_subtype Source: https://context7.com/tablebench/tablebench/llms.txt Evaluates and scores each question sub-type independently, applying the appropriate metric and saving the results. ```APIDOC ## `eval_by_subtype` — Evaluate and score each question sub-type independently ### Description Iterates over all `(model, instruction_type, subtype)` combinations, applies the appropriate metric (EM, EM_with_error_10, ROUGE-L, or Pass@1/ECR@1 for Visualization), and saves results to `llm_eval_subtype_results.json`. ### Usage ```python from eval_tablebench_script import eval_by_subtype, build_categoried_llm_inference_results from metrics.qa_metrics import QAMetric metric_engine = QAMetric() categorized = build_categoried_llm_inference_results( ["eval_examples/evaluation_results/overall_sim_inference_results.json"], [] ) subtype_results = eval_by_subtype(categorized, metric_engine) # Example output structure: # { # "o3-mini-2025-01-31/DP": { # "NumericalReasoning_Aggregation": { # "EM": 95.24, "EM_with_error_10": 97.62, "ROUGE-L": 95.24, # "Parse@1": 100.0, "total": 42 # }, # ... ``` ``` -------------------------------- ### parse_chart_code_then_exec Source: https://context7.com/tablebench/tablebench/llms.txt Executes matplotlib code generated for Visualization tasks. It evaluates the generated chart data against reference values and returns a boolean indicating if the chart data matches and if the code executed successfully. ```APIDOC ## parse_chart_code_then_exec ### Description For Visualization tasks, extracts the model's matplotlib code, executes it, then appends a test harness that reads back rendered chart data (lines, bars, pie segments, etc.) and compares it to the reference `y_references` values. Returns `(parsed_prediction: bool, ecr_1: bool)`. ### Parameters None ### Request Example ```python from parse_tablebench_instruction_response_script import parse_chart_code_then_exec sample = { "prediction": """ ```python import matplotlib.pyplot as plt y_references = [[10, 20, 30]] x = [1, 2, 3] plt.plot(x, [10, 20, 30]) plt.show() ``` ", "answer": "y_references = [[10, 20, 30]]", "chart_type": "line", "table": { "columns": ["x", "y"], "data": [[1, 10], [2, 20], [3, 30]] } } passed, ecr_1 = parse_chart_code_then_exec(sample) print(passed) # True (chart data matches reference) print(ecr_1) # True (code executed without error) ``` ### Response - **passed** (boolean) - True if the generated chart data matches the reference values, False otherwise. - **ecr_1** (boolean) - True if the chart generation code executed successfully without errors, False otherwise. ``` -------------------------------- ### compute_em_with_tolerance Source: https://context7.com/tablebench/tablebench/llms.txt Computes Exact Match (EM) with a specified relative error tolerance. This is useful for tasks where a small margin of error is acceptable. ```APIDOC ## `compute_em_with_tolerance` — Exact Match with percentage error margin ### Description Same as `compute_em` but allows a relative numeric error up to `error_range` percent. Used for Correlation Analysis, Trend Forecasting, and Statistical Analysis sub-tasks where a 10% margin is acceptable. ### Usage ```python from metrics.custom_em_metric import compute_em_with_tolerance references = ["100", "200"] predictions = ["105", "210"] # within 10% of references score = compute_em_with_tolerance(references, predictions, error_range=10) print(score) # 1.0 # 11% error → miss score = compute_em_with_tolerance(["100"], ["112"], error_range=10) print(score) # 0.0 ``` ``` -------------------------------- ### Roll up scores with eval_by_type Source: https://context7.com/tablebench/tablebench/llms.txt Aggregates sub-type results into top-level task categories like FactChecking, NumericalReasoning, DataAnalysis, Visualization, and Overall scores. Saves results to llm_eval_type_results.json. ```python from eval_tablebench_script import eval_by_type, build_categoried_llm_inference_results from metrics.qa_metrics import QAMetric metric_engine = QAMetric() categorized = build_categoried_llm_inference_results( ["eval_examples/evaluation_results/overall_sim_inference_results.json"], [] ) type_results = eval_by_type(categorized, metric_engine) print(type_results["o3-mini-2025-01-31/DP"]) # { # "FactChecking": {"EM": 96.0, "ROUGE-L": 96.0, "Parse@1": 100.0, "total": 50}, # "NumericalReasoning":{"EM": 90.0, ..., "total": 120}, # "DataAnalysis": {"Mix_Metric": 85.3, ..., "total": 80}, # "Visualization": {"ECR@1": 88.0, "Pass@1": 72.0, "total": 25}, # "Overall": {"Mix_Metric": 88.7, "Parse@1": 97.6} # } ``` -------------------------------- ### parse_python_code Source: https://context7.com/tablebench/tablebench/llms.txt Extracts Python code blocks enclosed in triple backticks (```python ... ```) from model outputs. This is used for both Program-of-Thought (PoT) and Visualization tasks. ```APIDOC ## parse_python_code ### Description Finds the last ```python ... ``` fenced block in the model's output. Used by both the Program-of-Thought (PoT) path and the Visualization chart-generation path. ### Parameters None ### Request Example ```python from parse_tablebench_instruction_response_script import parse_python_code prediction = """ Here is my solution: ```python total = 121 + 322 + 619 print(f"Final Answer: {total}") ``` " code = parse_python_code(prediction) print(code) # total = 121 + 322 + 619 # print(f"Final Answer: {total}") ``` ### Response - **code** (string) - The extracted Python code as a string. ``` -------------------------------- ### parse_dp_prediction Source: https://context7.com/tablebench/tablebench/llms.txt Extracts the final answer from standardized Direct Prompting (DP), Structured Chain-of-Thought (SCoT), and Table Chain-of-Thought (TCoT) outputs. It looks for the 'Final Answer: ...' pattern and returns an empty string if not found. ```APIDOC ## parse_dp_prediction ### Description Parses the standardized "Final Answer: ..." string produced by DP, SCoT, and TCoT instruction types. Returns an empty string if the pattern is not found, which sets `Parse@1 = False` downstream. ### Parameters None ### Request Example ```python from parse_tablebench_instruction_response_script import parse_dp_prediction prediction = "Based on the table, the total is 1062.\nFinal Answer: 1062" answer = parse_dp_prediction(prediction) print(answer) # "1062" # Missing pattern → empty string, Parse@1 = False bad = parse_dp_prediction("The answer is approximately 1062.") print(repr(bad)) # '' ``` ### Response - **answer** (string) - The extracted final answer or an empty string if the pattern is not found. ``` -------------------------------- ### compute_em Source: https://context7.com/tablebench/tablebench/llms.txt Computes Exact Match (EM) score across a list of reference and prediction pairs. It handles multi-answer responses, percentage normalization, and numeric precision. ```APIDOC ## `compute_em` — Exact Match with multi-answer and numeric normalization ### Description Computes EM across a list of `(reference, prediction)` pairs. Handles comma-separated multi-answer responses, percentage normalization, and decimal-precision-aware rounding. Returns a float in `[0.0, 1.0]`. ### Usage ```python from metrics.custom_em_metric import compute_em references = ["1062", "69.75%", "True, False"] predictions = ["1062", "69.75%", "True, False"] print(compute_em(references, predictions)) # 1.0 # Partial match (only first of two answers correct) print(compute_em(["True, False"], ["True, True"])) # 0.5 # Numeric normalization handles decimal rounding print(compute_em(["3.14"], ["3.1"])) # 0.0 (precision mismatch) print(compute_em(["3.1"], ["3.1"])) # 1.0 ``` ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.