### Install Polygon Module Source: https://github.com/qywh2023/ocrbench/blob/main/OCRBench_v2/eval_scripts/spotting_eval/readme.txt Install the Polygon module specifically, as it may have different installation requirements. ```bash pip install Polygon3 ``` -------------------------------- ### Set up OCRBench v2 Environment Source: https://github.com/qywh2023/ocrbench/blob/main/OCRBench_v2/README.md Use these commands to create a conda environment and install dependencies for OCRBench v2 evaluation. ```bash conda create -n ocrbench_v2 python==3.10 -y conda activate ocrbench_v2 pip install -r requirements.txt ``` -------------------------------- ### Install Python Module Source: https://github.com/qywh2023/ocrbench/blob/main/OCRBench_v2/eval_scripts/spotting_eval/readme.txt Install a required Python module using pip. Replace 'module' with the actual module name. ```bash pip install 'module' ``` -------------------------------- ### Run Standalone Script Source: https://github.com/qywh2023/ocrbench/blob/main/OCRBench_v2/eval_scripts/spotting_eval/readme.txt Execute the main evaluation script with ground truth and submission files. Ensure all required modules are installed beforehand. ```bash python script.py –g=gt.zip –s=submit.zip ``` -------------------------------- ### Run Standalone Script with Optional Parameters Source: https://github.com/qywh2023/ocrbench/blob/main/OCRBench_v2/eval_scripts/spotting_eval/readme.txt Execute the evaluation script with ground truth, submission, output directory, and JSON parameters for overriding default configurations. ```bash python script.py –g=gt.zip –s=submit.zip –o=./ -p={"IOU_CONSTRAINT":0.8} ``` -------------------------------- ### Compute Overall Metrics Source: https://github.com/qywh2023/ocrbench/blob/main/OCRBench_v2/README.md Use this script to compute the overall metrics for OCRBench v2 after individual sample scores are calculated. ```python python ./eval_scripts/get_score.py --json_file ./res_folder/internvl2_5_26b.json ``` -------------------------------- ### Run Complete OCRBench v2 Evaluation Source: https://context7.com/qywh2023/ocrbench/llms.txt Executes the end-to-end evaluation workflow, from model inference to score aggregation. This involves running your model to generate predictions, calculating per-sample scores, and then aggregating these scores by category. ```bash # Step 1: Run your model inference and save results # Your model should output predictions in the required JSON format # Step 2: Calculate per-sample scores python ./eval_scripts/eval.py \ --input_path ./pred_folder/my_model_predictions.json \ --output_path ./res_folder/my_model_scored.json # Step 3: Aggregate scores by category python ./eval_scripts/get_score.py \ --json_file ./res_folder/my_model_scored.json ``` -------------------------------- ### Run Evaluation Script Source: https://github.com/qywh2023/ocrbench/blob/main/OCRBench_v2/README.md Execute this script to calculate scores for individual samples using inference results. ```python python ./eval_scripts/eval.py --input_path ./pred_folder/internvl2_5_26b.json --output_path ./res_folder/internvl2_5_26b.json ``` -------------------------------- ### Wrap HTML Table for Evaluation Source: https://context7.com/qywh2023/ocrbench/llms.txt Wraps an incomplete HTML table string with necessary tags to form a complete HTML table for evaluation purposes. ```python incomplete_table = "AB12" complete_html = wrap_html_table(incomplete_table) print(complete_html) # Adds wrappers ``` -------------------------------- ### Math Expression Evaluation (English) Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates mathematical expressions by comparing normalized strings. Whitespace is ignored during comparison. ```python from eval_scripts.vqa_metric import math_expression_evaluation, cn_math_expression_evaluation # English formula evaluation (whitespace-normalized) predict = "x^2 + y^2 = r^2" answers = ["x^2+y^2=r^2"] score = math_expression_evaluation(predict, answers) print(f"Formula Score: {score}") # 1 if normalized match ``` -------------------------------- ### Calculate Aggregated Scores Across Task Categories Source: https://context7.com/qywh2023/ocrbench/llms.txt This script computes overall metrics grouped by task category for both English and Chinese subsets of the benchmark. It categorizes tasks into predefined groups for analysis. ```python # get_score.py - Calculate aggregated scores across task categories import json from eval_scripts.get_score import main, calculate_average # Run from command line: # python ./eval_scripts/get_score.py --json_file ./res_folder/model_results.json # The script categorizes tasks and outputs: # English Categories: # - text_recognition: text recognition, fine-grained text recognition, full-page OCR # - text_detection: text grounding, VQA with position # - text_spotting: text spotting # - relationship_extraction: key information extraction, key information mapping # - element_parsing: document/chart/table parsing, formula recognition # - mathematical_calculation: math QA, text counting # - visual_text_understanding: document classification, cognition VQA, diagram QA # - knowledge_reasoning: reasoning VQA, science QA, APP agent, ASCII art # Chinese Categories: # - text_recognition: full-page OCR cn # - relationship_extraction: key information extraction cn, handwritten answer extraction cn # - element_parsing: document/table parsing cn, formula recognition cn # - visual_text_understanding: cognition VQA cn # - knowledge_reasoning: reasoning VQA cn, text translation cn # Example output: # English Scores: # text_recognition: 0.723 (Count: 500) # text_detection: 0.612 (Count: 300) # ... # English Overall Score: 0.456 # Chinese Overall Score: 0.421 ``` -------------------------------- ### Key Information Extraction Evaluation (F1 Score) Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates structured key-value extraction using F1 score. Requires parsing model predictions and ground truth into dictionaries for comparison. ```python from eval_scripts.TEDS_metric import compute_f1_score, convert_str_to_dict, generate_combinations # Parse model prediction string to dictionary model_output = """ ```json { "company": "ACME Corp", "date": "2024-01-15", "total": "150.00" } ```""" pred_dict = convert_str_to_dict(model_output) print(f"Parsed dict: {pred_dict}") # Ground truth with possible alternative values gt_answer = { "company": ["ACME Corp", "ACME Corporation"], "date": ["2024-01-15", "2024/01/15"], "total": ["150.00", "$150.00"] } # Generate all combinations for flexible matching gt_combinations = generate_combinations(gt_answer) print(f"Number of GT combinations: {len(gt_combinations)}") # Compute F1 score (keys matched exactly, values normalized) best_score = 0 for gt in gt_combinations: score = compute_f1_score(pred_dict, gt) best_score = max(best_score, score) print(f"Best F1 Score: {best_score:.3f}") ``` -------------------------------- ### Process Model Predictions and Calculate Scores Source: https://context7.com/qywh2023/ocrbench/llms.txt Use this script to process model inference results and calculate per-sample scores for English and Chinese OCR tasks. Ensure your model outputs predictions in the specified JSON format. ```python # eval.py - Process model predictions and calculate per-sample scores import json import argparse from eval_scripts.eval import process_predictions # Run from command line: # python ./eval_scripts/eval.py --input_path ./pred_folder/model_predictions.json --output_path ./res_folder/model_results.json # Example prediction JSON format that your model should output: prediction_data = [ { "dataset_name": "rico", "type": "APP agent en", # Task type identifier "id": 0, "image_path": "EN_part/app/229.jpg", "question": "What is the coupon code?", "answers": ["APPVIA"], # Ground truth answers "predict": "APPVIA" # Model's prediction }, { "dataset_name": "formula", "type": "formula recognition en", "id": 100, "image_path": "EN_part/formula/img_001.png", "question": "Recognize the formula in the image.", "answers": ["x^2 + y^2 = r^2"], "predict": "x^2+y^2=r^2" } ] # Save predictions to JSON file with open("./pred_folder/my_model.json", "w") as f: json.dump(prediction_data, f, ensure_ascii=False, indent=4) # Then run evaluation process_predictions("./pred_folder/my_model.json", "./res_folder/my_model_scored.json") # Output: JSON file with "score" field added to each sample ``` -------------------------------- ### Math Expression Evaluation (Chinese) Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates Chinese mathematical expressions, normalizing them and removing \text{} tags for accurate comparison. ```python from eval_scripts.vqa_metric import math_expression_evaluation, cn_math_expression_evaluation # Chinese formula evaluation (also removes \text{} tags) cn_predict = r"\frac{1}{2}\text{米}" cn_answers = [r"\frac{1}{2}米"] cn_score = cn_math_expression_evaluation(cn_predict, cn_answers) print(f"CN Formula Score: {cn_score}") ``` -------------------------------- ### Counting Evaluation (Regression) Source: https://context7.com/qywh2023/ocrbench/llms.txt Uses regression-based scoring for counting tasks, allowing for a tolerance range. The score is calculated as 1 - |predicted - actual| / actual, provided it's greater than 0.5. ```python from eval_scripts.vqa_metric import counting_evaluation # Regression-based evaluation with tolerance predict = "I count approximately 12 items" answers = ["10"] # Ground truth is 10 score = counting_evaluation(predict, answers, eval_method="regression") # Score = 1 - |predicted - actual| / actual, if > 0.5 # Here: 1 - |12-10|/10 = 0.8 print(f"Regression Score: {score:.3f}") ``` -------------------------------- ### VQA Evaluation with ANLS Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates text-based VQA responses using substring matching for short answers and Average Normalized Levenshtein Similarity for longer responses. Supports both English and Chinese. ```python from eval_scripts.vqa_metric import vqa_evaluation, cn_vqa_evaluation, vqa_evaluation_case_sensitive # Basic VQA evaluation (case-insensitive) predict = "The answer is Facebook" answers = ["Facebook", "facebook application"] score = vqa_evaluation(predict, answers) print(f"VQA Score: {score}") # Output: VQA Score: 1 (substring match) # Chinese VQA evaluation (removes spaces for Chinese text comparison) cn_predict = "答案是 北京" cn_answers = ["北京", "北京市"] cn_score = cn_vqa_evaluation(cn_predict, cn_answers) print(f"Chinese VQA Score: {cn_score}") ``` -------------------------------- ### Full-Page OCR Evaluation Metrics Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates full-page OCR quality using BLEU, METEOR, F-measure, and edit distance. Supports both English and Chinese text. ```python from eval_scripts.page_ocr_metric import cal_per_metrics # Evaluate OCR output quality predicted_text = "The quick brown fox jumps over the lazy dog. This is a test document." ground_truth = "The quick brown fox jumps over the lazy dog. This is a test document." metrics = cal_per_metrics(predicted_text, ground_truth) print(f"BLEU: {metrics['bleu']:.3f}") print(f"METEOR: {metrics['meteor']:.3f}") print(f"F-measure: {metrics['f_measure']:.3f}") print(f"Edit Distance (normalized): {metrics['edit_dist']:.3f}") print(f"Precision: {metrics['precision']:.3f}") print(f"Recall: {metrics['recall']:.3f}") # Combined score used in evaluation (average of 4 metrics) combined_score = ( metrics['bleu'] + metrics['meteor'] + metrics['f_measure'] + (1 - metrics['edit_dist']) ) / 4 print(f"Combined OCR Score: {combined_score:.3f}") # Works with Chinese text (uses jieba for tokenization) cn_predicted = "这是一个中文文档测试" cn_ground_truth = "这是一个中文文档测试" cn_metrics = cal_per_metrics(cn_predicted, cn_ground_truth) print(f"Chinese BLEU: {cn_metrics['bleu']:.3f}") ``` -------------------------------- ### VQA with Position Evaluation Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates Visual Question Answering tasks that require both a textual answer and bounding box localization. The score is a 50/50 blend of content score and bounding box IoU. ```python from eval_scripts.IoUscore_metric import vqa_with_position_evaluation # Prediction should contain both 'answer' and 'bbox' fields prediction = { "answer": "Hello World", "bbox": "[100, 150, 250, 200]" # String format, will be parsed } # Ground truth metadata img_metas = { "answers": ["Hello World", "hello world"], "bbox": [95, 145, 255, 205] # List format } # Score = 0.5 * content_score + 0.5 * bbox_iou score = vqa_with_position_evaluation(prediction, img_metas) print(f"VQA+Position Score: {score:.3f}") ``` -------------------------------- ### TEDS Metric for Table Evaluation (Markdown) Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates table parsing accuracy using TEDS by first converting markdown tables into HTML format internally before comparison. ```python from eval_scripts.TEDS_metric import TEDS, convert_markdown_table_to_html, wrap_html_table # Initialize TEDS evaluator teds = TEDS(n_jobs=4) # Parallel processing with 4 workers # Evaluate markdown tables (converted to HTML internally) pred_markdown = """| Name | Age | | --- | --- | | Alice | 25 | | Bob | 30 |""" gt_markdown = """| Name | Age | | --- | --- | | Alice | 25 | | Bob | 30 |""" pred_table_html = convert_markdown_table_to_html(pred_markdown) gt_table_html = convert_markdown_table_to_html(gt_markdown) score = teds.evaluate(pred_table_html, gt_table_html) print(f"Markdown Table TEDS: {score:.3f}") ``` -------------------------------- ### Counting Evaluation (Exact Match) Source: https://context7.com/qywh2023/ocrbench/llms.txt Performs exact match evaluation for counting tasks. It checks if a specific numerical answer (as a string) is present within the predicted text. ```python from eval_scripts.vqa_metric import counting_evaluation # Exact match evaluation predict = "There are 5 words" answers = ["5"] score = counting_evaluation(predict, answers, eval_method="exact match") print(f"Exact Match Score: {score}") # 1 if "5" in predict, else 0 ``` -------------------------------- ### IoU Calculation for Bounding Boxes Source: https://context7.com/qywh2023/ocrbench/llms.txt Computes the Intersection over Union (IoU) score between predicted and ground truth bounding boxes, essential for localization tasks. ```python from eval_scripts.IoUscore_metric import calculate_iou, extract_coordinates # Calculate IoU between predicted and ground truth bounding boxes # Format: [x1, y1, x2, y2] where (x1,y1) is top-left, (x2,y2) is bottom-right pred_box = [50, 50, 150, 150] gt_box = [60, 60, 140, 140] iou = calculate_iou(pred_box, gt_box) print(f"IoU Score: {iou:.3f}") # Output: IoU Score: 0.642 ``` -------------------------------- ### Extract Coordinates from Text Source: https://context7.com/qywh2023/ocrbench/llms.txt Extracts bounding box coordinates from various string formats, including those with labels like 'bbox:' or 'position:', and comma-separated values. ```python from eval_scripts.IoUscore_metric import calculate_iou, extract_coordinates # Extract coordinates from model prediction string model_output = "The text is located at (100, 200, 300, 400) in the image." coords = extract_coordinates(model_output) print(f"Extracted coordinates: {coords}") # Output: [100, 200, 300, 400] ``` ```python from eval_scripts.IoUscore_metric import calculate_iou, extract_coordinates # Handles various formats: outputs = [ "bbox: [50, 100, 200, 300]", "position: (50, 100, 200, 300)", "coordinates are 50,100,200,300" ] for output in outputs: coords = extract_coordinates(output) if coords: print(f"Parsed: {coords}") ``` -------------------------------- ### End-to-End Text Spotting Evaluation (H-mean) Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates text spotting (detection and recognition) using H-mean. Handles various prediction formats and uses IoU threshold of 0.5 for matching. ```python from eval_scripts.spotting_metric import extract_bounding_boxes_robust, spotting_evaluation # Parse model prediction for text spotting # Format: [[x1, y1, x2, y2, "recognized_text"], ...] model_output = """[ [100, 50, 200, 80, "STOP"], [150, 100, 300, 140, "Main Street"], [50, 200, 180, 250, "Coffee Shop"] ]""" pred_boxes = extract_bounding_boxes_robust(model_output) print(f"Extracted {len(pred_boxes)} text instances") for box in pred_boxes: print(f" [{box[0]}, {box[1]}, {box[2]}, {box[3]}]: '{box[4]}'") # Ground truth format for evaluation img_metas = { "bbox": [ [100, 50, 200, 50, 200, 80, 100, 80], # 8-point polygon format [150, 100, 300, 100, 300, 140, 150, 140], [50, 200, 180, 200, 180, 250, 50, 250] ], "content": ["STOP", "Main Street", "Coffee Shop"] } # Evaluate spotting (creates temp files for RRC evaluation) # Uses IoU threshold of 0.5 for matching score = spotting_evaluation(pred_boxes, img_metas) print(f"Text Spotting H-mean: {score:.3f}") # Handle various prediction formats formats = [ "[[10, 20, 100, 50, 'Hello'], [120, 30, 200, 60, 'World']]", "[(10, 20, 100, 50, 'Hello'), (120, 30, 200, 60, 'World')]", "[10, 20, 100, 50, Hello], [120, 30, 200, 60, World]" ] for fmt in formats: boxes = extract_bounding_boxes_robust(fmt) if boxes: print(f"Parsed {len(boxes)} boxes from format variant") ``` -------------------------------- ### Case-Sensitive and ANLS Text Evaluation Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates text predictions against ground truth answers. Case-sensitive evaluation checks for exact matches, while ANLS (>= 0.5) is used for longer answers, computing Normalized Levenshtein similarity. ```python predict_sensitive = "APPVIA" answers_sensitive = ["APPVIA", "appvia"] score_sensitive = vqa_evaluation_case_sensitive(predict_sensitive, answers_sensitive) print(f"Case-Sensitive Score: {score_sensitive}") ``` ```python long_predict = "The quick brown fox jumps over the lazy dog" long_answers = ["The quick brown fox jumps over a lazy dog"] anls_score = vqa_evaluation(long_predict, long_answers) print(f"ANLS Score: {anls_score:.3f}") # Normalized Levenshtein similarity >= 0.5 ``` -------------------------------- ### TEDS Metric for Table Evaluation (HTML) Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates table parsing accuracy using the Tree Edit Distance-based Similarity (TEDS) metric on HTML table structures. It can detect minor content differences. ```python from eval_scripts.TEDS_metric import TEDS, convert_markdown_table_to_html, wrap_html_table # Initialize TEDS evaluator teds = TEDS(n_jobs=4) # Parallel processing with 4 workers # Evaluate HTML tables directly pred_html = """
NameAge
Alice25
Bob30
""" gt_html = """
NameAge
Alice25
Bob31
""" score = teds.evaluate(pred_html, gt_html) print(f"TEDS Score: {score:.3f}") # ~0.9+ for minor content difference ``` -------------------------------- ### Document Parsing Evaluation (STEDS) Source: https://context7.com/qywh2023/ocrbench/llms.txt Evaluates structured document parsing using Structure Tree Edit Distance Similarity (STEDS). Compares predicted and ground truth document structures. ```python from eval_scripts.TEDS_metric import doc_parsing_evaluation, get_tree, STEDS # Model output with markdown-style structure pred_doc = """# Title Introduction paragraph about the document. # Methods Description of methodology used. # Results Key findings and observations. """ gt_doc = """# Title Introduction paragraph about the document. # Methods Detailed methodology description. # Results Key findings and data analysis. """ # Evaluate document structure similarity score = doc_parsing_evaluation(pred_doc, gt_doc) print(f"Document Parsing Score: {score:.3f}") # For debugging, visualize the tree structure pred_tree = get_tree(pred_doc) gt_tree = get_tree(gt_doc) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.