### Install Dependencies and Start vLLM Server Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Install necessary Python packages and start the vLLM API server for HunyuanOCR. Ensure you have vLLM version 0.12.0 or higher. ```bash pip install vllm>=0.12.0 pip install torch torchvision accelerate Pillow numpy tqdm vllm serve tencent/HunyuanOCR \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 \ --gpu-memory-utilization 0.2 ``` -------------------------------- ### Install vLLM and Dependencies Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md Installs the vLLM library and project-specific requirements. Ensure you have the correct CUDA version installed. ```bash pip install vllm>=0.12.0 pip install -r requirements.txt ``` -------------------------------- ### Install vLLM and Dependencies Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README_zh.md Install the vLLM library and project-specific requirements. Ensure CUDA compatibility is set up if needed. ```bash pip install vllm>=0.12.0 pip install -r requirements.txt ``` ```bash sudo dpkg -i cuda-compat-12-9_575.57.08-0ubuntu1_amd64.deb echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.9/compat:$LD_LIBRARY_PATH' >> ~/.bashrc source ~/.bashrc # verify cuda-compat-12-9 ls /usr/local/cuda-12.9/compat ``` -------------------------------- ### Example: Multi-field Information Extraction Prompt Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Example of how to construct a prompt for extracting multiple fields from an image and returning the result in JSON format. ```python # Example: Information extraction from a receipt fields = ['商品名称', '单价', '数量', '总金额', '日期'] prompt = f"提取图片中的: {fields} 的字段内容,并按照JSON格式返回。" ``` -------------------------------- ### Example: Single Field Extraction Prompt Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Example of constructing a prompt to extract the value of a specific field from an image. ```python # Example: Extract specific field prompt = "输出发票号码的值。" ``` -------------------------------- ### Example: Translation Prompt Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Example of a prompt for extracting text from an image and translating it into English, with specific handling for documents, formulas, and tables. ```python # Example: Chinese to English translation prompt = "先提取文字,再将文字内容翻译为英文。若是文档,则其中页眉、页脚忽略。公式用latex格式表示,表格用html格式表示。" ``` -------------------------------- ### OCR Response Processing and Visualization Example Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Demonstrates the usage of `process_spotting_response` and `draw_text_detection_boxes` functions. Loads an image, processes OCR response, and saves the visualized image. ```python # Usage example image = Image.open("/path/to/image.jpg") ocr_response = "Hello(100,50),(300,80)World(100,100),(300,130)" # Denormalize coordinates to actual pixel values processed = process_spotting_response(ocr_response, image.width, image.height) # Visualize detection results result_image = draw_text_detection_boxes(image, processed) result_image.save("visualization.jpg") ``` -------------------------------- ### Install CUDA Compatibility Libraries Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md Installs CUDA compatibility libraries for vLLM. This step is crucial for ensuring proper GPU acceleration. Remember to source your .bashrc after changes. ```bash sudo dpkg -i cuda-compat-12-9_575.57.08-0ubuntu1_amd64.deb echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.9/compat:$LD_LIBRARY_PATH' >> ~/.bashrc source ~/.bashrc # verify cuda-compat-12-9 ls /usr/local/cuda-12.9/compat ``` -------------------------------- ### Install Transformers for HunyuanOCR Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md Install the Hugging Face Transformers library from a specific GitHub commit to use with HunyuanOCR. This is a prerequisite for the Transformers-based inference snippet. ```bash pip install git+https://github.com/huggingface/transformers@82a06db03535c49aa987719ed0746a76093b1ec4 ``` -------------------------------- ### Deploy HunyuanOCR with vLLM Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md Deploys the HunyuanOCR model using vLLM for inference. This command starts a server for the specified model with custom configurations for caching and GPU utilization. ```bash vllm serve tencent/HunyuanOCR \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 \ --gpu-memory-utilization 0.2 ``` -------------------------------- ### HunyuanOCR Inference with Transformers Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md This snippet demonstrates model inference using the Hugging Face Transformers library. It requires installing the transformers library from a specific commit. Note that performance might differ from vLLM. ```python from transformers import AutoProcessor from transformers import HunYuanVLForConditionalGeneration from PIL import Image import torch model_name_or_path = "tencent/HunyuanOCR" processor = AutoProcessor.from_pretrained(model_name_or_path, use_fast=False) img_path = "path/to/your/image.jpg" image_inputs = Image.open(img_path) messages1 = [ {"role": "system", "content": ""}, { "role": "user", "content": [ {"type": "image", "image": img_path}, {"type": "text", "text": ( "检测并识别图片中的文字,将文本坐标格式化输出。" )}, ], } ] messages = [messages1] texts = [ processor.apply_chat_template(msg, tokenize=False, add_generation_prompt=True) for msg in messages ] inputs = processor( text=texts, images=image_inputs, padding=True, return_tensors="pt", ) model = HunYuanVLForConditionalGeneration.from_pretrained( model_name_or_path, attn_implementation="eager", dtype=torch.bfloat16, device_map="auto" ) with torch.no_grad(): device = next(model.parameters()).device inputs = inputs.to(device) generated_ids = model.generate(**inputs, max_new_tokens=16384, do_sample=False) if "input_ids" in inputs: input_ids = inputs.input_ids else: print("inputs: # fallback", inputs) input_ids = inputs.inputs generated_ids_trimmed = [ out_ids[len(in_ids):] for in_ids, out_ids in zip(input_ids, generated_ids) ] output_texts = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(output_texts) ``` -------------------------------- ### Interact with HunyuanOCR via OpenAI-Compatible API Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Use the OpenAI-compatible API endpoint of a deployed vLLM server to interact with HunyuanOCR. This example demonstrates extracting specific fields from a receipt and returning them in JSON format. ```python import json import base64 from openai import OpenAI from tqdm import tqdm def encode_image(image_path: str) -> str: """Encode image file to base64 string""" with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode('utf-8') def create_chat_messages(image_path: str, prompt: str): """Create chat messages with image and prompt""" return [ {"role": "system", "content": ""}, { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{encode_image(image_path)}" } }, {"type": "text", "text": prompt} ] } ] # Initialize OpenAI client pointing to vLLM server client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) # Process a document image_path = "/path/to/receipt.jpg" prompt = "提取图片中的: ['商品名称','单价','总金额'] 的字段内容,并按照JSON格式返回。" messages = create_chat_messages(image_path, prompt) response = client.chat.completions.create( model="tencent/HunyuanOCR", messages=messages, temperature=0.0, top_p=0.95, seed=1234, stream=False, extra_body={ "top_k": 1, "repetition_penalty": 1.0 } ) result = response.choices[0].message.content print(result) # Output: {"商品名称": "Apple iPhone", "单价": "999.00", "总金额": "999.00"} ``` -------------------------------- ### HunyuanOCR Inference with vLLM Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md Use this snippet for efficient model inference with the vLLM framework. Ensure the vLLM library and necessary dependencies are installed. The `clean_repeated_substrings` function is a utility to post-process the model's output. ```python from vllm import LLM, SamplingParams from PIL import Image from transformers import AutoProcessor def clean_repeated_substrings(text): """Clean repeated substrings in text""" n = len(text) if n<8000: return text for length in range(2, n // 10 + 1): candidate = text[-length:] count = 0 i = n - length while i >= 0 and text[i:i + length] == candidate: count += 1 i -= length if count >= 10: return text[:n - length * (count - 1)] return text model_path = "tencent/HunyuanOCR" llm = LLM(model=model_path, trust_remote_code=True) processor = AutoProcessor.from_pretrained(model_path) sampling_params = SamplingParams(temperature=0, max_tokens=16384) img_path = "/path/to/image.jpg" img = Image.open(img_path) messages = [ {"role": "system", "content": ""}, {"role": "user", "content": [ {"type": "image", "image": img_path}, {"type": "text", "text": "检测并识别图片中的文字,将文本坐标格式化输出。"} ]} ] prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = {"prompt": prompt, "multi_modal_data": {"image": [img]}} output = llm.generate([inputs], sampling_params)[0] print(clean_repeated_substrings(output.outputs[0].text)) ``` -------------------------------- ### Run HunyuanOCR Demo Script (vLLM) Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md Execute the provided demo script for HunyuanOCR using the vLLM framework. Navigate to the specified directory and run the Python script. ```shell cd Hunyuan-OCR-master/Hunyuan-OCR-vllm && python run_hy_ocr.py ``` -------------------------------- ### Run HunyuanOCR Demo Script (Transformers) Source: https://github.com/tencent-hunyuan/hunyuanocr/blob/main/README.md Execute the provided demo script for HunyuanOCR using the Hugging Face Transformers framework. Navigate to the specified directory and run the Python script. ```shell cd Hunyuan-OCR-master/Hunyuan-OCR-hf && python run_hy_ocr.py ``` -------------------------------- ### Text Spotting with vLLM Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Perform text spotting on an image using the vLLM framework. This script detects text, recognizes it, and outputs bounding box coordinates along with the text content. ```python from vllm import LLM, SamplingParams from PIL import Image from transformers import AutoProcessor def clean_repeated_substrings(text): """Clean repeated substrings in text output""" n = len(text) if n < 8000: return text for length in range(2, n // 10 + 1): candidate = text[-length:] count = 0 i = n - length while i >= 0 and text[i:i + length] == candidate: count += 1 i -= length if count >= 10: return text[:n - length * (count - 1)] return text # Initialize model model_path = "tencent/HunyuanOCR" llm = LLM(model=model_path, trust_remote_code=True) processor = AutoProcessor.from_pretrained(model_path) sampling_params = SamplingParams(temperature=0, max_tokens=16384) # Load image and create request img_path = "/path/to/document.jpg" img = Image.open(img_path) messages = [ {"role": "system", "content": ""}, {"role": "user", "content": [ {"type": "image", "image": img_path}, {"type": "text", "text": "检测并识别图片中的文字,将文本坐标格式化输出。"} ]} ] prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = {"prompt": prompt, "multi_modal_data": {"image": [img]}} # Run inference output = llm.generate([inputs], sampling_params)[0] result = clean_repeated_substrings(output.outputs[0].text) print(result) # Output format: text(x1,y1),(x2,y2) where coordinates are normalized to [0,1000] # Example: "Hello World(100,50),(300,80)" ``` -------------------------------- ### Load Image from URL or Local Path Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Utility function to load an image from either a web URL or a local file path. Handles HTTP requests and file opening. ```python from PIL import Image import requests from io import BytesIO def get_image(input_source): """Load image from URL or local path""" if input_source.startswith(('http://', 'https://')): response = requests.get(input_source) response.raise_for_status() return Image.open(BytesIO(response.content)) else: return Image.open(input_source) ``` -------------------------------- ### Process Batch Documents from JSONL File Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Processes a batch of documents from a JSONL file, sending each image to the HunyuanOCR model via the OpenAI client and saving the OCR results to an output JSONL file. Includes error handling and progress tracking. ```python import json from openai import OpenAI from tqdm import tqdm def process_batch_jsonl(input_path: str, output_path: str, prompt: str): """Process a batch of images from JSONL file""" client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) with open(input_path, "r", encoding="utf-8") as fin, \ open(output_path, "w", encoding="utf-8") as fout: for line in tqdm(fin, desc="Processing documents"): if not line.strip(): continue try: data = json.loads(line) img_path = data['image_path'] messages = [ {"role": "system", "content": ""}, { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encode_image(img_path)}"} }, {"type": "text", "text": prompt} ] } ] response = client.chat.completions.create( model="tencent/HunyuanOCR", messages=messages, temperature=0.0, extra_body={"top_k": 1, "repetition_penalty": 1.0} ) data["ocr_result"] = response.choices[0].message.content fout.write(json.dumps(data, ensure_ascii=False) + "\n") except Exception as e: print(f"Error: {str(e)}") continue print(f"Results saved to: {output_path}") # Usage process_batch_jsonl( input_path="documents.jsonl", output_path="ocr_results.jsonl", prompt="提取文档图片中正文的所有信息用markdown格式表示。" ) ``` -------------------------------- ### Draw Text Detection Boxes on Image Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Visualizes OCR detection results by drawing bounding boxes and labels directly onto an image. Requires a PIL Image object and the processed OCR response string. ```python def draw_text_detection_boxes(image: Image, response: str) -> Image: """Draw text detection boxes on image with labels""" img_draw = image.copy() draw = ImageDraw.Draw(img_draw) font = ImageFont.load_default() pattern = r'([^()]+)(\(\d+,\d+\),\(\d+,\d+\))' matches = re.finditer(pattern, response) for match in matches: text = match.group(1).strip() coords = match.group(2) coord_pattern = r'\((\d+),(\d+)\)' coord_matches = re.findall(coord_pattern, coords) if len(coord_matches) == 2: x1, y1 = int(coord_matches[0][0]), int(coord_matches[0][1]) x2, y2 = int(coord_matches[1][0]), int(coord_matches[1][1]) color = (np.random.randint(0, 200), np.random.randint(0, 200), np.random.randint(0, 255)) draw.rectangle([x1, y1, x2, y2], outline=color, width=2) draw.text((x1, max(0, y1 - 15)), text, font=font, fill=color) return img_draw ``` -------------------------------- ### HunyuanOCR Task-Specific Prompts Dictionary Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt A dictionary containing predefined prompts for various OCR tasks, including text spotting, formula recognition, table parsing, chart parsing, full document parsing, general text extraction, single/multi-field extraction, subtitle extraction, and translation. ```python # Task-specific prompts dictionary TASK_PROMPTS = { # Text Spotting - detect and locate all text with coordinates "spotting": "检测并识别图片中的文字,将文本坐标格式化输出。", # Formula Recognition - parse mathematical formulas to LaTeX "formula": "识别图片中的公式,用LaTeX格式表示。", # Table Parsing - convert tables to HTML "table": "把图中的表格解析为 HTML。", # Chart Parsing - flowcharts to Mermaid, others to Markdown "chart": "解析图中的图表,对于流程图使用Mermaid格式表示,其他图表使用Markdown格式表示。", # Full Document Parsing - comprehensive markdown output "document": ( "提取文档图片中正文的所有信息用markdown格式表示," "其中页眉、页脚部分忽略,表格用html格式表达," "文档中公式用latex格式表示,按照阅读顺序组织进行解析。" ), # General Text Extraction "general": "提取图中的文字。", # Single Field Extraction "single_field": "输出{field_name}的值。", # Multi-field Information Extraction (JSON output) "extraction": "提取图片中的: {field_list} 的字段内容,并按照JSON格式返回。", # Video Subtitle Extraction "subtitle": "提取图中的字幕", # Translation - extract text and translate "translation": ( "先提取文字,再将文字内容翻译为英文。" "若是文档,则其中页眉、页脚忽略。" "公式用latex格式表示,表格用html格式表示。" ) } ``` -------------------------------- ### Find Best Matching Candidate Formula Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Finds the best matching predicted formula from a list of candidates against a ground truth formula using normalized sequence matching. Returns the best matching formula and its similarity score. ```python from difflib import SequenceMatcher def best_match(gt: str, candidates: list): """Find best matching candidate using sequence matching""" gt_norm = normalize_for_match(gt) best_score = -1 best_cand = None for cand in candidates: cand_norm = normalize_for_match(cand) score = SequenceMatcher(None, gt_norm, cand_norm).ratio() if score > best_score: best_score = score best_cand = cand return best_cand, best_score # Example: Match formula prediction to ground truth gt_formula = r"\\frac{a+b}{c} = d" predicted_formulas = [ r"\\frac{a + b}{c} = d", r"\\frac{a-b}{c} = d", r"a + b = c" ] best, score = best_match(gt_formula, predicted_formulas) print(f"Best match: {best} (score: {score:.3f})") # Output: Best match: \frac{a + b}{c} = d (score: 0.952) ``` -------------------------------- ### Process OCR Spotting Response with Denormalized Coordinates Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Processes a raw OCR spotting response string, replacing normalized coordinates with denormalized pixel coordinates. Requires image dimensions for conversion. ```python def process_spotting_response(response: str, image_width: int, image_height: int) -> str: """Process spotting task response and denormalize coordinates""" pattern = r'([^()]+)(\(\d+,\d+\),\(\d+,\d+\))' matches = re.finditer(pattern, response) new_response = response for match in matches: coords = match.group(2) coord_pattern = r'\((\d+),(\d+)\)' coord_matches = re.findall(coord_pattern, coords) if len(coord_matches) == 2: start_coord = (float(coord_matches[0][0]), float(coord_matches[0][1])) end_coord = (float(coord_matches[1][0]), float(coord_matches[1][1])) denorm_start = denormalize_coordinates(start_coord, image_width, image_height) denorm_end = denormalize_coordinates(end_coord, image_width, image_height) new_coords = f"({denorm_start[0]},{denorm_start[1]}),({denorm_end[0]},{denorm_end[1]})") new_response = new_response.replace(coords, new_coords) return new_response ``` -------------------------------- ### HunyuanOCR Document Parsing with Transformers Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Performs document parsing using HunyuanOCR and HuggingFace Transformers. It loads the model, prepares inputs with a specific prompt for markdown output, generates text, and cleans the output. ```python from transformers import AutoProcessor, HunYuanVLForConditionalGeneration import torch # Load model and processor model_name_or_path = "tencent/HunyuanOCR" processor = AutoProcessor.from_pretrained(model_name_or_path, use_fast=False) model = HunYuanVLForConditionalGeneration.from_pretrained( model_name_or_path, attn_implementation="eager", dtype=torch.bfloat16, device_map="auto" ) # Prepare input for document parsing img_path = "https://example.com/document.png" image_inputs = get_image(img_path) messages = [ {"role": "system", "content": ""}, { "role": "user", "content": [ {"type": "image", "image": img_path}, {"type": "text", "text": ( "提取文档图片中正文的所有信息用markdown格式表示," "其中页眉、页脚部分忽略,表格用html格式表达," "文档中公式用latex格式表示,按照阅读顺序组织进行解析。" )}, ], } ] # Process and generate texts = [processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)] inputs = processor(text=texts, images=image_inputs, padding=True, return_tensors="pt") with torch.no_grad(): device = next(model.parameters()).device inputs = inputs.to(device) generated_ids = model.generate(**inputs, max_new_tokens=16384, do_sample=False) # Decode output input_ids = inputs.input_ids if "input_ids" in inputs else inputs.inputs generated_ids_trimmed = [ out_ids[len(in_ids):] for in_ids, out_ids in zip(input_ids, generated_ids) ] output_texts = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) result = clean_repeated_substrings(output_texts[0]) print(result) ``` -------------------------------- ### Denormalize Coordinates to Image Dimensions Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Converts normalized coordinates (ranging from 0 to 1000) to actual pixel coordinates based on image width and height. ```python def denormalize_coordinates(coord: Tuple[float, float], image_width: int, image_height: int) -> Tuple[int, int]: """Denormalize coordinates from [0,1000] to image dimensions""" x, y = coord denorm_x = int(x * image_width / 1000) denorm_y = int(y * image_height / 1000) return (denorm_x, denorm_y) ``` -------------------------------- ### Normalize LaTeX Big Brace Commands Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Normalizes LaTeX commands for big braces like \\big{}, \\Big{}, \\bigg{}, and \\Bigg{}. Use this to simplify brace notation before matching. ```python import re def remove_big_braces(s: str): """Normalize LaTeX big brace commands""" pattern = r'\\(big|Big|bigg|Bigg)\{([^\}]+)\}' repl = r'\\\1\2' return re.sub(pattern, repl, s) ``` -------------------------------- ### Encode Image to Base64 Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Encodes an image file into a base64 string, which is required for sending image data in API requests. Ensure the image path is correct. ```python import base64 def encode_image(image_path: str) -> str: with open(image_path, "rb") as f: return base64.b64encode(f.read()).decode('utf-8') ``` -------------------------------- ### Parse Coordinate String to Tuple Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Parses a coordinate string in the format '(x,y)' and returns a tuple of floats. Returns (0,0) on parsing errors. ```python import re import numpy as np from PIL import Image, ImageDraw, ImageFont from typing import Tuple def parse_coords(coord_str: str) -> Tuple[float, float]: """Parse coordinate string and return (x,y) tuple""" try: x, y = coord_str.strip('()').split(',') return (float(x), float(y)) except: return (0, 0) ``` -------------------------------- ### Normalize Formulas for OmniDocBench Evaluation Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Normalizes both ground truth and predicted formulas for OmniDocBench evaluation. It cleans trailing equation numbers, removes LaTeX display math delimiters, and standardizes circled letter representations. ```python import re def norm_formula_HYOCR(gt_form: str, pre_form: str): """ Normalize formulas for OmniDocBench evaluation. Handles HunyuanOCR's flexible output granularity. """ pre_form = clean_gt_tail(pre_form) gt_form = gt_form.replace('\\[', '').replace('\\]', '').replace(',', ',').strip() pre_form = pre_form.replace('\\[', '').replace('\\]', '').replace(',', ',').strip() pre_form = ( pre_form .replace("\\text{ⓐ}", "\\textcircled{a}") .replace("\\text{ⓑ}", "\\textcircled{b}") .replace("\\text{ⓒ}", "\\textcircled{c}") .replace("\\text{ⓓ}", "\\textcircled{d}") ) return gt_form, pre_form ``` -------------------------------- ### Clean Trailing Equation Numbers from Ground Truth Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Removes trailing equation numbers, often enclosed in parentheses and preceded by \\quad or \\qquad, from ground truth formula strings. This ensures accurate matching by ignoring extraneous numbering. ```python import re def clean_gt_tail(gt: str): """Remove trailing equation numbers from ground truth""" pattern = r'(\\quad+|\\qquad+)\s*\{?\(\s*\d+\s*\)\}?\s*$' return re.sub(pattern, '', gt).rstrip() ``` -------------------------------- ### Normalize Text for Formula Matching Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Normalizes formula strings by replacing circled letters (e.g., \\textcircled{a}) with their Unicode equivalents (e.g., ⓐ) and removing all spaces. This is crucial for accurate sequence matching. ```python import re def normalize_for_match(text: str): """Normalize text for formula matching""" text = re.sub(r"\\textcircled\{a\}", "ⓐ", text) text = re.sub(r"\\textcircled\{b\}", "ⓑ", text) text = re.sub(r"\\textcircled\{c\}", "ⓒ", text) text = re.sub(r"\\textcircled\{d\}", "ⓓ", text) text = text.replace("\\text{ⓐ}", "ⓐ").replace("\\text{ⓑ}", "ⓑ") text = text.replace("\\text{ⓒ}", "ⓒ").replace("\\text{ⓓ}", "ⓓ") text = text.replace(" ", "") return text ``` -------------------------------- ### Clean Repeated Substrings in Text Source: https://context7.com/tencent-hunyuan/hunyuanocr/llms.txt Removes repeating substrings from the end of a text if they appear consecutively more than a threshold number of times. Useful for cleaning model outputs. ```python def clean_repeated_substrings(text): """Clean repeated substrings in text""" n = len(text) if n < 8000: return text for length in range(2, n // 10 + 1): candidate = text[-length:] count = 0 i = n - length while i >= 0 and text[i:i + length] == candidate: count += 1 i -= length if count >= 10: return text[:n - length * (count - 1)] return text ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.