### Install Python Requirements Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Installs all Python dependencies listed in the requirements.txt file using pip. ```bash pip install -r requirements.txt ``` -------------------------------- ### Install ImageMagick from Source Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Clones the ImageMagick repository, configures, builds, and installs the software from source code to ensure a specific version is used. ```bash git clone https://github.com/ImageMagick/ImageMagick.git ImageMagick-7.1.1 cd ImageMagick-7.1.1 ./configure make sudo make install sudo ldconfig /usr/local/lib convert --version ``` -------------------------------- ### Install Full TeX Live Distribution Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Installs the complete TeX Live distribution, which includes LaTeX and related packages required for rendering PDF files. ```bash apt-get update sudo apt-get install texlive-full ``` -------------------------------- ### Install Node.js v16.13.1 Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Installs Node.js version 16.13.1 from a tarball and creates symbolic links for the node and npm executables. ```bash wget https://registry.npmmirror.com/-/binary/node/latest-v16.x/node-v16.13.1-linux-x64.tar.gz tar -xvf node-v16.13.1-linux-x64.tar.gz mv node-v16.13.1-linux-x64/* /usr/local/nodejs/ ln -s /usr/local/nodejs/bin/node /usr/local/bin ln -s /usr/local/nodejs/bin/npm /usr/local/bin node -v ``` -------------------------------- ### Install Dependencies and Download Model Weights Source: https://github.com/opendatalab/unimernet/blob/main/MFD/README.md Sets up a conda environment, installs the ultralytics package, and downloads model weights from ModelScope. Ensure you are in the MFD directory before downloading weights. ```bash conda create -n mfd python=3.10 conda activate mfd pip install ultralytics # download with modelscope cd MFD/ wget -c https://www.modelscope.cn/models/wanderkid/PDF-Extract-Kit/resolve/master/models/MFD/weights.pt # you can also download with huggingface # https://huggingface.co/wanderkid/PDF-Extract-Kit/blob/main/models/MFD/weights.pt ``` -------------------------------- ### Run UniMERNet GUI Source: https://github.com/opendatalab/unimernet/blob/main/README.md Launch the Streamlit-based graphical user interface for interactive formula recognition. Ensure UniMERNet is installed with full dependencies. ```bash unimernet_gui ``` -------------------------------- ### Launch Gradio Demo for CDM Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Starts the local Gradio-based web demo for interacting with the CDM evaluation tool. ```python python app.py ``` -------------------------------- ### Local Installation of UniMERNet Source: https://github.com/opendatalab/unimernet/blob/main/README.md Install UniMERNet from the local source code using pip in editable mode. This is recommended for developers. ```bash pip install -e ."[full]" ``` -------------------------------- ### Install UniMERNet via Pip Source: https://github.com/opendatalab/unimernet/blob/main/README.md Install UniMERNet and its full dependencies using pip. This method is recommended for general users. ```bash pip install -U "unimernet[full]" ``` -------------------------------- ### Run UniMERNet Training Script Source: https://github.com/opendatalab/unimernet/blob/main/README.md Execute the training script to start the model training process. Ensure the dataset path is correctly configured. ```bash bash script/train.sh ``` -------------------------------- ### Start CDM Docker Container Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Starts an interactive bash session within the 'cdm:latest' Docker container. Optionally, volume mapping can be added to persist data. ```bash docker run -it cdm bash ``` -------------------------------- ### Download UniMERNet Models Source: https://github.com/opendatalab/unimernet/blob/main/README.md Download pre-trained UniMERNet models (base, small, tiny) and tokenizers from Hugging Face or ModelScope. Ensure git-lfs is installed. ```bash cd UniMERNet/models git lfs install git clone https://huggingface.co/wanderkid/unimernet_base # 1.3GB git clone https://huggingface.co/wanderkid/unimernet_small # 773MB git clone https://huggingface.co/wanderkid/unimernet_tiny # 441MB # you can also download the model from ModelScope git clone https://www.modelscope.cn/wanderkid/unimernet_base.git git clone https://www.modelscope.cn/wanderkid/unimernet_small.git git clone https://www.modelscope.cn/wanderkid/unimernet_tiny.git ``` -------------------------------- ### CDM Input JSON Format Example Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Example structure for the input JSON file used by the CDM evaluation script. Note the escaping requirement for special characters like backslashes. ```json [ { "img_id": "case_1", # optional key "gt": "y = 2z + 3x", "pred": "y = 2x + 3z" }, { "img_id": "case_2", "gt": "y = x^2 + 1", "pred": "y = x^2 + 1" }, ... ] `Note that in json files, some special characters such as \" need escaped character, for example \"\begin\" should be written as \\\\begin\".` ``` -------------------------------- ### Include KaTeX CSS and JavaScript from CDN Source: https://github.com/opendatalab/unimernet/blob/main/cdm/modules/tokenize_latex/third_party/katex/README.md Include these files to use KaTeX for rendering math on your web page. This is a common setup for front-end usage. ```html ``` -------------------------------- ### Run the Demo Source: https://github.com/opendatalab/unimernet/blob/main/MFD/README.md Executes the project's demonstration script. ```bash python demo.py ``` -------------------------------- ### Run UniMERNet Jupyter Notebook Demo Source: https://github.com/opendatalab/unimernet/blob/main/README.md Open and run the Jupyter Notebook for formula recognition and rendering from an image. ```bash jupyter-lab ./demo.ipynb ``` -------------------------------- ### Instantiate ImageProcessor Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Initializes the ImageProcessor with configuration and image directory paths. Ensures correct paths are set for model loading and image processing. ```python root_path = os.path.abspath(os.getcwd()) config_path = os.path.join(root_path, "configs/demo.yaml") image_directory = os.path.join(root_path, "asset/test_imgs") processor = ImageProcessor(config_path, image_directory) ``` -------------------------------- ### Download UniMER-1M Dataset Path Source: https://github.com/opendatalab/unimernet/blob/main/README.md Specify the directory for the UniMER-1M dataset. This is required before training. ```bash ./data/UniMER-1M ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/opendatalab/unimernet/blob/main/README.md Set up a dedicated Conda environment for UniMERNet with Python 3.10. ```bash conda create -n unimernet python=3.10 conda activate unimernet ``` -------------------------------- ### Model Initialization Output Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Logs indicating the successful initialization of various model components, including CustomVisionEncoderDecoderModel, VariableUnimerNetModel, and CustomMBartForCausalLM. ```text CustomVisionEncoderDecoderModel init VariableUnimerNetModel init VariableUnimerNetPatchEmbeddings init VariableUnimerNetModel init VariableUnimerNetPatchEmbeddings init CustomMBartForCausalLM init CustomMBartDecoder init ``` -------------------------------- ### Download UniMER-Test Dataset Path Source: https://github.com/opendatalab/unimernet/blob/main/README.md Specify the directory for the UniMER-Test dataset. This is required for evaluation. ```bash ./data/UniMER-Test ``` -------------------------------- ### Generation Configuration Warnings Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb User warnings related to generation configuration, specifically when 'do_sample' is set to False while 'temperature' or 'top_p' are also set. These flags are only used in sample-based generation modes. ```text /Users/bin/anaconda3/envs/unimernetv2_pip/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:540: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.2` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. warnings.warn( /Users/bin/anaconda3/envs/unimernetv2_pip/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:545: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. warnings.warn( ``` -------------------------------- ### Display Sample Image Information Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Displays a table with sample ID and the corresponding image path. This is used to present the input data for processing. ```text Result: ``` -------------------------------- ### Build CDM Docker Image Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Builds a Docker image tagged as 'cdm:latest' from the provided Dockerfile in the current directory. ```bash docker build -f DockerFile -t cdm:latest . ``` -------------------------------- ### Run UniMERNet Test Script Source: https://github.com/opendatalab/unimernet/blob/main/README.md Execute the test script to evaluate the trained model. Ensure the test dataset path is correctly configured in `configs/val` and `test.py`. ```bash bash script/test.sh ``` -------------------------------- ### Render TeX to String Source: https://github.com/opendatalab/unimernet/blob/main/cdm/modules/tokenize_latex/third_party/katex/README.md Use `katex.renderToString` to generate an HTML string of the rendered math, suitable for server-side rendering or dynamic content insertion. Throws a `katex.ParseError` for invalid expressions. ```js var html = katex.renderToString("c = \pm\sqrt{a^2 + b^2}"); // '...' ``` -------------------------------- ### Clone UniMERNet Repository Source: https://github.com/opendatalab/unimernet/blob/main/README.md Clone the official UniMERNet GitHub repository to your local machine. ```bash git clone https://github.com/opendatalab/UniMERNet.git ``` -------------------------------- ### Render TeX with Display Mode Option Source: https://github.com/opendatalab/unimernet/blob/main/cdm/modules/tokenize_latex/third_party/katex/README.md Render a TeX expression in display mode by passing `{ displayMode: true }` as an option to `katex.render`. This centers the math on its own line and uses display style for elements like integrals. ```js katex.render("c = \pm\sqrt{a^2 + b^2}", element, { displayMode: true }); ``` -------------------------------- ### Run CDM Batch Evaluation Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Executes the CDM evaluation script using a prepared input JSON file. ```python python evaluation.py -i {path_to_your_input_json} ``` -------------------------------- ### katex.renderToString Source: https://github.com/opendatalab/unimernet/blob/main/cdm/modules/tokenize_latex/third_party/katex/README.md Generates an HTML string of the rendered math expression, suitable for server-side rendering or when direct DOM manipulation is not desired. It also supports rendering options and throws a ParseError for invalid expressions. ```APIDOC ## katex.renderToString ### Description Generates an HTML string of the rendered math expression, suitable for server-side rendering or when direct DOM manipulation is not desired. It also supports rendering options and throws a ParseError for invalid expressions. ### Method ```javascript katex.renderToString(texExpression, options?) ``` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **texExpression** (string) - Required - The LaTeX math expression to render. - **options** (object) - Optional - An object containing rendering options. - **displayMode** (boolean) - If true, renders in display mode (centered, larger math). Default: false. - **throwOnError** (boolean) - If true, throws a ParseError on unsupported commands. Default: true. - **errorColor** (string) - Color for unsupported commands when throwOnError is false. Format: "#XXX" or "#XXXXXX". Default: "#cc0000". ### Request Example ```javascript var html = katex.renderToString("c = \\pm\\sqrt{a^2 + b^2}"); // '...' ``` ### Response #### Success Response (200) - **html** (string) - The rendered math expression as an HTML string. #### Response Example ```json { "html": "..." } ``` ### Error Handling - Throws `katex.ParseError` if the `texExpression` is invalid and `throwOnError` is true. ``` -------------------------------- ### katex.render Source: https://github.com/opendatalab/unimernet/blob/main/cdm/modules/tokenize_latex/third_party/katex/README.md Renders a LaTeX math expression into a specified DOM element for in-browser display. It supports various rendering options and throws a ParseError if the expression is invalid. ```APIDOC ## katex.render ### Description Renders a LaTeX math expression into a specified DOM element for in-browser display. It supports various rendering options and throws a ParseError if the expression is invalid. ### Method ```javascript katex.render(texExpression, element, options?) ``` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **texExpression** (string) - Required - The LaTeX math expression to render. - **element** (DOMElement) - Required - The DOM element to render the math into. - **options** (object) - Optional - An object containing rendering options. - **displayMode** (boolean) - If true, renders in display mode (centered, larger math). Default: false. - **throwOnError** (boolean) - If true, throws a ParseError on unsupported commands. Default: true. - **errorColor** (string) - Color for unsupported commands when throwOnError is false. Format: "#XXX" or "#XXXXXX". Default: "#cc0000". ### Request Example ```javascript katex.render("c = \\pm\\sqrt{a^2 + b^2}", element, { displayMode: true }); ``` ### Response None (renders directly into the DOM element). #### Success Response None #### Response Example None ### Error Handling - Throws `katex.ParseError` if the `texExpression` is invalid and `throwOnError` is true. ``` -------------------------------- ### Process All Images in Directory Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Processes all images within a specified directory. Uncomment the line to enable batch processing. ```python # Uncomment the following line to process all images in the specified directory # processor.process_images() ``` -------------------------------- ### Rendered Image from LaTeX Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Represents a rendered image generated from LaTeX code. This is typically displayed using IPython's Math object. ```text Result: ``` -------------------------------- ### Render TeX to DOM Element Source: https://github.com/opendatalab/unimernet/blob/main/cdm/modules/tokenize_latex/third_party/katex/README.md Use `katex.render` to convert a TeX expression into a rendered math element within a specified DOM element. Throws a `katex.ParseError` if the expression is invalid. ```js katex.render("c = \pm\sqrt{a^2 + b^2}", element); ``` -------------------------------- ### Process Single Image Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Processes a single image file located at a specified path. Use this for individual image analysis. ```python # Process a single image located at the specified path processor.process_single_image(os.path.join(image_directory, '0000001.png')) ``` -------------------------------- ### ImageProcessor Class Definition Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Defines the ImageProcessor class for handling image-to-LaTeX conversion. It loads models and processors, processes images, and renders results. ```python import argparse import os import random import sys from IPython.display import display, Math from PIL import Image from rich import print as rprint from rich.panel import Panel from rich.rule import Rule from rich.table import Table from termcolor import colored import torch sys.path.insert(0, os.path.join(os.getcwd(), "..")) from unimernet.common.config import Config from unimernet.datasets.builders import * from unimernet.models import * from unimernet.processors import * import unimernet.tasks as tasks from unimernet.processors import load_processor class ImageProcessor: def __init__(self, cfg_path, image_dir): self.cfg_path = cfg_path self.image_dir = image_dir self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") self.model, self.vis_processor = self.load_model_and_processor() def load_model_and_processor(self): args = argparse.Namespace(cfg_path=self.cfg_path, options=None) cfg = Config(args) task = tasks.setup_task(cfg) model = task.build_model(cfg).to(self.device) vis_processor = load_processor('formula_image_eval', cfg.config.datasets.formula_rec_eval.vis_processor.eval) return model, vis_processor def process_single_image(self, image_path): try: raw_image = Image.open(image_path) except IOError: print(f"Error: Unable to open image at {image_path}") return resized_image = self.resize_image(raw_image) image = self.vis_processor(raw_image).unsqueeze(0).to(self.device) output = self.model.generate({"image": image}) pred = output["pred_str"][0] self.print_result(0, image_path, resized_image, pred) rprint(Rule(style="black")) def process_images(self): image_names = os.listdir(self.image_dir) image_paths = [os.path.join(self.image_dir, name) for name in image_names] for id, image_path in enumerate(image_paths): raw_image = Image.open(image_path) resized_image = self.resize_image(raw_image) image = self.vis_processor(raw_image).unsqueeze(0).to(self.device) output = self.model.generate({"image": image}) pred = output["pred_str"][0] self.print_result(id, image_path, resized_image, pred) rprint(Rule(style="black")) @staticmethod def resize_image(image, max_len=600): width, height = image.size if max(width, height) > max_len : if width > height: scale = float(max_len) / width width = max_len height = int(height * scale) else: scale = float(max_len) / height height = max_len width = int(width * scale) return image.resize((width, height)) @staticmethod def print_result(id, image_path, raw_image, pred): colors = ['red', 'green', 'yellow', 'blue', 'magenta', 'cyan'] chosen_color = random.choice(colors) table = Table(show_header=True, header_style=chosen_color) table.add_column("Sample ID", style="dim", width=12) table.add_column("Image Path", style="dim", width=80) table.add_row(str(id), image_path) rprint(table) print(colored(f"{id}_1: Source image", chosen_color), end=" ") display(raw_image) print(colored(f'${id}_2: Rendered image from LaTeX', chosen_color), end=" ") render_katex(pred) print(colored(f'${id}_3: Predicted LaTeX code', chosen_color), end=" ") pred_text_panel = Panel.fit(pred, title="Predicted LaTeX", border_style=chosen_color) rprint(pred_text_panel) def render_katex(latex_string, show=True): display(Math(latex_string)) ``` -------------------------------- ### Predicted LaTeX Code Output Source: https://github.com/opendatalab/unimernet/blob/main/demo.ipynb Displays the predicted LaTeX code generated by the model. The output is formatted within a bordered box for clarity. ```text Result: ╭─ ─────────────────────────────────────────────── Predicted LaTeX ─────────────────────────────────────────────── ╮ │ \begin{array} { r l } { \mathrm { M i n i m i s e ~ } } & { { } J ( u . ; s , y ) = \mathbb { E } [ \int _ │ │ { s } ^ { T } \left( u _ { t } ^ { 2 } + 1 \right) d t - \ln \left( \cosh \left( X _ { T } \right) \right) │ │ \right] } \\ { \mathrm { s u b j e c t ~ t o ~ } } & { { } \left\{ \begin{array} { l l } { d X _ { t } = 2 u _ │ │ { t } d t + \sqrt { 2 } d W _ { t } , t \in [ s , T ] } \\ { X _ { s } = y } \\ { u _ { t } \in [ - 1 , 1 ] , │ │ \quad t \in [ s , T ] } \end{array} \right. } \end{array} │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ``` -------------------------------- ### Convert UniMERNet Predictions to CDM JSON Format Source: https://github.com/opendatalab/unimernet/blob/main/cdm/README.md Converts UniMERNet prediction results into the JSON format required by the CDM evaluation script. ```python python convert2cdm_format.py -i {UniMERNet predictions} -o {save path} ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.