### Run the Marker API server Source: https://github.com/datalab-to/marker/blob/master/README.md Install the necessary dependencies and start the server on a specified port. ```shell pip install -U uvicorn fastapi python-multipart marker_server --port 8001 ``` -------------------------------- ### Setup olmOCR-bench Environment Source: https://github.com/datalab-to/marker/blob/master/benchmarks/README.md Commands to clone the olmOCR repository, install dependencies, and download the necessary benchmark datasets. ```bash git clone https://github.com/allenai/olmocr cd olmocr && pip install -e . # download the bench PDFs + test jsonls (see the olmocr repo for the current # command; it populates olmocr/bench/bench_data/) python -m olmocr.bench.download ``` -------------------------------- ### Install Marker Source: https://github.com/datalab-to/marker/blob/master/README.md Standard installation command for the Marker library. ```shell pip install marker-pdf ``` -------------------------------- ### Install Marker with Full Dependencies Source: https://github.com/datalab-to/marker/blob/master/README.md Installation command for users requiring support for non-PDF document formats. ```shell pip install marker-pdf[full] ``` -------------------------------- ### PdfConverter Usage Examples Source: https://github.com/datalab-to/marker/blob/master/_autodocs/pdf-converter.md Demonstrates basic initialization, configuration with custom settings, and specifying custom processors and renderers. ```python from marker.converters.pdf import PdfConverter from marker.models import create_model_dict # Basic initialization artifact_dict = create_model_dict() converter = PdfConverter(artifact_dict=artifact_dict) # With custom configuration config = { "mode": "balanced", "output_format": "json", "use_llm": True, } converter = PdfConverter( artifact_dict=artifact_dict, config=config, ) # With custom processors and renderer converter = PdfConverter( artifact_dict=artifact_dict, processor_list=[ "marker.processors.text.TextProcessor", "marker.processors.table.TableProcessor", ], renderer="marker.renderers.json.JSONRenderer", ) ``` -------------------------------- ### Run the Marker Interactive App Source: https://github.com/datalab-to/marker/blob/master/README.md Install the required dependencies and launch the Streamlit-based GUI for Marker. ```shell pip install -U streamlit streamlit-ace marker_gui ``` -------------------------------- ### Install llama.cpp for CPU/Apple Silicon Source: https://github.com/datalab-to/marker/blob/master/README.md Prerequisite installation for running inference on CPU or Apple Silicon hardware. ```shell brew install llama.cpp # macOS # or grab a release from https://github.com/ggml-org/llama.cpp/releases ``` -------------------------------- ### Create Model Dictionary Examples Source: https://github.com/datalab-to/marker/blob/master/_autodocs/models-and-inference.md Demonstrates initializing models with default settings, specific backends, or custom inference managers. ```python from marker.models import create_model_dict # Create models with default settings (auto-detects backend) models = create_model_dict() # Create models with specific backend models = create_model_dict(inference_backend='vllm') # Use with a custom inference manager from surya.inference import SuryaInferenceManager manager = SuryaInferenceManager(method='llamacpp') models = create_model_dict(inference_manager=manager) ``` -------------------------------- ### Use HTML output format Source: https://github.com/datalab-to/marker/blob/master/_autodocs/output-formats.md Example of converting a document to HTML and saving it to a file. ```python rendered = converter("document.pdf") # With output_format="html" with open("output.html", "w") as f: f.write(rendered.html) ``` -------------------------------- ### Configure PDF Converter with OpenAI Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Example of initializing a PdfConverter with OpenAI service settings and custom configuration parameters. ```python from marker.converters.pdf import PdfConverter from marker.models import create_model_dict # High-quality processing with OpenAI converter = PdfConverter( artifact_dict=create_model_dict(), config={ "use_llm": True, "output_format": "json", "openai_api_key": "sk-...", "openai_model": "gpt-4-turbo", "timeout": 60, # Longer timeout for complex documents "max_retries": 3, # More retries }, llm_service="marker.services.openai.OpenAIService" ) rendered = converter("complex_document.pdf") ``` -------------------------------- ### Complex Class Diagram Setup Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Imports necessary classes for a more complex example involving PokerHand, Hand, and Deck. It initializes Lumpy and creates instances of these classes. ```python from swampy.Lumpy import Lumpy from PokerHand import * lumpy = Lumpy() lumpy.make_reference() deck = Deck() hand = PokerHand() deck.move_cards(hand, 7) ``` -------------------------------- ### Simple For Loop Example Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates a basic for loop that prints 'Hello!' four times. This illustrates the syntax and execution of a simple loop. ```python for i in range(4): print 'Hello!' ``` -------------------------------- ### Use Markdown output format Source: https://github.com/datalab-to/marker/blob/master/_autodocs/output-formats.md Example of converting a document to Markdown and accessing its fields. ```python rendered = converter("document.pdf") # With output_format="markdown" print(rendered.markdown) print(f"Images: {len(rendered.images)}") print(f"Table of Contents: {rendered.metadata['table_of_contents']}") ``` -------------------------------- ### Execute Batch Conversions via CLI Source: https://github.com/datalab-to/marker/blob/master/_autodocs/converters.md Command line interface examples for processing folders using default or specific converter classes. ```bash # Convert entire folder with appropriate converter marker /input/folder --output_dir /output/folder # Using TableConverter on folder marker /input/folder \ --converter_cls marker.converters.table.TableConverter \ --output_format json ``` -------------------------------- ### Create Canvas and Column Frame Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Initializes a white canvas widget and starts a new column frame for subsequent widgets. This is part of the GUI setup process. ```Python self.canvas = self.ca(width=400, height=400, bg='white') self.col() ``` -------------------------------- ### Shutdown Models Usage Example Source: https://github.com/datalab-to/marker/blob/master/_autodocs/models-and-inference.md Demonstrates the recommended pattern for initializing and safely shutting down model resources using a try-finally block. ```python from marker.models import create_model_dict, shutdown_models try: models = create_model_dict() # Use models... finally: shutdown_models(models) ``` -------------------------------- ### Time Object Instantiation Examples Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates creating Time objects with default values, one argument, two arguments, and three arguments, showing how parameters override defaults. ```Python >>> time = Time() >>> time.print_time() 00:00:00 ``` ```Python >>> time = Time (9) >>> time.print_time() 09:00:00 ``` ```Python >>> time = Time(9, 45) >>> time.print_time() 09:45:00 ``` -------------------------------- ### JSON Output Structure Example Source: https://github.com/datalab-to/marker/blob/master/README.md Example of a page block structure in the JSON output format, demonstrating the tree-like hierarchy of blocks. ```json { "id": "/page/10/Page/366", "block_type": "Page", "html": "", "polygon": [[0.0, 0.0], [612.0, 0.0], [612.0, 792.0], [0.0, 792.0]], "children": [ { "id": "/page/10/SectionHeader/0", "block_type": "SectionHeader", "html": "

Supplementary Material for Subspace Adversarial Training

", "polygon": [ [217.845703125, 80.630859375], [374.73046875, 80.630859375], [374.73046875, 107.0], [217.845703125, 107.0] ], "children": null, "section_hierarchy": { "1": "/page/10/SectionHeader/1" }, "images": {} }, ... ] } ``` -------------------------------- ### Order of Operations Examples Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates the use of parentheses and the order of operations (PEMDAS) in Python expressions. ```python 2 * (3-1) ``` ```python (1+1)**(5-2) ``` ```python (minute * 100) / 60 ``` ```python 2**1+1 ``` ```python 3*1**3 ``` ```python 2*3-1 ``` ```python 6+4/2 ``` -------------------------------- ### Use Chunk output format Source: https://github.com/datalab-to/marker/blob/master/_autodocs/output-formats.md Example of iterating through flattened document chunks. ```python rendered = converter("document.pdf") # With output_format="chunks" for chunk in rendered.chunks: print(f"Page {chunk.page_id}: {chunk.block_type}") print(f"Text preview: {chunk.html[:100]}...") ``` -------------------------------- ### Use JSON output format Source: https://github.com/datalab-to/marker/blob/master/_autodocs/output-formats.md Example of converting a document to JSON and iterating through the block tree. ```python import json rendered = converter("document.pdf") # With output_format="json" json_text = json.dumps(rendered.model_dump(), indent=2) # Access specific pages for page_output in rendered.children: print(f"Page: {page_output.id}") for child in page_output.children or []: print(f" Block: {child.block_type}") ``` -------------------------------- ### Initial Debugging Example Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md This function contains errors and is used to demonstrate debugging. It aims to compare two words to see if one is the reverse of the other. ```python def is_reverse(word1, word2): if len(word1) != len(word2): return False i = 0 j = len(word2) while j > 0: if word1[i] != word2[j]: return False i = i+1 j = j-1 return True ``` -------------------------------- ### Configure Processing Performance Source: https://github.com/datalab-to/marker/blob/master/_autodocs/quick-start.md Examples for setting processing modes ranging from fast CPU-only execution to high-quality GPU-accelerated refinement. ```python # CPU-only, no OCR config = { "mode": "fast", "disable_ocr": True, "extract_images": False, } converter = PdfConverter(artifact_dict=artifact_dict, config=config) ``` ```python # GPU, full VLM, LLM refinement config = { "mode": "balanced", "use_llm": True, "gemini_api_key": "YOUR_KEY", } converter = PdfConverter(artifact_dict=artifact_dict, config=config) ``` ```python # Medium speed, good quality config = { "mode": "fast", # Lightweight layout "disable_ocr": False, # OCR where needed } converter = PdfConverter(artifact_dict=artifact_dict, config=config) ``` -------------------------------- ### Configure OCRConverter with Character Boxes Source: https://github.com/datalab-to/marker/blob/master/_autodocs/converters.md Example of configuring the converter to output character-level bounding boxes and specific formats. ```python from marker.converters.ocr import OCRConverter from marker.models import create_model_dict config = { "output_format": "ocr_json", "keep_chars": True, # Include character-level boxes } converter = OCRConverter( artifact_dict=create_model_dict(), config=config ) rendered = converter("scanned_page.png") # Access OCR output print(rendered.text) # Raw OCR text print(rendered.blocks) # Character blocks with coordinates ``` -------------------------------- ### Importing a Module in Python Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates the standard way to import a module and access its contents using dot notation. Requires the module to be installed. ```python >>> import math >>> print math >>> print math.pi 3.14159265359 ``` -------------------------------- ### Batch processing configurations Source: https://github.com/datalab-to/marker/blob/master/README.md Various command-line configurations for different hardware setups and processing requirements. ```shell marker /folder --output_dir out ``` ```shell VLLM_GPUS=0,1,2,3 marker /folder ... ``` ```shell marker /folder --num_chunks --chunk_idx ``` ```shell marker /folder --disable_ocr ``` -------------------------------- ### Create and Display a Basic GUI Window Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md This snippet demonstrates how to create a basic GUI window using the Gui module. It initializes a Gui object, sets the window title, and starts the event loop. ```Python from swampy.Gui import * g = Gui() g.title('Gui') g.mainloop() ``` -------------------------------- ### Multi-Worker Inference Setup Source: https://github.com/datalab-to/marker/blob/master/_autodocs/models-and-inference.md Utilize multiprocessing to process files in parallel by sharing the inference manager through process inheritance. ```python from marker.models import create_model_dict from multiprocessing import Pool def convert_file(filepath): from marker.converters.pdf import PdfConverter # Workers inherit the artifact_dict with shared inference_manager artifact_dict = create_model_dict() converter = PdfConverter(artifact_dict=artifact_dict) return converter(filepath) artifacts = create_model_dict() # Parent process holds the inference_manager # Workers share it via process inheritance filepaths = ["doc1.pdf", "doc2.pdf", "doc3.pdf"] # The marker CLI automatically handles this: # marker /folder --workers 8 ``` -------------------------------- ### Modulus Operator Example Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates the use of the modulus operator (%) to find the remainder of a division and its application in checking divisibility. ```Python >>> quotient = 7 / 3 >>> print quotient 2 >>> remainder = 7 % 3 >>> print remainder 1 So 7 divided by 3 is 2 with 1 left over. ``` -------------------------------- ### CLI Usage for OCRConverter Source: https://github.com/datalab-to/marker/blob/master/_autodocs/converters.md Command line interface examples for running OCR-only mode with or without character box extraction. ```bash # OCR only (no layout detection) marker_single document.pdf \ --converter_cls marker.converters.ocr.OCRConverter # OCR with character boxes marker_single document.pdf \ --converter_cls marker.converters.ocr.OCRConverter \ --keep_chars true ``` -------------------------------- ### Get All Text from Text Widget Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Retrieves all text content from a Text widget, from a starting index (0.0) to the end (END). Includes newline characters. ```Python text.get(0.0, END) ``` -------------------------------- ### Delete Text from Text Widget Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Removes a portion of text from a Text widget, specified by start and end indices. This example deletes all text after the first two characters. ```Python text.delete(1.2, END) ``` -------------------------------- ### Example Usage of find_center Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Shows how to use the find_center function with a Rectangle object and print the resulting center Point. Requires a print_point function to be defined. ```python >>> center = find_center(box) >>> print_point(center) (50.0, 100.0) ``` -------------------------------- ### Initialize document providers Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Standard initialization pattern for various file format providers using a configuration object. ```python from marker.providers.pdf import PdfProvider provider = PdfProvider("document.pdf", config) ``` ```python from marker.providers.image import ImageProvider provider = ImageProvider("image.png", config) ``` ```python from marker.providers.epub import EPubProvider provider = EPubProvider("document.epub", config) ``` ```python from marker.providers.html import HtmlProvider provider = HtmlProvider("document.html", config) ``` ```python from marker.providers.powerpoint import PowerPointProvider provider = PowerPointProvider("presentation.pptx", config) ``` ```python from marker.providers.spreadsheet import SpreadsheetProvider provider = SpreadsheetProvider("data.xlsx", config) ``` -------------------------------- ### Function Call Example Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md A basic example of a function call in Python, showing the 'type' function with an integer argument and its return value. ```python >>> type(32) ``` -------------------------------- ### Complete Pipeline Execution Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Demonstrates the manual initialization of builders and providers to process a PDF document into a structured format. ```python from marker.converters.pdf import PdfConverter from marker.models import create_model_dict from marker.providers.registry import provider_from_filepath from marker.builders.document import DocumentBuilder from marker.builders.layout import LayoutBuilder from marker.builders.line import LineBuilder from marker.builders.ocr import OcrBuilder from marker.builders.structure import StructureBuilder from marker.schema.document import Document # Setup config = { "mode": "balanced", "disable_ocr": False, } models = create_model_dict() # Get provider ProviderClass = provider_from_filepath("document.pdf") provider = ProviderClass("document.pdf", config) # Build document (normally done by converter) layout_builder = LayoutBuilder(config, inference_manager=models['inference_manager']) line_builder = LineBuilder(config) ocr_builder = OcrBuilder(config, inference_manager=models['inference_manager']) doc_builder = DocumentBuilder(config) document = doc_builder(provider, layout_builder, line_builder, ocr_builder) # Structure structure_builder = StructureBuilder(config) structure_builder(document) # Apply processors # ... processors run here ... # Render from marker.renderers.markdown import MarkdownRenderer renderer = MarkdownRenderer(config) rendered = renderer(document) ``` -------------------------------- ### Get User Input with raw_input Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Use `raw_input()` to get string input from the user. The program pauses until the user presses Enter. ```python >>> text = raw_input() What are you waiting for? >>> print text What are you waiting for? ``` -------------------------------- ### Initialize Provider via Registry Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Automatically select and instantiate a provider class based on the file extension of the input document. ```python from marker.providers.registry import provider_from_filepath provider_class = provider_from_filepath("document.pdf") provider = provider_class("document.pdf", config) ``` ```python from marker.providers.registry import provider_from_filepath ProviderClass = provider_from_filepath("report.pdf") provider = ProviderClass("report.pdf", config) ``` -------------------------------- ### Define __init__ method for Time class Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Shows the structure of the __init__ method, which is a special method invoked during object instantiation. This example is a placeholder for a complete implementation. ```python # inside class Time: ``` ```python def __init__(self): pass ``` -------------------------------- ### Open and Create a Database Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Opens a database file, creating it if it doesn't exist. The 'c' mode ensures creation. ```python import anydbm db = anydbm.open('captions.db', 'c') ``` -------------------------------- ### Import TurtleWorld Without Package Installation Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Import all functions from the TurtleWorld module if Swampy is downloaded but not installed as a package. Ensure the directory is in Python's search path. ```python from TurtleWorld import * ``` -------------------------------- ### Python Infinite Recursion Example Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md A minimal example demonstrating infinite recursion, where a function calls itself without a base case to stop. This will lead to a RuntimeError when Python's maximum recursion depth is exceeded. ```python def recurse(): recurse() ``` -------------------------------- ### String Slices from Beginning or to End Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Shows how to create slices that start from the beginning of the string (by omitting the first index) or go to the end of the string (by omitting the second index). ```Python >>> fruit = 'banana' >>> fruit[:3] 'ban' >>> fruit[3:] 'ana' ``` -------------------------------- ### Find Substring Index Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Use the .find() method to locate the first occurrence of a substring within a string. It returns the starting index or -1 if not found. Optional start and end arguments can specify the search range. ```python >>> word = 'banana' >>> index = word.find('a') >>> print index 1 ``` ```python >>> word.find('na') 2 ``` ```python >>> word.find('na', 3) 4 ``` ```python >>> name = 'bob' >>> name.find('b', 1, 2) -1 ``` -------------------------------- ### Traceback Example: Math Domain Error Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md A `ValueError: math domain error` can occur if an operation receives an invalid input, like taking the logarithm of zero or a negative number. This example highlights integer division issues in Python 2. ```python Traceback (most recent call last): File "snr.py", line 5, in ? decibels = 10 * math.log10(ratio) ValueError: math domain error ``` -------------------------------- ### Loading Configuration in Python Source: https://github.com/datalab-to/marker/blob/master/_autodocs/configuration.md Demonstrates how to load a JSON configuration file and pass it to the PdfConverter. ```python import json from marker.converters.pdf import PdfConverter with open("config.json") as f: config = json.load(f) converter = PdfConverter( artifact_dict=create_model_dict(), config=config ) ``` -------------------------------- ### Using dictionary.get() for Concise Histogram Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md This exercise demonstrates a more concise way to write the histogram function using the dictionary's get method, eliminating the need for an if statement. The get method retrieves a value for a key, or a default value if the key is not found. ```Python >>> h = histogram('a') >>> print h {'a': 1} >>> h.get('a', 0) 1 >>> h.get('b', 0) 0 ``` -------------------------------- ### Initialize PdfConverter from Configuration Source: https://github.com/datalab-to/marker/blob/master/_autodocs/quick-start.md Methods to configure the converter using a dictionary, a JSON file, or the library's ConfigParser. ```python config = { "output_format": "json", "mode": "balanced", "use_llm": True, } converter = PdfConverter(artifact_dict=artifact_dict, config=config) ``` ```python import json with open("config.json") as f: config = json.load(f) converter = PdfConverter(artifact_dict=artifact_dict, config=config) ``` ```python from marker.config.parser import ConfigParser parser = ConfigParser({ "output_format": "json", "use_llm": True, "gemini_api_key": "YOUR_KEY", }) config = parser.generate_config_dict() renderer = parser.get_renderer() processor_list = parser.get_processors() converter = PdfConverter( artifact_dict=artifact_dict, config=config, renderer=renderer, processor_list=processor_list, ) ``` -------------------------------- ### OpenAIService Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Initializes the OpenAI service. ```APIDOC ## OpenAIService ### Description Initializes the OpenAI service for GPT model access. ### Constructor `OpenAIService(config)` ### Configuration Parameters - **openai_api_key** (str) - Required - OpenAI API key - **openai_model** (str) - Optional - Model name (Default: gpt-4-turbo) - **openai_base_url** (str) - Optional - API base URL (Default: https://api.openai.com/v1) ``` -------------------------------- ### Example Usage of is_divisible Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates the output of the is_divisible function with different inputs. ```python >>> is_divisible(6, 4) False >>> is_divisible(6, 3) True ``` -------------------------------- ### Get previous page Source: https://github.com/datalab-to/marker/blob/master/_autodocs/schema-and-blocks.md Retrieves the page immediately preceding the current page. ```python prev_page = document.get_prev_page(current_page) ``` -------------------------------- ### Handle Converter Exceptions Source: https://github.com/datalab-to/marker/blob/master/_autodocs/converters.md Example of catching common errors during the document conversion process. ```python try: rendered = converter("document.pdf") except FileNotFoundError: print("File not found") except RuntimeError as e: print(f"Processing failed: {e}") ``` -------------------------------- ### Initialize and Run OCRConverter Source: https://github.com/datalab-to/marker/blob/master/_autodocs/converters.md Basic instantiation and execution of the OCRConverter using default settings. ```python from marker.converters.ocr import OCRConverter from marker.models import create_model_dict converter = OCRConverter(artifact_dict=create_model_dict()) rendered = converter("scanned_document.pdf") ``` -------------------------------- ### Initialize OpenAI Service Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Requires an OpenAI API key and model specification. ```python from marker.services.openai import OpenAIService config = { "use_llm": True, "openai_api_key": "sk-...", "openai_model": "gpt-4-turbo", } service = OpenAIService(config) ``` -------------------------------- ### Reference global settings Source: https://github.com/datalab-to/marker/blob/master/_autodocs/configuration.md Default global settings for output directories, encoding, and hardware device selection. ```python from marker.settings import settings output_dir = "{marker_module_dir}/conversion_results" output_encoding = "utf-8" output_image_format = "JPEG" loglevel = "INFO" torch_device = "cuda" if GPU else ("mps" if MPS else "cpu") ``` -------------------------------- ### Dictionary Inversion Example Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates the usage of the invert_dict function with a sample histogram dictionary. ```python >>> hist = histogram('parrot') >>> print hist {'a': 1, 'p': 1, 'r': 2, 't': 1, 'o': 1} >>> inverse = invert_dict(hist) >>> print inverse {1: ['a', 'p', 't', 'o'], 2: ['r']} ``` -------------------------------- ### List Initialization and Printing Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates the creation of lists with different data types (integers, strings, empty list) and how to print them in Python. ```Python cheeses = ['Cheddar', 'Edam', 'Gouda'] numbers = [17, 123] empty = [] print cheeses, numbers, empty ``` -------------------------------- ### Get Dictionary Size Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md The len() function returns the number of key-value pairs in a dictionary. ```python >>> len(eng2sp) 3 ``` -------------------------------- ### Troubleshooting Backend Availability Source: https://github.com/datalab-to/marker/blob/master/_autodocs/models-and-inference.md Switch to a CPU-based backend if the default inference server fails to initialize. ```python # Check backend availability import os os.environ['SURYA_INFERENCE_BACKEND'] = 'llamacpp' # Try CPU backend models = create_model_dict() ``` -------------------------------- ### Initialize TurtleWorld and Create a Turtle Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Set up the TurtleWorld environment and create a new turtle instance. The 'wait_for_user()' function keeps the window open until the user interacts with it. ```python from swampy.TurtleWorld import * world = TurtleWorld() bob = Turtle() print bob wait_for_user() ``` -------------------------------- ### Python Example of Raising a ValueError with a Message Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Shows how to raise a ValueError with a custom error message. ```python >>> raise ValueError('value does not appear in the dictionary') Traceback (most recent call last): File "", line 1, in ? ValueError: value does not appear in the dictionary ``` -------------------------------- ### Create a Color Selection Menubutton Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Initializes a Gui and creates a Menubutton with a default color. Requires the Gui class to be defined. ```python g = Gui() g.la('Select a color:') colors = ['red', 'green', 'blue'] mb = g.mb(text=colors[0]) ``` -------------------------------- ### Define print_spam function Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md A simple function that prints the string 'spam'. Used as an example for do_twice. ```python def print_spam(): print 'spam' ``` -------------------------------- ### Initialize Google Gemini Service Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Requires a valid Google Gemini API key. ```python from marker.services.gemini import GoogleGeminiService config = { "use_llm": True, "gemini_api_key": "YOUR_API_KEY", } service = GoogleGeminiService(config) ``` -------------------------------- ### Initialize Local Ollama Service Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Connects to a local Ollama instance; no API key is required. ```python from marker.services.ollama import OllamaService config = { "use_llm": True, "ollama_base_url": "http://localhost:11434", "ollama_model": "llama2", } service = OllamaService(config) ``` -------------------------------- ### Getting Object Type Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Shows how to use the built-in type() function to determine the class of an object. ```Python >>> type(p) ``` -------------------------------- ### 3D Graphics Initialization with Visual Module Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Initializes a 3D space using the 'visual' module, setting the scene range and center. It then draws a single blue sphere at the center of the scene. Requires the 'visual' module to be installed. ```python from visual import * scene.range = (256, 256, 256) scene.center = (128, 128, 128) color = (0.1, 0.1, 0.9) # mostly blue sphere(pos=scene.center, radius=128, color=color) ``` -------------------------------- ### Basic PDF Conversion Configuration Source: https://github.com/datalab-to/marker/blob/master/_autodocs/configuration.md Initializes the converter with standard markdown output and image extraction enabled. ```python from marker.converters.pdf import PdfConverter from marker.models import create_model_dict config = { "output_format": "markdown", "extract_images": True, } converter = PdfConverter( artifact_dict=create_model_dict(), config=config ) rendered = converter("document.pdf") ``` -------------------------------- ### Interactive Mode Calculation Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Example of using Python interactively for calculations, showing immediate output of expressions. ```python >>> miles = 26.2 >>> miles * 1.61 42.182 ``` -------------------------------- ### Import Lumpy Library Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Imports the Lumpy class from the swampy.Lumpy library. This is the initial setup for generating diagrams. ```python from swampy.Lumpy import Lumpy ``` -------------------------------- ### Run Marker Benchmarks Source: https://github.com/datalab-to/marker/blob/master/benchmarks/README.md Commands to execute different benchmark modes including balanced, fast, and accurate configurations. ```bash BENCH=/path/to/olmocr/olmocr/bench/bench_data SURYA=http://localhost:8000/v1 # balanced (VLM layout + full-page OCR) python benchmarks/inference.py --bench-dir $BENCH --out /tmp/marker_balanced \ --mode balanced --workers 48 --surya-url $SURYA # fast (rf-detr layout + pdftext, VLM for equations only) python benchmarks/inference.py --bench-dir $BENCH --out /tmp/marker_fast \ --mode fast --workers 32 --surya-url $SURYA # fast, no OCR (pure CPU text layer, no VLM) python benchmarks/inference.py --bench-dir $BENCH --out /tmp/marker_fast_noocr \ --mode fast --disable-ocr --workers 32 # accurate (balanced + LLM refinement; needs an LLM service configured) python benchmarks/inference.py --bench-dir $BENCH --out /tmp/marker_accurate \ --mode balanced --use-llm --workers 32 --surya-url $SURYA ``` -------------------------------- ### Instantiate a Point Object Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates how to create an instance of the Point class and assign it to a variable. The output shows the representation of the new object. ```Python >>> blank = Point() >>> print blank <__main__.Point instance at 0xb7e9d3ac> ``` -------------------------------- ### Initialize StructureBuilder Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Instantiate the structure builder to organize blocks into a hierarchical document format. ```python from marker.builders.structure import StructureBuilder builder = StructureBuilder(config) builder(document) ``` -------------------------------- ### Nested Lists Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Shows an example of a list containing other lists as elements. Each nested list is treated as a single element. ```Python ['spam', 1, ['Brie', 'Roquefort', 'Pol le Veq'], [1, 2, 3]] ``` -------------------------------- ### Initialize Google Vertex AI Service Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Requires a GCP project ID and region. ```python from marker.services.vertex import GoogleVertexService config = { "use_llm": True, "vertex_project_id": "my-project", "vertex_region": "us-central1", } service = GoogleVertexService(config) ``` -------------------------------- ### Import and Use Python Module Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates how to import a custom Python module (`wc`) and access its functions, such as `linecount`, from the interpreter. ```python >>> import wc 7 >>> print wc >>> wc.linecount('wc.py') 7 ``` -------------------------------- ### Getting String Length with len() Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Shows how to use the built-in len() function to find the number of characters in a string. ```Python >>> fruit = 'banana' >>> len(fruit) 6 ``` -------------------------------- ### Initialize Azure OpenAI Service Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Requires Azure endpoint, API key, and deployment name. ```python from marker.services.azure_openai import AzureOpenAIService config = { "use_llm": True, "azure_endpoint": "https://myinstance.openai.azure.com/", "azure_api_key": "...", "deployment_name": "gpt-4", } service = AzureOpenAIService(config) ``` -------------------------------- ### Initialize LayoutBuilder Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Instantiate the layout builder to detect page regions and block structures. ```python from marker.builders.layout import LayoutBuilder builder = LayoutBuilder(config, inference_manager=...) builder(document, provider) ``` -------------------------------- ### GoogleGeminiService Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Initializes the Google Gemini service using an API key. ```APIDOC ## GoogleGeminiService ### Description Initializes the Google Gemini service for LLM operations. ### Constructor `GoogleGeminiService(config)` ### Configuration Parameters - **gemini_api_key** (str) - Required - Google Gemini API key - **gemini_model_name** (str) - Optional - Model to use (Default: gemini-3.5-flash) ``` -------------------------------- ### Python Example of Unsuccessful Reverse Lookup Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Illustrates an unsuccessful reverse lookup where the value is not found in the dictionary, resulting in a ValueError. ```python >>> k = reverse_lookup(h, 3) Traceback (most recent call last): File "", line 1, in ? File "", line 5, in reverse_lookup ValueError ``` -------------------------------- ### Initialize PdfConverter Source: https://github.com/datalab-to/marker/blob/master/_autodocs/converters.md Standard usage for converting PDF documents to markdown, JSON, or HTML. ```python from marker.converters.pdf import PdfConverter converter = PdfConverter(artifact_dict=create_model_dict()) rendered = converter("document.pdf") ``` -------------------------------- ### Configure PdfConverter with Gemini Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Initialize the PdfConverter using the Google Gemini service by providing the API key in the configuration dictionary. ```python from marker.converters.pdf import PdfConverter from marker.models import create_model_dict config = { "use_llm": True, "output_format": "json", "gemini_api_key": "YOUR_API_KEY", } converter = PdfConverter( artifact_dict=create_model_dict(), config=config, llm_service="marker.services.gemini.GoogleGeminiService" ) rendered = converter("document.pdf") ``` -------------------------------- ### Debugging NameError in Python Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Example of a 'NameError' that occurs when a variable is used before it has been assigned a value, often due to a spelling mistake. ```python principal = 327.68 interest = principle * rate ``` -------------------------------- ### View Project File Organization Source: https://github.com/datalab-to/marker/blob/master/_autodocs/README.md Displays the directory structure of the documentation. ```text output/ ├── README.md # This file ├── quick-start.md # Common usage patterns ├── pdf-converter.md # PdfConverter API ├── converters.md # Specialized converters ├── models-and-inference.md # Model management ├── schema-and-blocks.md # Document structure ├── providers-and-builders.md # Input pipeline ├── processors.md # Processing stages ├── output-formats.md # Output types ├── configuration.md # Configuration reference └── llm-services.md # LLM integration ``` -------------------------------- ### Metadata Dictionary Structure Source: https://github.com/datalab-to/marker/blob/master/README.md Example of the metadata dictionary returned with output formats, containing table of contents and page statistics. ```json { "table_of_contents": [ { "title": "Introduction", "heading_level": 1, "page_id": 0, "polygon": [...] } ], // computed PDF table of contents "page_stats": [ { "page_id": 0, "text_extraction_method": "pdftext", "block_counts": [("Span", 200), ...] }, ... ] } ``` -------------------------------- ### shutdown_models Source: https://github.com/datalab-to/marker/blob/master/_autodocs/models-and-inference.md Stops the shared inference server if the current process spawned it. This function is safe to call even if the servers were not started. ```APIDOC ## shutdown_models ### Description Stop the shared inference server if this process spawned it. Retrieves the inference_manager from model_dict and stops it if present, gracefully shutting down any running VLM or layout inference servers. ### Parameters - **model_dict** (dict) - Required - The dictionary returned by create_model_dict() ### Example ```python from marker.models import create_model_dict, shutdown_models try: models = create_model_dict() # Use models... finally: shutdown_models(models) ``` ``` -------------------------------- ### Initialize LineBuilder Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Instantiate the line builder to extract text and organize it into reading order. ```python from marker.builders.line import LineBuilder builder = LineBuilder(config) builder(document, provider) ``` -------------------------------- ### Empty String Slice Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Illustrates that a slice where the start index is greater than or equal to the end index results in an empty string. ```Python >>> fruit = 'banana' >>> fruit[3:3] '' ``` -------------------------------- ### Custom Provider Implementation Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Instructions for extending the system by creating and registering a custom provider for new file formats. ```APIDOC ## Custom Provider Implementation ### Description To support new file formats, inherit from `marker.providers.BaseProvider` and implement the required methods. Once defined, register the provider in the `PROVIDER_MAP` using the desired file extension. ### Required Methods - **get_page_count()** (int) - Returns the total number of pages in the document. - **get_images(page_ids, dpi)** (list) - Returns a list of PIL Image objects for the requested pages. ### Registration ```python from marker.providers.registry import PROVIDER_MAP PROVIDER_MAP[".extension"] = CustomProvider ``` ``` -------------------------------- ### Prompt User for Input with raw_input Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Provide a prompt string to `raw_input()` to guide the user on what to enter. Use '\n' for a newline. ```python >>> name = raw_input('What...is your name?\n') What...is your name? Arthur, King of the Britons! >>> print name Arthur, King of the Britons! ``` -------------------------------- ### Using Math Functions: Logarithm and Sine Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates using math.log10 for decibel calculations and math.sin for trigonometric operations with radians. Assumes signal_power and noise_power are defined. ```python ratio = signal_power / noise_power decibels = 10 * math.log10(ratio) radians = 0.7 height = math.sin(radians) ``` -------------------------------- ### Example Usage of find_defining_class Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates how to use the `find_defining_class` function to determine which class's `shuffle` method is invoked for a `Hand` object. ```Python >>> hand = Hand() >>> print find_defining_class(hand, 'shuffle') ``` -------------------------------- ### Configure PdfConverter with OpenAI Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Initialize the PdfConverter using the OpenAI service by specifying the API key and model in the configuration dictionary. ```python converter = PdfConverter( artifact_dict=create_model_dict(), config={ "use_llm": True, "openai_api_key": "sk-...", "openai_model": "gpt-4-turbo", }, llm_service="marker.services.openai.OpenAIService" ) ``` -------------------------------- ### Python Traceback Example Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Illustrates a typical Python traceback when a NameError occurs, showing the sequence of function calls leading to the error. ```python Traceback (innermost last): File "test.py", line 13, in __main__ cat_twice(line1, line2) File "test.py", line 5, in cat_twice print_twice(cat) File "test.py", line 9, in print_twice print cat NameError: name 'cat' is not defined ``` -------------------------------- ### Iterate with Index and Element using Enumerate Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Use the 'enumerate' built-in function to iterate over a sequence, getting both the index and the element in each iteration. ```python for index, element in enumerate('abc'): print index, element ``` -------------------------------- ### Opening a File for Writing Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Opens a file named 'output.txt' in write mode ('w'). If the file exists, its contents are cleared. If it does not exist, a new file is created. Be cautious when using 'w' mode as it overwrites existing data. ```python >>> fout = open('output.txt', 'w') >>> print fout ``` -------------------------------- ### Accessing List Elements Source: https://github.com/datalab-to/marker/blob/master/data/examples/markdown/thinkpython/thinkpython.md Demonstrates accessing elements of a list using bracket notation, similar to strings. Indices start at 0. ```Python >>> print cheeses[0] Cheddar ``` -------------------------------- ### OpenRouterService Source: https://github.com/datalab-to/marker/blob/master/_autodocs/llm-services.md Initializes the OpenRouter service. ```APIDOC ## OpenRouterService ### Description Initializes the OpenRouter service for OpenAI-compatible model access. ### Constructor `OpenRouterService(config)` ### Configuration Parameters - **openrouter_api_key** (str) - Required - OpenRouter API key - **openrouter_model** (str) - Optional - Model identifier (Default: google/gemini-3.5-flash) - **openrouter_base_url** (str) - Optional - API base URL (Default: https://openrouter.ai/api/v1) ``` -------------------------------- ### Processor Configuration Source: https://github.com/datalab-to/marker/blob/master/_autodocs/processors.md How to pass configuration dictionaries to processors to modify their behavior. ```APIDOC ## Processor Configuration ### Description Processors accept a `config` dictionary during initialization to control specific behaviors such as image extraction or debug output. ### Configuration Example ```python from marker.processors.table import TableProcessor config = { "extract_images": True, "debug_json": True, } processor = TableProcessor(config=config) processor(document) ``` ### Common Configuration Keys | Key | Type | Default | Description | |-----|------|---------|-------------| | `extract_images` | `bool` | `True` | Extract images from document | | `debug_pdf_images` | `bool` | `False` | Save layout debug visualizations | | `debug_json` | `bool` | `False` | Save JSON with bounding box info | | `debug_data_folder` | `str` | — | Directory to save debug data | | `keep_pageheader_in_output` | `bool` | `False` | Keep page headers in output | | `keep_pagefooter_in_output` | `bool` | `False` | Keep page footers in output | ``` -------------------------------- ### Provider Configuration Source: https://github.com/datalab-to/marker/blob/master/_autodocs/providers-and-builders.md Shows how to pass a configuration dictionary to a provider to control processing parameters like page ranges and OCR settings. ```python config = { "page_range": [0, 1, 2], # Only process these pages "extract_images": True, "disable_ocr": False, } provider = PdfProvider("document.pdf", config) ```