### DetectListLevels Input/Output Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/detection-transformations.md Shows an example of input blocks with varying indentation and their corresponding output with assigned list levels. ```text Input blocks (all type LIST): Block 1: x=50, text="First" Block 2: x=80, text="Sub-item" ← Indented Block 3: x=80, text="Another sub" Block 4: x=50, text="Second" ← Back to root level Output: Level 0: "First" Level 1: "Sub-item" Level 1: "Another sub" Level 0: "Second" ``` -------------------------------- ### Creating a Bold Word Instance Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Example of creating a Word instance with bold formatting. ```javascript new Word({ string: 'important', format: WordFormat.BOLD }) ``` -------------------------------- ### CLI Output Path Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Demonstrates how the CLI preserves the directory structure and replaces the file extension from .pdf to .md. ```text Input: /pdfs/company/contracts/document.pdf Output: /markdown/company/contracts/document.md ``` -------------------------------- ### Creating a Regular Word Instance Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Example of creating a basic Word instance with only the required string property. ```javascript new Word({ string: 'hello' }) ``` -------------------------------- ### Format Transition Example: Link Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md Demonstrates the rendering of NORMAL text followed by a LINK, showing the standard Markdown link syntax with text and URL. ```markdown Input: NORMAL("See") LINK("example.com") Output: See [example.com](example.com) ``` -------------------------------- ### Creating a Hyperlink Word Instance Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Example of creating a Word instance representing a hyperlink. ```javascript new Word({ string: 'example.com', type: WordType.LINK }) ``` -------------------------------- ### Creating a Footnote Link Word Instance Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Example of creating a Word instance designated as a footnote link. ```javascript new Word({ string: '1', type: WordType.FOOTNOTE_LINK }) ``` -------------------------------- ### TOC Detection Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/detection-transformations.md Demonstrates the input and output of the DetectTOC transformation, showing how a TOC page is identified and how link titles and page numbers are extracted. ```text Input TOC page: Introduction .......................... 5 Chapter 1 ............................ 10 Section 1.1 ...................... 12 Section 1.2 ...................... 15 Output: - Page marked as TOC (type: BlockType.TOC) - globals.tocPages = [0] (if first page) - Links extracted with page numbers - Indent levels detected (left margin determines level) - Later transformations use tocPages to confirm headline structure ``` -------------------------------- ### Format Transition Example: Bold and Normal Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md Demonstrates how consecutive BOLD and NORMAL formatted words are rendered, showing the application and closure of Markdown bold syntax. ```markdown Input: BOLD("Hello") BOLD("world") NORMAL("!") Output: **Hello world**! ``` -------------------------------- ### NPX Installation Usage Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md If installed via NPX, use the npx command to execute the pdf2md tool, specifying input and output folders, and enabling recursive conversion if needed. ```bash npx @opendocsg/pdf2md \ --inputFolderPath=/data/pdfs \ --outputFolderPath=/data/markdown \ --recursive ``` -------------------------------- ### List Item Detection Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/detection-transformations.md Illustrates the input and output of the DetectListItems transformation, showing how different bullet and numbered list formats are processed and annotated. ```text Input: • First item - Second item 1. Third item Output (3 items): • First item (REMOVED) - First item (ADDED) - Second item (DETECTED) 1. Third item (DETECTED) ``` -------------------------------- ### Format Transition Example: Footnote Link Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md Shows the rendering of a FOOTNOTE_LINK followed by NORMAL text, highlighting that no space is added between the footnote marker and subsequent text. ```markdown Input: FOOTNOTE_LINK("1") NORMAL("text") Output: ^1text (no space before footnote) ``` -------------------------------- ### Markdown List Indentation Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/detection-transformations.md Illustrates how different nesting levels are rendered in Markdown output with corresponding indentation. ```markdown - Level 0 item - Level 1 item - Level 2 item ``` -------------------------------- ### Format Transition Example: Mixed Formatting Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md Illustrates the rendering of text with mixed formatting (BOLD, NORMAL, OBLIQUE), showing how different Markdown syntaxes are applied sequentially. ```markdown Input: BOLD("Bold") NORMAL("normal") OBLIQUE("italic") Output: **Bold** normal _italic_ ``` -------------------------------- ### Single Folder Conversion Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Convert all PDF files located directly within the specified input folder to Markdown format in the output folder. ```bash pdf2md --inputFolderPath=/data/pdfs --outputFolderPath=/data/markdown ``` -------------------------------- ### Recursive Conversion Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Recursively convert all PDF files within the input folder and its subfolders to Markdown, maintaining the directory structure in the output folder. ```bash pdf2md --inputFolderPath=/data/pdfs --outputFolderPath=/data/markdown --recursive ``` -------------------------------- ### DetectHeaders Example Output Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/detection-transformations.md Illustrates the output of the DetectHeaders transformation on a sample input page, showing how text lines are classified into header types (H1, H2) or as paragraph text based on their height. ```text Input page with most-used height 12: Line 1: height 24 "Main Title" ← maxHeight Line 2: height 18 "Section One" ← > baseline Line 3: height 12 "Paragraph text" ← baseline Output: Line 1: type H1, annotation DETECTED Line 2: type H2, annotation DETECTED Line 3: type PARAGRAPH ``` -------------------------------- ### Convert PDF Text Items to Markdown Lines Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/line-conversion.md Demonstrates the setup and usage of LineConverter and TextItemLineGrouper to process text items. It shows how to group text items by their vertical position and then convert each group into a compact line representation, suitable for markdown conversion. ```javascript const LineConverter = require('./LineConverter'); const TextItemLineGrouper = require('./TextItemLineGrouper'); const TextItem = require('./TextItem'); const WordFormat = require('./markdown/WordFormat'); // Setup const fontToFormats = new Map([ ['g_d0_f1', WordFormat.BOLD.name], ['g_d0_f2', undefined] ]); const grouper = new TextItemLineGrouper({ mostUsedDistance: 14 }); const converter = new LineConverter(fontToFormats); // Sample text items (from PDF parser) const textItems = [ new TextItem({ x: 0, y: 100, text: 'Hello', font: 'g_d0_f1', width: 50, height: 12 }), new TextItem({ x: 60, y: 100, text: 'world', font: 'g_d0_f2', width: 40, height: 12 }), new TextItem({ x: 0, y: 120, text: 'Footnote', font: 'g_d0_f2', width: 60, height: 12 }), new TextItem({ x: 200, y: 100, text: '1', font: 'g_d0_f3', width: 8, height: 6 }) ]; // Group by Y coordinate const lines = grouper.group(textItems); // lines[0] = [item0, item1, item3] (y=100) // lines[1] = [item2] (y=120) // Convert each line lines.forEach(lineItems => { const lineItem = converter.compact(lineItems); console.log(lineItem.text()); // "Hello world", "Footnote" console.log(lineItem.words); // Word objects with format/type }); ``` -------------------------------- ### Instantiate and Use LineConverter Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/line-conversion.md Demonstrates how to create a LineConverter instance and use its compact method to convert TextItems into a LineItem. Shows how to access the resulting text and word count. ```javascript const converter = new LineConverter(fontToFormats); const lineItem = converter.compact([ new TextItem({ x: 0, y: 100, text: 'Hello', font: 'g_d0_f1' }), new TextItem({ x: 60, y: 100, text: 'World', font: 'g_d0_f2' }) ]); lineItem.text() // Returns: "Hello World" lineItem.words.length // 2 ``` -------------------------------- ### Get Headline BlockType by Level Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Returns the appropriate BlockType for a headline given its level (e.g., 1 for H1, 2 for H2). ```typescript headlineByLevel(level): BlockType ``` -------------------------------- ### Directory Creation Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/configuration.md Output directories are created automatically and recursively using fs.mkdirSync. Full path trees are created as needed to accommodate the output file path. ```javascript // If output path is /output/a/b/c/document.md // Creates: /output, /output/a, /output/a/b, /output/a/b/c ``` -------------------------------- ### Basic CLI Usage Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Use this command to convert PDFs in a specified folder to Markdown. The output folder will be created if it does not exist. ```bash pdf2md --inputFolderPath= --outputFolderPath= [--recursive] ``` -------------------------------- ### getAllFileAndFolderPaths Algorithm Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Illustrates the iterative algorithm used by getAllFileAndFolderPaths to discover all PDF files and subfolders recursively. ```text while folderPaths not empty: for each folder in folderPaths: list contents with getFileAndFolderPaths() add PDFs to filePaths add subfolders to next iteration's folderPaths move to next iteration ``` -------------------------------- ### Get Headline BlockType by Level Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md Retrieves the BlockType corresponding to a specific heading level (1-6). If the level is out of range, it logs a warning and defaults to H6. ```javascript const type = BlockType.headlineByLevel(3); // Returns BlockType.H3 ``` -------------------------------- ### Headline BlockType Implementation Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md The internal implementation for checking if a BlockType is a headline. It specifically looks for types with a name array starting with 'H' and having a length of 2. ```javascript function isHeadline(type) { return type && type.name.length === 2 && type.name[0] === 'H' } ``` -------------------------------- ### Getting Word Strings Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md The wordStrings() method returns an array containing the string values of each word within the LineItem. This is helpful when you need to process individual words. ```javascript lineItem.wordStrings() // Returns: ["Hello", "World"] ``` -------------------------------- ### Run Tests with npm Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/README.md Use these npm commands to execute tests. 'npm test' runs all tests, 'npm run test:watch' enables watch mode for continuous testing, and 'npm run test:coverage' generates a test coverage report. ```bash npm test # Run tests ``` ```bash npm run test:watch # Watch mode ``` ```bash npm run test:coverage # With coverage ``` -------------------------------- ### Getting Line Text Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Use the text() method to retrieve the joined text of all words in a LineItem, separated by spaces. This is useful for displaying the full line content. ```javascript const lineItem = new LineItem({ words: [ new Word({ string: 'Hello' }), new Word({ string: 'World' }) ] }); lineItem.text() // Returns: "Hello World" ``` -------------------------------- ### Detect Uppercase Character in Middle of Word Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/utilities.md Use to identify if a string contains uppercase letters within words, excluding those at the start of a word. Ignores numbers. ```javascript const { hasUpperCaseCharacterInMiddleOfWord } = require('./string-functions'); hasUpperCaseCharacterInMiddleOfWord('CamelCase') // true hasUpperCaseCharacterInMiddleOfWord('hello world') // false hasUpperCaseCharacterInMiddleOfWord('Hello World') // false (caps only at start) hasUpperCaseCharacterInMiddleOfWord('XMLParser') // true ``` -------------------------------- ### Run ESLint Linter Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/configuration.md Execute the ESLint linter to check code for style and potential errors. Configuration is handled by a default ESLint configuration file. ```bash npm run lint # Run linting ``` -------------------------------- ### WordFormat Enum Definition Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Defines text formatting options like bold, italic, and bold-italic for markdown. These enums specify the start and end symbols used for each format. ```typescript enum WordFormat { BOLD // **bold text** (startSymbol: **, endSymbol: **) OBLIQUE // _italic text_ (startSymbol: _, endSymbol: _) BOLD_OBLIQUE // **_bold italic_** (startSymbol: **_, endSymbol: _**) } ``` -------------------------------- ### Run Tests with Mocha Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/configuration.md Execute tests using the Mocha framework. Options include running tests once, in watch mode, or with coverage reporting. ```bash npm test # Run tests once npm run test:watch # Run tests in watch mode npm run test:coverage # Run with coverage reporting ``` -------------------------------- ### CLI Usage Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Command-line interface for pdf2md. Specify input and output folder paths. Use --recursive for subdirectories. ```bash // CLI pdf2md --inputFolderPath= --outputFolderPath= [--recursive] ``` -------------------------------- ### ParsedElements Add Method Example Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Merges another ParsedElements object into the current one, combining footnote links and formatted word counts. This is useful for aggregating parsed data from multiple blocks. ```javascript const block1 = new ParsedElements({ footnoteLinks: [1, 2], formattedWords: 3 }); const block2 = new ParsedElements({ footnoteLinks: [3], formattedWords: 2 }); block1.add(block2); // block1.footnoteLinks = [1, 2, 3] // block1.formattedWords = 5 ``` -------------------------------- ### Convert PDF Buffer to Markdown using JavaScript Library Source: https://github.com/opengovsg/pdf2md/blob/master/README.md Use this snippet to convert a PDF file buffer to Markdown text programmatically. Ensure you have the 'path', 'fs', and '@opendocsg/pdf2md' modules installed. ```javascript const path = require('path'); const fs = require('fs') const pdf2md = require('@opendocsg/pdf2md') const pdfBuffer = fs.readFileSync(filePath) pdf2md(pdfBuffer, callbacks) .then(text => { let outputFile = allOutputPaths[i] + '.md' console.log(`Writing to ${outputFile}...`) fs.writeFileSync(path.resolve(outputFile), text) console.log('Done.') }) .catch(err => { console.error(err) }) ``` -------------------------------- ### CLI Usage Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Command-line interface for converting PDF files or folders to Markdown. ```APIDOC ## pdf2md CLI ### Description Command-line tool for converting PDF files or folders to Markdown. ### Usage `pdf2md --inputFolderPath= --outputFolderPath= [--recursive]` ### Options - **--inputFolderPath** (string) - Required - Path to the input PDF file or folder. - **--outputFolderPath** (string) - Required - Path to the output directory for Markdown files. - **--recursive** (boolean) - Optional - If true, processes subdirectories recursively. ``` -------------------------------- ### CLI Interface Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md The command-line interface for batch converting PDF files within directories to Markdown. It supports recursive conversion and automatic output directory creation. ```APIDOC ## CLI Interface ### Description Provides a command-line tool for batch conversion of PDF files to Markdown. It can process single directories or recursively process subdirectories. ### Method Command-line execution ### Endpoint `pdf2md` ### Parameters #### Path Parameters None #### Query Parameters - **--inputFolderPath** (string) - Required - The path to the input folder containing PDF files. - **--outputFolderPath** (string) - Required - The path to the output folder where Markdown files will be saved. - **--recursive** (boolean) - Optional - If present, the tool will process subdirectories recursively. ### Request Example ```bash # Convert PDFs in a single directory pdf2md --inputFolderPath=./pdfs --outputFolderPath=./md_output # Convert PDFs recursively including subdirectories pdf2md --inputFolderPath=./pdfs --outputFolderPath=./md_output --recursive ``` ### Response #### Success Response Markdown files are generated in the specified output directory. #### Response Example Output directory `./md_output` will contain markdown files corresponding to the input PDFs. ``` -------------------------------- ### Lint Code with npm Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/README.md Execute the linting command using npm to check code style and identify potential issues. ```bash npm run lint # Check code style ``` -------------------------------- ### Apply PDF to Markdown Transformations Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/transformation-system.md Demonstrates the process of parsing a PDF, creating a transformation pipeline using extracted font data, applying these transformations to the parsed pages, and finally extracting the Markdown content, including page breaks. ```javascript const { makeTransformations, transform } = require('./util/transformations'); const { parse } = require('./util/pdf'); // After parsing PDF const parseResult = await parse(pdfBuffer); const { fonts, pages } = parseResult; // Create transformation pipeline const transformations = makeTransformations(fonts.map); // Apply transformations const finalResult = transform(pages, transformations); // Extract Markdown const markdown = finalResult.pages .map(page => page.items.join('\n') + '\n') .join('\n'); ``` -------------------------------- ### PDF to Markdown Data Flow Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Illustrates the step-by-step transformation process from raw PDF input to the final Markdown output, detailing the intermediate data structures and transformations. ```text PDF Input ↓ Parser creates: Page { items: [TextItem, TextItem, TextItem, ...] - Each TextItem has x, y, font, height, text - Multiple TextItems on same Y ↓ CompactLines transformation: Page { items: [LineItem, LineItem, ...] - TextItems on same Y grouped into one LineItem - LineItem contains Word[] with format/type - Words inherit font format (bold, italic) ↓ DetectHeaders, DetectListItems, etc.: Page { items: [LineItem, LineItem, ...] - LineItems gain type (H1, LIST, PARAGRAPH, etc.) ↓ GatherBlocks transformation: Page { items: [LineItemBlock, LineItemBlock, ...] - LineItems grouped by spacing - LineItemBlocks have type and items[] ↓ ToMarkdown transformation: Page { items: ["# Heading\n\n", "Para text\n\n"] - Each block rendered to Markdown string - Block type determines rendering (# for H1, etc.) ↓ Final Output: "# Heading\n\nPara text\n\n ... ``` -------------------------------- ### Convert PDFs using CLI Tool Source: https://github.com/opengovsg/pdf2md/blob/master/README.md Execute the CLI tool to convert PDFs from an input folder to an output folder. The --recursive flag enables conversion of PDFs in subdirectories. ```bash $ cd [project_folder] $ npx @opendocsg/pdf2md --inputFolderPath=[your input folder path] --outputFolderPath=[your output folder path] --recursive ``` -------------------------------- ### pdf2md Usage with Progress Callbacks Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/pdf2md.md Shows how to use pdf2md with optional callbacks to track the conversion progress, such as metadata parsing, page processing, and font detection. The final Markdown is written to a file. ```javascript const pdf2md = require('@opendocsg/pdf2md'); const fs = require('fs'); const pdfBuffer = fs.readFileSync('./document.pdf'); const callbacks = { metadataParsed: (metadata) => { console.log('Metadata loaded'); }, pageParsed: (pages) => { console.log(`Processing page ${pages.length}...`); }, fontParsed: (font) => { console.log(`Font detected: ${font.ids}`); }, documentParsed: (doc, pages) => { console.log(`Total pages: ${pages.length}`); } }; pdf2md(pdfBuffer, callbacks) .then(markdown => { fs.writeFileSync('./output.md', markdown); }); ``` -------------------------------- ### Monitor PDF Conversion with Callbacks (JavaScript) Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/README.md Use callbacks to monitor the progress of PDF conversion. This is useful for providing user feedback or performing actions at different stages of the parsing process. ```javascript const callbacks = { metadataParsed: (metadata) => console.log('Metadata:', metadata.info), pageParsed: (pages) => console.log(`Pages: ${pages.length}`), fontParsed: (fonts) => console.log(`Fonts: ${fonts.ids.size}`), documentParsed: (doc, pages) => console.log('Done parsing') }; pdf2md(pdfBuffer, callbacks).then(markdown => { // Use markdown }); ``` -------------------------------- ### Main API Function Signature Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Use this function to convert a PDF buffer to Markdown. It returns a Promise that resolves to a string. ```javascript // Main API pdf2md(pdfBuffer, callbacks?): Promise ``` -------------------------------- ### LineItem Constructor Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Initializes a new LineItem instance. It can be created with an array of Word objects or a single text string. ```APIDOC ## Class: LineItem Represents a single line of text with word-level granularity and semantic type. **Module**: `lib/models/LineItem.js` **Extends**: `PageItem` ```javascript class LineItem extends PageItem { constructor(options: { x: number, y: number, width: number, height: number, words?: Word[], text?: string, // Alternative to words, auto-split type?: BlockType, annotation?: Annotation, parsedElements?: ParsedElements }) text(): string wordStrings(): string[] } ``` ### Properties | Property | Type | Description | |----------|------|-------------| | x | number | X position of first word | | y | number | Y coordinate (all words on same Y) | | width | number | Total width of all words | | height | number | Maximum height of words on line | | words | Word[] | Array of Word objects with formatting | | type | BlockType | Semantic type: PARAGRAPH, LIST, H1-H6, FOOTNOTES, CODE, TOC | | annotation | Annotation | Change tracking: ADDED, REMOVED, DETECTED, etc. | | parsedElements | ParsedElements | Detected links, footnotes, and formatting counts | ### Creation Patterns **From Word Array**: ```javascript const lineItem = new LineItem({ x: 50, y: 100, width: 200, height: 12, words: [ new Word({ string: 'Bold', format: WordFormat.BOLD }), new Word({ string: 'text' }) ] }); ``` **From Text String** (auto-splits on spaces): ```javascript const lineItem = new LineItem({ x: 50, y: 100, width: 200, height: 12, text: 'Hello World' // Auto-converted to words }); ``` ``` -------------------------------- ### Transformation step details Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/transformation-system.md Details the lifecycle of a single transformation within the pipeline: receiving a ParseResult, modifying pages and globals, and returning a new ParseResult for the next step. ```plaintext Each transformation: 1. Receives a `ParseResult` object 2. Analyzes and modifies `parseResult.pages` items 3. Optionally updates `parseResult.globals` with computed state 4. Returns a new `ParseResult` with changes 5. The next transformation receives the output of the previous ``` -------------------------------- ### Run CLI Tool with Increased Memory Allocation Source: https://github.com/opengovsg/pdf2md/blob/master/README.md If you encounter 'JavaScript heap out of memory' errors when converting many files recursively, use this command to increase the Node.js maximum old space size. ```bash $ node lib/pdf2md-cli.js --max-old-space-size=4096 --inputFolderPath=[your input folder path] --outputFolderPath=[your output folder path] --recursive ``` -------------------------------- ### Find First Page with Page Number Pattern Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/utilities.md Identifies the first page in a sequence that exhibits a consistent page number pattern. This is useful for skipping introductory pages without numbers and aligning with the document's main content. ```javascript const { findFirstPage } = require('./page-number-functions'); const firstPage = findFirstPage(pageIndexNumMap); // Returns: { pageIndex: 5, pageNum: 1, pattern: 'page N' } ``` -------------------------------- ### Convert PDF Folder to Markdown (CLI) Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/README.md Use the CLI to convert all PDFs in a specified folder to Markdown. The --recursive flag enables conversion of subfolders. ```bash # Single folder conversion pdf2md --inputFolderPath=/pdfs --outputFolderPath=/markdown # Recursive conversion pdf2md --inputFolderPath=/pdfs --outputFolderPath=/markdown --recursive ``` -------------------------------- ### Batch Process PDFs via CLI (Bash) Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/README.md Use the CLI for batch processing large PDF directories. Set the Node.js heap size before running the CLI command for memory-intensive operations. ```bash # Set heap size before CLI node --max-old-space-size=4096 lib/pdf2md-cli.js \ --inputFolderPath=/data/pdfs \ --outputFolderPath=/data/markdown \ --recursive ``` -------------------------------- ### Basic pdf2md Usage Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/pdf2md.md Demonstrates the basic usage of the pdf2md function by reading a PDF file and logging the resulting Markdown to the console. Requires the '@opendocsg/pdf2md' package and 'fs' module. ```javascript const pdf2md = require('@opendocsg/pdf2md'); const fs = require('fs'); const pdfBuffer = fs.readFileSync('./document.pdf'); pdf2md(pdfBuffer) .then(markdown => { console.log(markdown); }) .catch(err => { console.error('Conversion failed:', err); }); ``` -------------------------------- ### Batch Convert PDF Folder to Markdown (CLI) Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Utilize the CLI interface for batch conversion of PDF files within a specified folder to Markdown. Supports recursive conversion of subdirectories. ```bash pdf2md --inputFolderPath=/pdfs --outputFolderPath=/markdown --recursive ``` -------------------------------- ### Convert LineItemBlock to Markdown Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md Demonstrates converting a `LineItemBlock` with multiple items into markdown text. Ensure `BlockType` and related models are imported. ```javascript const BlockType = require('./BlockType'); const lineItemBlock = new LineItemBlock({ items: [ new LineItem({ words: [new Word({ string: 'Hello' })] }), new LineItem({ words: [new Word({ string: 'World' })] }) ], type: BlockType.PARAGRAPH }); const markdown = BlockType.blockToText(lineItemBlock); // Returns: "Hello\nWorld\n" ``` -------------------------------- ### Create PDF to Markdown Transformation Pipeline Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/transformation-system.md Creates the standard transformation pipeline for PDF-to-Markdown conversion. It takes a font map from the PDF parser to detect text formatting. ```typescript function makeTransformations(fontMap: Map): Transformation[] ``` -------------------------------- ### CLI Processing Flow Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Outlines the sequence of operations performed by the CLI, from argument parsing to file conversion and logging. ```text 1. Parse arguments with minimist 2. Validate required arguments (inputFolderPath, outputFolderPath) 3. Validate recursive argument (if present, must be boolean-like) 4. getFileAndFolderPaths() — List PDFs and folders in input 5. If recursive, getAllFileAndFolderPaths() — Recursively list all PDFs 6. Compute output paths with same directory structure 7. makeOutputDirs() — Create all required output directories 8. createMarkdownFiles() — Convert each PDF sequentially 9. Log progress for each file ``` -------------------------------- ### Increase Node.js Heap Size for Memory Usage Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Demonstrates how to increase the Node.js maximum old space size to mitigate memory exhaustion issues when processing large PDFs or batches. ```bash node --max-old-space-size=4096 lib/pdf2md-cli.js ... ``` -------------------------------- ### makeOutputDirs Function Signature Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Provides the signature for the makeOutputDirs function, responsible for creating necessary output directories. ```typescript function makeOutputDirs(allOutputPaths: string[]): void ``` -------------------------------- ### Basic PDF Parsing Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/pdf-parser.md Use this for a straightforward way to parse a PDF document and access its metadata and page count. Ensure the PDF file exists at the specified path. ```javascript const { parse } = require('./util/pdf'); const fs = require('fs'); const pdfBuffer = fs.readFileSync('document.pdf'); parse(pdfBuffer) .then(result => { console.log(`Document has ${result.pages.length} pages`); console.log(`Found ${result.fonts.ids.size} fonts`); console.log(`Title: ${result.metadata.info.Title}`); }); ``` -------------------------------- ### CLI with Custom Heap Size for Large Batches Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/README.md For large conversion batches, increase the Node.js heap size using --max-old-space-size. This command also enables recursive conversion. ```bash node --max-old-space-size=4096 lib/pdf2md-cli.js \ --inputFolderPath=/pdfs \ --outputFolderPath=/markdown \ --recursive ``` -------------------------------- ### Package JSON Configuration Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/configuration.md Defines the main package metadata and dependency configuration for the pdf2md project. ```json { "name": "@opendocsg/pdf2md", "version": "0.2.7", "description": "A PDF to Markdown Converter", "main": "lib/pdf2md.js", "types": "types/pdf2md.d.ts", "bin": { "pdf2md": "lib/pdf2md-cli.js" }, "files": [ "/lib", "/test", "/types" ] } ``` -------------------------------- ### String and Page Item Utility Functions Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/utilities.md Demonstrates the usage of various utility functions for analyzing text, detecting list items, checking for numbers, identifying dot leaders, sorting items by position, and normalizing character codes. These functions are essential for parsing and transforming document content. ```javascript const { isNumber, isListItemCharacter, isNumberedListItem, hasOnly, normalizedCharCodeArray } = require('./string-functions'); const { sortByX, minXFromPageItems } = require('./page-item-functions'); // Analyzing a potential list item const text = '- First item'; if (isListItemCharacter(text[0])) { console.log('Found bullet point'); } // Detecting footnote numbers if (isNumber(text)) { console.log('Pure number:', text); } // Table of contents detection if (hasOnly(text, '.')) { console.log('Dot leader:', text); } // Ordering text in reading order const items = [{ x: 100 }, { x: 50 }, { x: 150 }]; sortByX(items); const leftMargin = minXFromPageItems(items); console.log('Left margin at:', leftMargin); // Pattern matching const normalized = normalizedCharCodeArray('Table of Contents'); console.log('Normalized:', normalized); ``` -------------------------------- ### Block Rendering Functions Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Utilities for rendering and identifying blocks of text. ```APIDOC ## blockToText ### Description Converts a block object into its string representation. ### Method Signature `blockToText(block): string` ### Parameters - **block** (Object) - Required - The block object to convert. ### Returns - **string** - The text representation of the block. ``` ```APIDOC ## isHeadline ### Description Checks if a block type represents a headline. ### Method Signature `isHeadline(type): boolean` ### Parameters - **type** (BlockType) - Required - The block type to check. ### Returns - **boolean** - True if the type is a headline, false otherwise. ``` ```APIDOC ## headlineByLevel ### Description Returns the BlockType corresponding to a given headline level. ### Method Signature `headlineByLevel(level): BlockType` ### Parameters - **level** (number) - Required - The headline level (e.g., 1 for H1). ### Returns - **BlockType** - The BlockType enum value for the specified headline level. ``` -------------------------------- ### Validate Input Folder Argument Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Checks if the 'inputFolderPath' argument is provided. If not, it logs a message and exits without running the conversion. ```javascript if (!argv['inputFolderPath']) { console.log('Please specify inputFolderPath') // Exit without running } ``` -------------------------------- ### PDF Parsing with Progress Callbacks Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/pdf-parser.md Utilize callbacks to monitor the parsing progress, such as when metadata, pages, or fonts are parsed. This is useful for providing user feedback during long parsing operations. ```javascript const callbacks = { metadataParsed: (metadata) => { console.log('Metadata:', metadata.info); }, pageParsed: (pages) => { console.log(`Parsed ${pages.length} pages so far`); }, fontParsed: (fonts) => { console.log(`Total fonts: ${fonts.ids.size}`); }, documentParsed: (doc, pages) => { console.log(`Complete: ${doc.numPages} pages`); } }; parse(pdfBuffer, callbacks) .then(result => { // Use parsed result }); ``` -------------------------------- ### Node.js Parameters for Large Batches Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md When processing large numbers of PDF files that may cause memory issues, increase the Node.js heap size using the --max-old-space-size flag. ```bash node --max-old-space-size=4096 lib/pdf2md-cli.js \ --inputFolderPath=/data/pdfs \ --outputFolderPath=/data/markdown \ --recursive ``` -------------------------------- ### Creating LineItem from Text String Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Create a LineItem using a plain text string. The constructor will automatically split the string into words based on spaces. This is a convenient way to initialize a line with simple text content. ```javascript const lineItem = new LineItem({ x: 50, y: 100, width: 200, height: 12, text: 'Hello World' // Auto-converted to words }); ``` -------------------------------- ### Validate Output Folder Argument Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Checks if the 'outputFolderPath' argument is provided. If not, it logs a message and exits without running the conversion. ```javascript if (!argv['outputFolderPath']) { console.log('Please specify outputFolderPath') // Exit without running } ``` -------------------------------- ### Instantiate and Use TextItemLineGrouper Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/line-conversion.md Instantiate TextItemLineGrouper with a specific distance threshold and use it to group TextItems into lines. The grouper sorts items by Y coordinate and groups those within half of the mostUsedDistance. ```javascript const grouper = new TextItemLineGrouper({ mostUsedDistance: 14 }); const lines = grouper.group(textItems); // lines[0] = [item1, item2, item3] (same Y) // lines[1] = [item4, item5] (same Y, different from line 0) // lines[2] = [item6] (alone) ``` -------------------------------- ### makeTransformations Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/transformation-system.md Creates the standard transformation pipeline for PDF-to-Markdown conversion. This function takes a font map and returns an ordered array of transformation instances. ```APIDOC ## Function: makeTransformations ### Description Creates the standard transformation pipeline for PDF-to-Markdown conversion. This function takes a font map and returns an ordered array of transformation instances. ### Signature ```javascript function makeTransformations(fontMap: Map): Transformation[] ``` ### Parameters #### Path Parameters - **fontMap** (Map) - Required - Font ID → font object mapping from PDF parser. Used to detect text formatting (bold, italic). ### Return **Type**: `Transformation[]` Ordered array of transformation instances. Each transformation is applied sequentially in order: 1. **CalculateGlobalStats** — Analyzes font sizes, line heights, spacing to establish document baseline 2. **CompactLines** — Groups text items on same Y coordinate into line items 3. **RemoveRepetitiveElements** — Filters out headers, footers, and repeated text 4. **VerticalToHorizontal** — Handles text rotations and vertical text 5. **DetectTOC** — Identifies table of contents pages and extracts links 6. **DetectHeaders** — Detects headings by font size relative to baseline 7. **DetectListItems** — Identifies bullet points and numbered lists 8. **GatherBlocks** — Groups lines into semantic blocks by spacing 9. **DetectCodeQuoteBlocks** — Marks code blocks and block quotes 10. **DetectListLevels** — Determines nesting levels for multi-level lists 11. **ToTextBlocks** — Converts line items into final text block objects 12. **ToMarkdown** — Renders text blocks as Markdown-formatted strings ``` -------------------------------- ### Line-by-Line Text Rendering Function Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/block-type-rendering.md Renders an array of line items sequentially, applying word-level formatting and handling spacing rules. It manages opening and closing format symbols to produce Markdown output. ```javascript function linesToText(lineItems, disableInlineFormats) { var text = '' var openFormat = null lineItems.forEach((line, lineIndex) => { line.words.forEach((word, i) => { const wordType = word.type const wordFormat = word.format // Close format if changed if (openFormat && (!wordFormat || wordFormat !== openFormat)) { text += openFormat.endSymbol openFormat = null } // Add spacing if (i > 0 && !(wordType?.attachWithoutWhitespace) && !isPunctuation(word.string)) { text += ' ' } // Open format if needed if (wordFormat && !openFormat && !disableInlineFormats) { openFormat = wordFormat text += openFormat.startSymbol } // Render word if (wordType && (!disableInlineFormats || wordType.plainTextFormat)) { text += wordType.toText(word.string) } else { text += word.string } }) // Close format at line end if (openFormat && (lineIndex === lineItems.length - 1 || firstFormat(lineItems[lineIndex + 1]) !== openFormat)) { text += openFormat.endSymbol openFormat = null } text += '\n' }) return text } ``` -------------------------------- ### Main API Function Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md The primary function for converting PDF buffers to Markdown. ```APIDOC ## pdf2md ### Description Converts a PDF buffer into Markdown format. ### Method Signature `pdf2md(pdfBuffer, callbacks?): Promise` ### Parameters - **pdfBuffer** (Buffer) - Required - The buffer containing the PDF data. - **callbacks** (Object) - Optional - Callback functions for progress or events. ### Returns - **Promise** - A promise that resolves with the Markdown string. ``` -------------------------------- ### getAllFileAndFolderPaths Function Signature Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Defines the signature for the getAllFileAndFolderPaths function, used for recursively traversing nested folders. ```typescript function getAllFileAndFolderPaths( filePaths: string[], folderPaths: string[], recursive: boolean ): [string[], string[]] ``` -------------------------------- ### LineItem Object Structure Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/line-conversion.md Illustrates the structure of a LineItem object created by the `compact()` method. This object aggregates text items into a single line, providing properties like text content, word details (including formatting and type), and parsed elements such as footnote links. ```json { "x": 0, // X of first item "y": 100, // Y coordinate "width": 108, // Sum of all item widths "height": 12, // Max height "words": [ "Word { string: 'Hello', format: WordFormat.BOLD }", "Word { string: 'world' }", "Word { string: '1', type: WordType.FOOTNOTE_LINK }" ], "parsedElements": { "footnoteLinks": [1], "footnotes": [], "containLinks": false, "formattedWords": 1 } } ``` -------------------------------- ### Prepend Prefix While Preserving Whitespace Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/utilities.md This utility prepends a prefix to a string, intelligently preserving any leading whitespace from the original string by placing it at the beginning of the result. Useful for maintaining indentation. ```javascript const { prefixAfterWhitespace } = require('./string-functions'); prefixAfterWhitespace('>>', 'hello') // '>>hello' prefixAfterWhitespace('>>', ' hello') // ' >> hello' ``` -------------------------------- ### Creating LineItem from Word Array Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Instantiate a LineItem by providing an array of Word objects, along with positional and dimensional properties. This allows for detailed control over each word's formatting. ```javascript const lineItem = new LineItem({ x: 50, y: 100, width: 200, height: 12, words: [ new Word({ string: 'Bold', format: WordFormat.BOLD }), new Word({ string: 'text' }) ] }); ``` -------------------------------- ### Footnote and Number Detection Logic Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/line-conversion.md Illustrates the logic within WordDetectionStream for identifying and classifying numbers as footnote links, footnotes, or regular text based on their vertical position. ```javascript if (stashedNumber) { const joinedNumber = stash.map(item => item.text).join('').trim() if (stash[0].y > this.firstY) { // Footnote link (number below first line) results.push(new Word({ string: joinedNumber, type: WordType.FOOTNOTE_LINK })) this.footnoteLinks.push(parseInt(joinedNumber)) } else if (this.currentItem && this.currentItem.y < stash[0].y) { // Footnote (number above current position) results.push(new Word({ string: joinedNumber, type: WordType.FOOTNOTE })) this.footnotes.push(joinedNumber) } else { // Regular number, convert to text this.copyStashItemsAsText(stash, results) } } ``` -------------------------------- ### prefixAfterWhitespace Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/utilities.md Prepends a prefix to a string while preserving the original string's leading whitespace structure. ```APIDOC ## prefixAfterWhitespace(prefix, string): string ### Description Prepends a prefix to a string, preserving leading whitespace. If the original string starts with spaces, those spaces are moved to the front of the result. ### Parameters #### Path Parameters - **prefix** (string) - Required - The string to prepend. - **string** (string) - Required - The target string. ### Returns - **string** - The modified string with the prefix prepended and whitespace preserved. ### Example ```javascript const { prefixAfterWhitespace } = require('./string-functions'); prefixAfterWhitespace('>>', 'hello') // '>>hello' prefixAfterWhitespace('>>', ' hello') // ' >> hello' ``` ``` -------------------------------- ### pdf2md Usage with URL Input Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/pdf2md.md Illustrates converting a PDF document directly from a URL using the pdf2md function. The resulting Markdown is then processed. ```javascript const pdf2md = require('@opendocsg/pdf2md'); pdf2md('https://example.com/document.pdf') .then(markdown => { // Process markdown }); ``` -------------------------------- ### File Naming Convention Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/configuration.md Output filenames are generated by replacing the input PDF extension with .md, after removing the input folder prefix and prepending the output folder path. ```javascript // Input: /data/pdfs/folder/document.pdf // Output: /data/markdown/folder/document.md ``` -------------------------------- ### minXFromPageItems Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/utilities.md Finds the minimum X coordinate among a collection of page items. This is helpful for determining the left boundary or indentation of content. ```APIDOC ## minXFromPageItems(pageItems): number ### Description Finds the minimum X coordinate among all items. ### Parameters - `pageItems` (Array): An array of page item objects, each expected to have an `x` property. ### Return - `number`: The smallest X coordinate found among the provided page items. ### Example ```javascript const { minXFromPageItems } = require('./page-item-functions'); const minX = minXFromPageItems([ { x: 100 }, { x: 50 }, { x: 150 } ]); // 50 ``` ``` -------------------------------- ### Convert PDF Buffer to Markdown (JavaScript) Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Use the pdf2md function with a PDF buffer to convert a single PDF file to Markdown. Requires the pdf2md library and optionally accepts callbacks for progress or error handling. ```javascript const pdf2md = require('@opendocsg/pdf2md'); const markdown = await pdf2md(pdfBuffer, callbacks); ``` -------------------------------- ### text() Method Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/text-models.md Retrieves the full text of the line item, with words joined by spaces. ```APIDOC #### text(): string Returns joined text of all words separated by spaces. ```javascript const lineItem = new LineItem({ words: [ new Word({ string: 'Hello' }), new Word({ string: 'World' }) ] }); lineItem.text() // Returns: "Hello World" ``` ``` -------------------------------- ### Transformation processing flow Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/transformation-system.md Illustrates the sequence of operations applied to TextItems to convert them into Markdown strings, including steps like line compaction, element removal, orientation normalization, and block detection. ```plaintext pages (TextItem[]) → CalculateGlobalStats (analyzes heights, fonts, spacing) → CompactLines (groups TextItems into LineItems) → RemoveRepetitiveElements (filters headers/footers) → VerticalToHorizontal (normalizes text orientation) → DetectTOC (identifies contents) → DetectHeaders (font-size based heading detection) → DetectListItems (bullet/number detection) → GatherBlocks (spacing-based grouping) → DetectCodeQuoteBlocks (code/quote detection) → DetectListLevels (nesting analysis) → ToTextBlocks (block object creation) → ToMarkdown (final rendering) → pages (Markdown strings) ``` -------------------------------- ### Block Rendering Utilities Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Utilities for rendering PDF blocks to text and checking block properties. blockToText converts a block to a string, isHeadline checks if a block is a headline, and headlineByLevel retrieves a headline type by its level. ```javascript // Block rendering blockToText(block): string isHeadline(type): boolean headlineByLevel(level): BlockType ``` -------------------------------- ### Handle Per-File Conversion Errors Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/api-reference/cli-interface.md Uses a try-catch block to handle errors during individual PDF to Markdown conversion. Errors are logged to stderr, but processing continues for subsequent files. ```javascript try { const text = await pdf2md(new Uint8Array(pdfBuffer), callbacks) // Write output } catch (err) { console.error(err) // Continue to next file } ``` -------------------------------- ### Convert PDF Buffer to Markdown String Source: https://github.com/opengovsg/pdf2md/blob/master/_autodocs/INDEX.md Primary library function for converting a PDF buffer to a Markdown string. Supports optional callbacks for progress monitoring. ```typescript pdf2md(pdfBuffer, callbacks?: object): Promise ```