### Print Installed Version Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Displays the currently installed version of the Percollate tool. ```bash percollate --version ``` -------------------------------- ### CLI Invocation Example Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md Demonstrates how to invoke the Percollate CLI with multiple operands and an output file option. ```bash percollate pdf url1 url2 url3 --output bundle.pdf ``` -------------------------------- ### Install Project Dependencies Source: https://github.com/danburzo/percollate/blob/main/CONTRIBUTING.md Install the necessary Node.js dependencies for the Percollate project. ```bash npm install ``` -------------------------------- ### Nunjucks Template Example Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md An example of Nunjucks template syntax for rendering content. It demonstrates how to access context variables like title, author, and date, and how to loop through items. ```nunjucks {{ title }} {{ author }} {{ date | humandate }} {% for item in items %}
{{ item.content }}
{% endfor %} ``` -------------------------------- ### Run Percollate CLI Example Source: https://github.com/danburzo/percollate/blob/main/CONTRIBUTING.md Execute the Percollate CLI to generate a PDF from a URL. Ensure the script has execution permissions. ```bash ./cli.js pdf --output some.pdf http://example.com ``` -------------------------------- ### Generate PDF with Output Path Source: https://github.com/danburzo/percollate/blob/main/README.md Example of generating a PDF file from a URL and specifying the output file name. ```bash percollate pdf https://example.com -o my-example.pdf ``` -------------------------------- ### API: EPUB with Inline Images Source: https://github.com/danburzo/percollate/blob/main/_autodocs/README.md API usage example for generating a self-contained EPUB with all images inlined. ```javascript import { configure, epub } from 'percollate'; configure(); await epub(['https://example.com'], { output: 'article.epub', inline: true // Self-contained }); ``` -------------------------------- ### Install Percollate from AUR Source: https://github.com/danburzo/percollate/blob/main/README.md Install the community-maintained package for Percollate from the Arch User Repository using an AUR helper. ```bash yay -S nodejs-percollate ``` -------------------------------- ### Install Percollate Globally Source: https://github.com/danburzo/percollate/blob/main/README.md Install the Percollate command-line tool globally using npm. Ensure you have Node.js 14.17.0 or later. ```bash npm install -g percollate ``` -------------------------------- ### Feed Processing Integration Example Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-feeds.md Demonstrates how to use `isFeed` and `processFeed` to parse RSS feed XML. It then processes each entry, creating a JSDOM for each item and cleaning it up. ```javascript import { isFeed, processFeed } from './src/feeds.js'; const dom = new JSDOM(feedXmlBuffer, { contentType: 'text/xml', url: feedUrl }); if (isFeed(dom.window.document)) { const entries = processFeed(dom.window.document); // Process each entry as a separate article const items = await Promise.all( entries.map(entry => { const itemDOM = new JSDOM(wrapHTMLFragment(entry), { url: entry.url }); return cleanupItem(itemDOM, options); }) ); } ``` -------------------------------- ### API: Generate PDF Source: https://github.com/danburzo/percollate/blob/main/_autodocs/README.md API usage example for generating a PDF from a list of URLs with custom output, title, and author options. ```javascript import { configure, pdf } from 'percollate'; configure(); const result = await pdf(['https://example.com'], { output: 'page.pdf', title: 'My Page', author: 'Jane Doe' }); ``` -------------------------------- ### Basic EPUB Conversion Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts a single URL to an EPUB file with a specified title. Ensure 'percollate' is installed and configured. ```javascript import { configure, epub } from 'percolate'; configure(); // Basic EPUB await epub(['https://example.com'], { output: 'book.epub', title: 'My Book' }); ``` -------------------------------- ### HTML Conversion to File Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts a web page URL to an HTML file. Ensure 'percollate' is installed and configured. ```javascript import { configure, html } from 'percollate'; configure(); // Save to file await html(['https://example.com'], { output: 'page.html' }); ``` -------------------------------- ### API: Custom Styling for PDF Source: https://github.com/danburzo/percollate/blob/main/_autodocs/README.md API usage example for generating a PDF with custom page size, margins, text alignment, and external CSS/template files. ```javascript await pdf(['https://example.com'], { output: 'custom.pdf', css: ' @page { size: A3 landscape; margin: 1cm; } .article__content p { text-align: left; } ', template: './my-template.html', style: './my-styles.css' }); ``` -------------------------------- ### Set Puppeteer Product Before Installation Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Configures the browser product (Chrome or Firefox) that Puppeteer will use before installing Percollate. This determines which browser binaries are downloaded. ```bash PUPPETEER_PRODUCT=firefox npm install percollate ``` -------------------------------- ### HTML Conversion to Stdout Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts a web page URL to HTML and outputs it to standard output. Ensure 'percollate' is installed and configured. ```javascript // Output to stdout await html(['https://example.com'], { output: '-' }); ``` -------------------------------- ### Style Header/Footer Templates Source: https://github.com/danburzo/percollate/blob/main/README.md Example of applying CSS styles to a footer template. Styles for header/footer templates do not inherit from the main page. ```css .footer-template { font-size: 10pt; font-weight: bold; } ``` -------------------------------- ### Basic Markdown Conversion Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts a web page URL to a Markdown file. Ensure 'percollate' is installed and configured. ```javascript import { configure, md } from 'percollate'; configure(); // Basic Markdown output await md(['https://example.com'], { output: 'article.md' }); ``` -------------------------------- ### Custom Markdown Formatting Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts a web page to Markdown with custom formatting options for emphasis and strong text. Ensure 'percollate' is installed and configured. ```javascript // Custom Markdown formatting await md(['https://example.com'], { output: 'custom.md', 'md.emphasis': 'asterisk', 'md.strong': 'asterisk' }); ``` -------------------------------- ### EPUB Bundling Process Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md Describes the steps involved in creating an EPUB file using the epubgen library and an archiver for zip structure. ```javascript items[] → bundleEpub() ↓ epubgen(data, output_path, options) ↓ archiver('zip') → EPUB structure: mimetype META-INF/ container.xml com.apple.ibooks.display-options.xml OEBPS/ content.opf (metadata) nav.xhtml (navigation) toc.ncx (table of contents) style.css {item.id}.xhtml (articles) cover.png (if enabled) rr-{uuid}.* (remote resources if bundled) ↓ fs.createWriteStream(output_path) ``` -------------------------------- ### configure() Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Initializes the percollate environment by setting up the Nunjucks templating engine with custom filters. This function must be called once before any bundle operations. ```APIDOC ## Function: `configure()` ### Description Initializes the percollate environment by setting up the Nunjucks templating engine with custom filters. This function must be called once before any bundle operations. ### Returns `void` ### Example ```javascript import { configure, pdf } from 'percollate'; configure(); await pdf(['https://example.com'], { output: 'example.pdf' }); ``` ``` -------------------------------- ### Main Library API Functions Source: https://github.com/danburzo/percollate/blob/main/_autodocs/README.md Functions for initializing percollate and generating outputs in various formats. Use `configure()` to initialize before calling output generation functions like `pdf()`, `epub()`, `html()`, or `md()`. ```javascript // Initialize percollate configure(): void // Generate outputs async pdf(urls, options): Promise<{items, options} | null> async epub(urls, options): Promise<{items, options} | null> async html(urls, options): Promise<{items, options} | null> async md(urls, options): Promise<{items, options} | null> ``` -------------------------------- ### Set Bundle Title Source: https://github.com/danburzo/percollate/blob/main/README.md Generate an EPUB file from multiple URLs and provide a custom title for the bundle. ```bash percollate epub http://example.com/page-1 http://example.com/page-2 --title="Best Of Example" ``` -------------------------------- ### Show Percollate Help Text Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Displays all available commands and options for the Percollate CLI. Use this to understand the full range of functionalities. ```bash percollate --help ``` -------------------------------- ### Importing All Main Percollate Functions Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Imports and demonstrates the usage of core Percollate functions for configuration and content conversion. ```javascript import { configure, pdf, epub, html, md } from 'percolate'; configure(); ``` -------------------------------- ### Markdown Conversion to Stdout Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts a web page URL to Markdown and outputs it to standard output. Ensure 'percollate' is installed and configured. ```javascript // To stdout await md(['https://example.com'], { output: '-' }); ``` -------------------------------- ### Convert Web Pages to PDF (Basic) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts a single web page URL to a PDF file. Ensure configure() has been called prior to this operation. ```javascript import { configure, pdf } from 'percollate'; configure(); // Basic usage await pdf(['https://example.com'], { output: 'example.pdf' }); ``` -------------------------------- ### Convert Single Web Page to PDF Source: https://github.com/danburzo/percollate/blob/main/README.md Basic command to convert a single web page to a PDF file. ```bash percollate pdf --output=some.pdf https://example.com ``` -------------------------------- ### outputPath(items, options, ext, cache?) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-utilities.md Generates an output file path for a bundle based on provided items, options, extension, and an optional cache. It handles different output scenarios like individual files or a single bundle. ```APIDOC ## Function: `outputPath(items, options, ext, cache?)` Generates an output file path for the bundle. ### Parameters | Parameter | Type | Description | |-----------|------|-------------| | items | `ParsedItem[]` | Array of parsed items (at least one) | | options | object | Must have `output` and `individual` properties | | ext | `string` | File extension (e.g., `.pdf`, `.epub`) | | cache | object | Slug cache to prevent filename collisions | ### Returns File path string based on: 1. If `individual` and `output`: `{output}-{slugified-title}{ext}` 2. If `output`: `{output}` (as-is) 3. If not individual and multiple items: `percollate-{timestamp}{ext}` 4. If single item: `{slugified-title}{ext}` ### Example ```javascript import outputPath from './src/util/output-path.js'; const items = [{ title: 'My Article' }]; // With explicit output outputPath(items, {output: 'custom.pdf'}, '.pdf'); // 'custom.pdf' // Individual files outputPath(items, {output: 'output.pdf', individual: true}, '.pdf', {}); // 'output-my-article.pdf' // Auto-generated outputPath(items, {}, '.pdf'); // 'my-article.pdf' ``` ``` -------------------------------- ### Language Utilities Exports Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Language utility functions from `src/util/language.js` for converting text to ISO 639-1 codes and getting language attributes. ```APIDOC ## Language Utilities Exports #### `textToIso6391(text: string)` - **Type**: function - **Parameters**: `text` (string) - **Returns**: `string | null` #### `getLanguageAttribute(doc: Document)` - **Type**: function - **Parameters**: `doc` (Document) - **Returns**: `string | null` ``` -------------------------------- ### Run ESLint for Code Style Source: https://github.com/danburzo/percollate/blob/main/CONTRIBUTING.md Execute ESLint to enforce coding style rules. ```bash npm run lint ``` -------------------------------- ### Use Percollate PDF API with Options Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Demonstrates how to use the Percollate PDF function as a library, passing configuration options as the second argument. This allows programmatic control over output, metadata, and features like table of contents. ```javascript import { pdf } from 'percollate'; const result = await pdf(['https://example.com'], { output: 'my.pdf', title: 'My Document', author: 'Jane Doe', toc: true, css: '@page { size: A4; }', debug: true }); ``` -------------------------------- ### Self-Contained HTML with Embedded Images Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Creates a self-contained HTML file with images embedded as base64 data URLs. Ensure 'percollate' is installed and configured. ```javascript // Self-contained HTML with embedded images await html(['https://example.com'], { output: 'standalone.html', inline: true }); ``` -------------------------------- ### epub(urls, options) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts one or more web pages into an EPUB file. It accepts an array of URLs or file paths and an optional options object for customization. The function returns a promise that resolves to an object containing the parsed items and the options used, or null if no valid URLs were provided. ```APIDOC ## Function: `epub(urls, options)` Converts one or more web pages into an EPUB file. ### Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | urls | `string[]` or `stream.Readable[]` | Yes | — | Array of URLs, file paths, or stdin streams | | options | `EPUBOptions` | No | `{}` | Configuration object for EPUB generation | ### Options Object (EPUBOptions) Inherits all options from PDFOptions, except: - `browser`: Ignored for EPUB (Puppeteer used only for cover image) - `hyphenate`: defaults to `false` (not `true`) Additional EPUB-specific options: | Key | Type | Default | Description | |-----|------|---------|-------------| | inline | `boolean` | false | Embed images as base64 data URLs instead of fetching them during reading | | xhtml | `boolean` | true | Serialize content as XHTML (required for EPUB spec compliance) | | mapRemoteResources | `boolean` | !inline | Bundle external images with EPUB; disabled if inline is true | ### Returns Promise resolving to an object containing: - `items`: array of `ParsedItem` objects - `options`: the final options object used - Returns `null` if no valid URLs were provided ### Example ```javascript import { configure, epub } from 'percollate'; configure(); // Basic EPUB await epub(['https://example.com'], { output: 'book.epub', title: 'My Book' }); // Multiple articles await epub( [ 'https://example.com/article1', 'https://example.com/article2' ], { output: 'articles.epub', title: 'Article Collection', author: 'Editor' } ); // Self-contained EPUB with embedded images await epub(['https://example.com'], { output: 'offline.epub', inline: true }); ``` ``` -------------------------------- ### Get Language Attribute from Document Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-utilities.md Retrieves the language code specified in the HTML document's `lang` attribute. Returns null if the attribute is not present. ```javascript import { getLanguageAttribute } from './src/util/language.js'; import { JSDOM } from 'jsdom'; const dom = new JSDOM(''); getLanguageAttribute(dom.window.document); // 'en' const dom2 = new JSDOM(''); getLanguageAttribute(dom2.window.document); // null ``` -------------------------------- ### Image Fetching in EPUB Archiving Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md Demonstrates the sequential fetching and appending of remote resources (images) to an EPUB archive. Each image is fetched and then added to the archiver with its mapped name. ```javascript for (let resource of remoteResources) { fetch(resource.original) .then(body => archiver.append(body, {name: resource.mapped})) } ``` -------------------------------- ### EPUB Conversion of Multiple Articles Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts multiple article URLs into a single EPUB file, specifying title and author. Ensure 'percollate' is installed and configured. ```javascript // Multiple articles await epub( [ 'https://example.com/article1', 'https://example.com/article2' ], { output: 'articles.epub', title: 'Article Collection', author: 'Editor' } ); ``` -------------------------------- ### Self-Contained EPUB with Embedded Images Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Creates a self-contained EPUB where images are embedded as base64 data URLs. This is useful for offline reading. Ensure 'percollate' is installed and configured. ```javascript // Self-contained EPUB with embedded images await epub(['https://example.com'], { output: 'offline.epub', inline: true }); ``` -------------------------------- ### EPUB Generation Options Source: https://github.com/danburzo/percollate/blob/main/_autodocs/types.md Configuration object for EPUB generation. Extends PDFOptions and includes specific fields for XHTML serialization and remote resource mapping. ```typescript { xhtml?: boolean; mapRemoteResources?: boolean; // ... all PDFOptions fields } ``` -------------------------------- ### Get File Extension from MIME Type Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-utilities.md Converts a MIME type string into its corresponding file extension. Useful for determining the file format based on its MIME type. ```javascript import { extForMimetype } from './src/util/file-mimetype.js'; extForMimetype('image/png'); // '.png' extForMimetype('image/jpeg'); // '.jpg' extForMimetype('image/webp'); // '.webp' ``` -------------------------------- ### html(urls, options) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts one or more web pages into a single HTML file. It takes an array of URLs or file paths and an optional configuration object. The function returns a promise that resolves to an object containing parsed items and options, or null if no valid URLs are supplied. ```APIDOC ## Function: `html(urls, options)` Converts one or more web pages into a single HTML file. ```javascript async function html( urls: string[] | Readable[], options?: HTMLOptions ): Promise<{items: ParsedItem[], options: HTMLOptions} | null> ``` ### Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | urls | `string[]` or `stream.Readable[]` | Yes | — | Array of URLs, file paths, or stdin streams | | options | `HTMLOptions` | No | `{}` | Configuration object for HTML generation | ### Options Object (HTMLOptions) Same as PDFOptions, except: - `hyphenate`: defaults to `false` - `browser`: Ignored (not used for HTML) - `output`: Can be `-` to write to stdout - `sandbox`: Ignored (not used for HTML) ### Returns Promise resolving to an object containing: - `items`: array of `ParsedItem` objects - `options`: the final options object used - Returns `null` if no valid URLs were provided ### Example ```javascript import { configure, html } from 'percollate'; configure(); // Save to file await html(['https://example.com'], { output: 'page.html' }); // Output to stdout await html(['https://example.com'], { output: '-' }); // Self-contained HTML with embedded images await html(['https://example.com'], { output: 'standalone.html', inline: true }); ``` ``` -------------------------------- ### Override Default Fonts with --css Source: https://github.com/danburzo/percollate/blob/main/README.md Overrides the default font stacks for main and alternative fonts using the --css option. Ensure fonts are installed locally. ```bash percollate pdf --css ":root { --main-font: 'PT Serif'; --alt-font: Roboto; }" http://example.com ``` -------------------------------- ### Bundle Multiple Web Pages into a Single PDF Source: https://github.com/danburzo/percollate/blob/main/README.md Command to bundle several web pages into one PDF document. ```bash percollate pdf --output=some.pdf https://example.com/page1 https://example.com/page2 ``` -------------------------------- ### addExif Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-other-modules.md Embeds metadata (Title, Author) into a PDF file. It loads the PDF, sets the specified metadata, and returns the modified PDF data as a Uint8Array. Existing metadata is not updated; it starts fresh. ```APIDOC ## Function: addExif ### Description Embeds metadata (Title, Author) into a PDF file. It loads the PDF, sets the specified metadata, and returns the modified PDF data as a Uint8Array. Existing metadata is not updated; it starts fresh. ### Parameters - **existingPdfBytes** (`Buffer`) - Required - PDF file data as Buffer - **metaData** (`object`) - Optional - Metadata to embed; keys: `Title`, `Author` ### Returns - `Promise` - Resolves to a `Uint8Array` containing the modified PDF data (with metadata embedded). ### Example ```javascript import { readFileSync, writeFileSync } from 'fs'; import addExif from './src/exif.js'; const pdfBuffer = readFileSync('input.pdf'); const newPdf = await addExif(pdfBuffer, { Title: 'My Document', Author: 'Jane Doe' }); writeFileSync('output.pdf', newPdf); ``` ### Source `src/exif.js:8–19` ``` -------------------------------- ### Bundle URLs from a Text File into a PDF Source: https://github.com/danburzo/percollate/blob/main/README.md Uses a newline-delimited text file of URLs and pipes them to percollate for PDF bundling. ```bash cat urls.txt | xargs percollate pdf --output=some.pdf ``` -------------------------------- ### Select Browser for Rendering Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Specifies the rendering engine (e.g., Firefox) for PDF and cover page generation. Note that Firefox rendering is experimental and has limitations. ```bash percollate pdf --browser=firefox https://example.com ``` -------------------------------- ### Parse CLI Arguments Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-other-modules.md Demonstrates parsing command-line arguments into command, options, and operands using the `cliopts` function. This is useful for building command-line interfaces. ```javascript import { cliopts } from './src/cli-opts.js'; const args = ['pdf', '--output=my.pdf', '--title=My Page', 'https://example.com']; const result = cliopts(args); // { // command: 'pdf', // opts: {output: 'my.pdf', title: 'My Page'}, // operands: ['https://example.com'] // } ``` -------------------------------- ### Basic Command Structure Source: https://github.com/danburzo/percollate/blob/main/README.md General syntax for using percollate commands, specifying a command, options, and one or more URLs. ```bash percollate [options] url [url]... ``` -------------------------------- ### Convert Web Feed Entries to Individual EPUB Files Source: https://github.com/danburzo/percollate/blob/main/README.md Creates separate EPUB files for each entry in an Atom or RSS feed using the --individual flag. ```bash percollate epub --individual https://example.com/posts.xml ``` -------------------------------- ### Use Custom CSS Stylesheet Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Employ the --style flag to specify a custom CSS file for styling the output. This replaces the default stylesheet entirely. The path is relative to the current directory. ```bash percollate pdf --style=./my-styles.css https://example.com ``` -------------------------------- ### CLI: Multiple URLs with TOC Source: https://github.com/danburzo/percollate/blob/main/_autodocs/README.md Command-line interface command to generate a PDF from multiple URLs, including a table of contents with a specified level. ```bash percollate pdf --title="Collection" --toc --toc-level=2 \ https://example.com/article1 \ https://example.com/article2 ``` -------------------------------- ### Configure Percollate Environment Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Initializes the percollate environment by setting up the Nunjucks templating engine. This function must be called once before any bundle operations. ```javascript import { configure, pdf } from 'percollate'; configure(); await pdf(['https://example.com'], { output: 'example.pdf' }); ``` -------------------------------- ### CLI: Basic PDF Generation Source: https://github.com/danburzo/percollate/blob/main/_autodocs/README.md Command-line interface command to generate a PDF from a single URL. ```bash percollate pdf --output=article.pdf https://example.com ``` -------------------------------- ### Strategy Pattern for Output Formatting Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md Abstracts the output format by using a strategy pattern. This allows the same input to be processed into different output formats (PDF, EPUB, HTML, Markdown) by selecting the appropriate bundling function. ```javascript const strategies = { pdf: bundlePdf, epub: bundleEpub, html: bundleHtml, md: bundleMd }; await strategies[command](items, options); ``` -------------------------------- ### md(urls, options) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts one or more web pages into Markdown format. This function accepts an array of URLs or file paths and an optional configuration object. It returns a promise that resolves to an object containing Markdown-formatted items and options, or null if no valid URLs were provided. ```APIDOC ## Function: `md(urls, options)` Converts one or more web pages into Markdown format. ```javascript async function md( urls: string[] | Readable[], options?: MarkdownOptions ): Promise<{items: ParsedItem[], options: MarkdownOptions} | null> ``` ### Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | urls | `string[]` or `stream.Readable[]` | Yes | — | Array of URLs, file paths, or stdin streams | | options | `MarkdownOptions` | No | `{}` | Configuration object for Markdown generation | ### Options Object (MarkdownOptions) Same as HTMLOptions, plus: | Key | Type | Default | Description | |-----|------|---------|-------------| | md.* | `string` or `boolean` | See defaults | Options passed to `mdast-util-to-markdown`. Use `--md.option=value` syntax in CLI. Keys: `bullet`, `bulletOther`, `bulletOrdered`, `bulletOrderedOther`, `closeAtx`, `emphasis`, `fence`, `fences`, `incrementListMarker`, `listItemIndent`, `quote`, `resourceLink`, `rule`, `ruleRepetition`, `ruleSpaces`, `setext`, `strong`, `tightDefinitions` | Default Markdown options: ```javascript { fences: true, emphasis: '_', strong: '_', resourceLink: true, rule: '-' } ``` ### Returns Promise resolving to an object containing: - `items`: array of `ParsedItem` objects (Markdown variant) - `options`: the final options object used - Returns `null` if no valid URLs were provided ### Example ```javascript import { configure, md } from 'percollate'; configure(); // Basic Markdown output await md(['https://example.com'], { output: 'article.md' }); // To stdout await md(['https://example.com'], { output: '-' }); // Custom Markdown formatting await md(['https://example.com'], { output: 'custom.md', 'md.emphasis': 'asterisk', 'md.strong': 'asterisk' }); ``` ``` -------------------------------- ### Configuration Options Source: https://github.com/danburzo/percollate/blob/main/_autodocs/INDEX.txt Documents over 40 command-line and environment variable options for configuring Percollate's behavior, including output, bundling, content processing, and styling. ```APIDOC ## Configuration Options ### Description Provides a comprehensive reference for all command-line options and environment variables available for configuring Percollate's behavior across various aspects of content processing and output generation. ### Option Categories - **Command-line Options**: Over 40 options documented. - **Output Options**: `--output`, `--individual`, `--wait`, etc. - **Bundling Options**: `--title`, `--author`, `--toc`, etc. - **Content Processing**: `--url`, `--no-amp`, `--unsafe`, etc. - **Styling Options**: `--style`, `--css`, `--template`. - **Table of Contents Options** - **Cover Page Options** - **Text Processing Options** - **Image Options** - **Browser/Rendering Options** - **Markdown-Specific Options** - **Debug/Help Options** - **Environment Variables** - **API Usage Mapping** ### Documentation Location Detailed documentation for all options can be found in `configuration.md`. ``` -------------------------------- ### Convert Web Pages to PDF from Stdin Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Reads HTML content from stdin and converts it to a PDF file, specifying the base URL for resolving relative links. Requires prior configuration. ```javascript import { createReadStream } from 'fs'; import { configure, pdf } from 'percollate'; configure(); const stream = createReadStream('article.html'); await pdf([stream], { output: 'stdin.pdf', url: 'https://example.com' }); ``` -------------------------------- ### Clone Percollate Repository Source: https://github.com/danburzo/percollate/blob/main/CONTRIBUTING.md Clone the Percollate repository to your local machine to begin development. ```bash git clone git@github.com:danburzo/percollate.git cd percollate ``` -------------------------------- ### Convert Web Feed to EPUB Source: https://github.com/danburzo/percollate/blob/main/README.md Processes an Atom or RSS feed and creates an EPUB book from its entries. ```bash percollate epub https://example.com/posts.xml ``` -------------------------------- ### cliopts Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Parses command-line arguments into command, options, and operands. ```APIDOC ## cliopts ### Description Parses command-line arguments into command, options, and operands. ### Parameters - **args** (string[]) - An array of command-line arguments. ### Returns - **{command, opts, operands}** - An object containing the parsed command, options, and operands. ``` -------------------------------- ### Browser Rendering Workflow Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md Illustrates the workflow for browser rendering using Puppeteer. A single browser instance is launched, new pages are created for specific tasks like PDF generation or screenshots, and the browser is closed upon completion. ```javascript const browser = await launch(options); const page = await browser.newPage(); await page.pdf(pdfOptions); // or page.screenshot() for cover await browser.close(); ``` -------------------------------- ### CLI: EPUB from Feed Source: https://github.com/danburzo/percollate/blob/main/_autodocs/README.md Command-line interface command to generate an EPUB from an RSS or Atom feed URL, specifying a custom title. ```bash percollate epub --title="My Feed" https://example.com/feed.xml ``` -------------------------------- ### epub(urls, options) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts one or more web pages into an EPUB file. It accepts an array of URLs or readable streams and an optional configuration object for EPUB generation. ```APIDOC ## Function: `epub(urls, options)` ### Description Converts one or more web pages into an EPUB file. It accepts an array of URLs or readable streams and an optional configuration object for EPUB generation. ### Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | urls | `string[]` or `stream.Readable[]` | Yes | — | Array of URLs, file paths, or stdin streams | | options | `EPUBOptions` | No | `{}` | Configuration object for EPUB generation | ``` -------------------------------- ### pdf(urls, options) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Converts one or more web pages into a PDF file. It accepts an array of URLs or readable streams and an optional configuration object for PDF generation. ```APIDOC ## Function: `pdf(urls, options)` ### Description Converts one or more web pages into a PDF file. It accepts an array of URLs or readable streams and an optional configuration object for PDF generation. ### Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | urls | `string[]` or `stream.Readable[]` | Yes | — | Array of URLs (http/https/file), local file paths, or stdin streams (pass `-` for stdin) | | options | `PDFOptions` | No | `{}` | Configuration object for PDF generation | ### Options Object (PDFOptions) | Key | Type | Default | Description | |-----|------|---------|-------------| | output | `string` | Auto-generated filename | Output file path for the PDF; use `-` for stdout (not allowed for PDF) | | style | `string` | Default stylesheet | Path to custom CSS stylesheet | | css | `string` | `''` | Inline CSS to append to stylesheet | | template | `string` | Default template | Path to custom HTML template (Nunjucks format) | | title | `string` | First page title or "Untitled" | Bundle title, used in metadata and cover | | author | `string` | First page byline | Bundle author, used in metadata and cover | | url | `string[]` | — | Base URLs for resolving relative links when using stdin | | wait | `number` | 0 | Delay in seconds between processing multiple URLs (0 = parallel) | | individual | `boolean` | false | Export each URL as a separate PDF file | | cover | `boolean` | auto | Generate a cover page (auto: enabled if multiple items or title provided) | | toc | `boolean` | auto | Generate a table of contents (auto: enabled if multiple items) | | toc-level | `number` | 1 | Heading depth for TOC (1–6); implies `--toc` if > 1 | | hyphenate | `boolean` | true | Enable hyphenation and text justification | | inline | `boolean` | false | Embed images as base64 data URLs | | amp | `boolean` | true | Use AMP version of pages if available | | browser | `string` | 'chrome' | Browser engine: 'chrome' or 'firefox' | | sandbox | `boolean` | true | Enable Puppeteer sandbox (set false to disable for restricted environments) | | debug | `boolean` | false | Print verbose debug information | | unsafe | `boolean` | false | Disable JSDOM HTML validation for invalid markup | | fetch | `object` | `{}` | Custom fetch options (headers, timeout, etc.) | ### Returns Promise resolving to an object containing: - `items`: array of `ParsedItem` objects (see types.md) - `options`: the final options object used - Returns `null` if no valid URLs were provided ### Throws - When PDF rendering fails (Puppeteer error) - When file I/O fails - When URL fetch fails (unless caught and ignored during cleanup) ### Example ```javascript import { configure, pdf } from 'percollate'; configure(); // Basic usage await pdf(['https://example.com'], { output: 'example.pdf' }); // Multiple pages with title await pdf( ['https://example.com/1', 'https://example.com/2'], { output: 'bundle.pdf', title: 'My Collection', author: 'Jane Doe' } ); // With custom CSS await pdf(['https://example.com'], { output: 'custom.pdf', css: '@page { size: A3 landscape; margin: 0 }' }); // Read from stdin with explicit URL import { createReadStream } from 'fs'; const stream = createReadStream('article.html'); await pdf([stream], { output: 'stdin.pdf', url: 'https://example.com' }); ``` ``` -------------------------------- ### Use Custom HTML Template Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Specify a custom Nunjucks HTML template using the --template flag. This allows for complete control over the output structure. Available context variables include filetype, title, author, date, items, style, and options. ```bash percollate pdf --template=./my-template.html https://example.com ``` -------------------------------- ### Set Bundle Author Source: https://github.com/danburzo/percollate/blob/main/README.md Generate a PDF file from a URL and specify the author for the document. ```bash percollate pdf --author="Ella Example" http://example.com ``` -------------------------------- ### Process HTML from Standard Input for PDF Conversion Source: https://github.com/danburzo/percollate/blob/main/README.md Reads HTML from stdin and converts it to PDF, specifying the source URL with --url for correct relative link resolution. ```bash curl https://example.com/page1 | percollate pdf --url=https://example.com/page1 - ``` -------------------------------- ### cliopts(args) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-other-modules.md Parses command-line arguments into command, options, and operands. It supports various flag formats and value casting. ```APIDOC ## Function: cliopts(args) ### Description Parses command-line arguments into command, options, and operands. It supports various flag formats and value casting. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **args** (`string[]`) - Required - Array of CLI arguments (usually from `process.argv.slice(2)`) ### Returns Object with: - `command`: First recognized command ('pdf', 'epub', 'html', 'md') or `undefined` - `opts`: Parsed options object with camelCase keys - `operands`: Non-option arguments (URLs, file paths, etc.) ### Option Formats - Short flags: `-o path` (single letter) - Long flags: `--output=path` or `--output path` - Negation: `--no-flag` sets flag to `false` - Multi-value: `--url=url1 --url=url2` creates array ### Value Casting - `'true'` → `true` - `'false'` → `false` - Numeric strings → numbers - Other strings → strings as-is ### Example ```javascript import { cliopts } from './src/cli-opts.js'; const args = ['pdf', '--output=my.pdf', '--title=My Page', 'https://example.com']; const result = cliopts(args); // { // command: 'pdf', // opts: {output: 'my.pdf', title: 'My Page'}, // operands: ['https://example.com'] // } ``` ``` -------------------------------- ### Content Fetching Strategies Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md Illustrates the different methods used by Percollate to fetch content based on the input source type. ```javascript URL (http/https) ↓ fetch(url) → response.arrayBuffer() File path ↓ fs.readFile(path) → Buffer File URL (file://) ↓ fs.readFile(decoded_path) → Buffer Stream (stdin) ↓ slurp(stream) → Buffer ``` -------------------------------- ### Export Each URL as a Separate File Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Enable the --individual flag to export each processed URL into its own output file. This applies to all commands. ```bash percollate pdf --individual url1 url2 url3 # Produces 3 PDFs ``` -------------------------------- ### Transform Multiple Web Pages into Individual PDFs Source: https://github.com/danburzo/percollate/blob/main/README.md Converts several web pages into separate PDF files using the --individual flag. ```bash percollate pdf --individual https://example.com/page1 https://example.com/page2 ``` -------------------------------- ### Cleanup Item with Options Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-cleanup-item.md Demonstrates how to use the `cleanupItem` function with various options for processing a JSDOM instance. It shows the expected output for title, content, and table of contents. ```javascript import { JSDOM } from 'jsdom'; import cleanupItem from './src/cleanup-item.js'; const html = '

Title

Content

'; const dom = new JSDOM(html, { url: 'https://example.com' }); const item = await cleanupItem(dom, { mapRemoteResources: false, inline: false, xhtml: false, hyphenate: true, 'toc-level': 2, debug: false }, null, { err: process.stderr, out: process.stdout, UA: 'percollate/4.3.0' }); console.log(item.title); console.log(item.content); console.log(item.toc); ``` -------------------------------- ### Generate Cover Page Source: https://github.com/danburzo/percollate/blob/main/_autodocs/configuration.md Enables the generation of a cover page for PDF and EPUB outputs. This is often auto-enabled when bundling multiple items or using the `--title` option. ```bash percollate pdf --cover --title="My Book" https://example.com ``` -------------------------------- ### Enable Debugging for HTML Output Source: https://github.com/danburzo/percollate/blob/main/CONTRIBUTING.md Use the --debug flag to save the temporary HTML file for inspection. ```bash percollate --debug ... ``` -------------------------------- ### Main Module Public Exports Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Exports the core functions for configuring and processing content in various formats. ```javascript export { configure, pdf, epub, html, md }; ``` -------------------------------- ### Exporting Available and Default Markdown Options Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Exports the set of available markdown options and the default configuration for markdown processing. ```javascript export { AVAILABLE_MARKDOWN_OPTIONS, DEFAULT_MARKDOWN_OPTIONS }; ``` -------------------------------- ### Validate EPUB with epubcheck Source: https://github.com/danburzo/percollate/blob/main/CONTRIBUTING.md Use the epubcheck tool to validate the generated EPUB file. ```bash npx epubcheck ./book.epub ``` -------------------------------- ### Run Automated Tests Source: https://github.com/danburzo/percollate/blob/main/CONTRIBUTING.md Execute the automated test suite to ensure functionality. ```bash npm run test ``` -------------------------------- ### Percollate Options Object Source: https://github.com/danburzo/percollate/blob/main/_autodocs/architecture.md Illustrates the structure of the options object used for configuration during a run. It includes various settings for output, styling, and internal caching mechanisms like a slug cache to prevent filename collisions. ```javascript { output?: string, style?: string, css?: string, // ... 20+ more options slugCache: {} // Tracks generated filenames to prevent collisions } ``` -------------------------------- ### Importing File Mimetype Utilities Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Imports utility functions for looking up file MIME types and checking if a URL points to an image. ```javascript import { lookupMimetype, isImageURL } from './src/util/file-mimetype.js'; ``` -------------------------------- ### fetchBase64 Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Fetches a resource from a URL and returns its content as a Base64 encoded string. ```APIDOC ## fetchBase64 ### Description Fetches a resource from a URL and returns its content as a Base64 encoded string. ### Parameters - **url** (string) - The URL of the resource to fetch. - **fetchOptions** (object) - Optional. Options to pass to the fetch request. ### Returns - **Promise** - A promise that resolves with the Base64 encoded content. ``` -------------------------------- ### remoteResources Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-other-modules.md Scans a document for external image sources (img src, picture source, srcset) and maps them to local filenames for bundling with EPUB files. It generates unique filenames, replaces URLs with relative paths, and returns a list of resources to be fetched. ```APIDOC ## Function: remoteResources ### Description Scans a document for external image sources (img src, picture source, srcset) and maps them to local filenames for bundling with EPUB files. It generates unique filenames, replaces URLs with relative paths, and returns a list of resources to be fetched. ### Parameters - **doc** (`Document`) - Required - JSDOM document with `` and `` elements ### Returns - `RemoteResource[]` - Array of `RemoteResource` objects: ```javascript { original: string, // Original URL mapped: string, // Generated local filename (e.g., 'rr-12345678.jpg') origin: string, // Document origin (for referrer) mimetype: string // Detected MIME type (e.g., 'image/jpeg') } ``` ### Example ```javascript import remoteResources from './src/remote-resources.js'; import { JSDOM } from 'jsdom'; const dom = new JSDOM(`
`, { url: 'https://example.com' }); const resources = remoteResources(dom.window.document); // Returns: [ // {original: 'https://example.com/photo.jpg', mapped: 'rr-uuid1.jpg', origin: 'https://example.com', mimetype: 'image/jpeg'}, // {original: 'https://example.com/large.webp', mapped: 'rr-uuid2.webp', origin: 'https://example.com', mimetype: 'image/webp'}, // ... // ] ``` ### Source `src/remote-resources.js:1–60` ``` -------------------------------- ### Convert Web Pages to PDF with Custom CSS Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-main.md Generates a PDF from web pages using a custom CSS stylesheet for formatting. The configure() function must be called first. ```javascript import { configure, pdf } from 'percollate'; configure(); // With custom CSS await pdf(['https://example.com'], { output: 'custom.pdf', css: '@page { size: A3 landscape; margin: 0 }' }); ``` -------------------------------- ### imagesAtFullSize(doc) Source: https://github.com/danburzo/percollate/blob/main/_autodocs/api-enhancements.md Extracts full-size image URLs from thumbnail links, applies URL mapping corrections, and cleans up image attributes and query parameters. ```APIDOC ## imagesAtFullSize(doc) ### Description Extracts full-size image URLs from thumbnail links. ### Method `imagesAtFullSize(doc: Document): void` ### Parameters #### Path Parameters - **doc** (`Document`) - JSDOM window.document object ### Behavior Transforms patterns like: ```html ``` To: ```html ``` ### Exclusions The following URL patterns are excluded from image extraction: - Wikipedia image file pages (`wikipedia.org/wiki/[lang]:`) - GitHub-hosted images (may be embedded Markdown) ### URL Mapping Special URL patterns are corrected: - **Blogger images**: `/s1600-h/` → `/s1600/` (removes redirect wrapper) ### Additional Processing - Removes `width` and `height` attributes from all images - Strips query parameters from image URLs (e.g., `?w=1024`) ### Example ```javascript doc.body.innerHTML = ' '; imagesAtFullSize(doc); // Result: ``` ``` -------------------------------- ### Main Module Exports Source: https://github.com/danburzo/percollate/blob/main/_autodocs/exports.md Exports from the main module (`index.js`) include functions for configuring the library and processing content into various formats (PDF, EPUB, HTML, Markdown). ```APIDOC ## Main Module Exports ### Public Exports #### `configure()` - **Type**: function - **Parameters**: `()` - **Returns**: `void` #### `pdf(urls: string[], options?: object)` - **Type**: async function - **Parameters**: `urls` (string[]), `options` (object, optional) - **Returns**: `Promise<{items, options} | null>` #### `epub(urls: string[], options?: object)` - **Type**: async function - **Parameters**: `urls` (string[]), `options` (object, optional) - **Returns**: `Promise<{items, options} | null>` #### `html(urls: string[], options?: object)` - **Type**: async function - **Parameters**: `urls` (string[]), `options` (object, optional) - **Returns**: `Promise<{items, options} | null>` #### `md(urls: string[], options?: object)` - **Type**: async function - **Parameters**: `urls` (string[]), `options` (object, optional) - **Returns**: `Promise<{items, options} | null>` ### Test Exports #### `__test__.fetchContent(ref: string | Readable, fetchOptions?: object)` - **Type**: async function - **Parameters**: `ref` (string or Readable), `fetchOptions` (object, optional) - **Returns**: `Promise<{buffer, contentType}>` ```