element if available // Reduces processing area ``` -------------------------------- ### Parallelize data collection with Makefile Source: https://github.com/chromium/dom-distiller/blob/main/heuristics/distillable/README.md Uses a Makefile to run multiple instances of the screenshot script concurrently for faster processing. ```make ALL=$(addsuffix .target,$(shell seq 1000)) all: $(ALL) %.target : xvfb-run -a -s "-screen 0 1600x5000x24" ./get_screenshots.py --out out_dir --urls-file urls.txt --resume ``` -------------------------------- ### DomDistiller.applyWithOptions(DomDistillerOptions options) Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/README.md Extracts content from the current page using a custom DomDistillerOptions object to control extraction behavior. ```APIDOC ## DomDistiller.applyWithOptions(DomDistillerOptions options) ### Description Performs content extraction with specific user-defined configurations. ### Parameters - **options** (DomDistillerOptions) - Required - Configuration object created via DomDistillerOptions.create(). ### Returns - **DomDistillerResult** - An object containing the distilled HTML, metadata, and statistics. ``` -------------------------------- ### Run screenshot generation Source: https://github.com/chromium/dom-distiller/blob/main/heuristics/distillable/README.md Executes the screenshot and feature extraction script for a list of URLs. ```bash ./get_screenshots.py --out out_dir --urls-file urls.txt ``` -------------------------------- ### Extract content with default options Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/ContentExtractor.md Retrieves the main article content as an HTML string. Returns an empty string if no content is identified. ```java public String extractContent() ``` ```java String html = extractor.extractContent(); // Render in WebView or insert into DOM ``` -------------------------------- ### DomDistiller.apply() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomDistiller.md Extracts content from the current document using default extraction options. ```APIDOC ## DomDistiller.apply() ### Description Extracts content from the current document using default options. Returns a result containing title, distilled content, pagination info, and metadata. ### Signature `public static DomDistillerProtos.DomDistillerResult apply()` ### Returns `DomDistillerProtos.DomDistillerResult` - The extraction result. ### Example ```java DomDistillerResult result = DomDistiller.apply(); String title = result.getTitle(); ``` ``` -------------------------------- ### Initialize DomWalker with a Visitor Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomWalker.md Create a new DomWalker instance by implementing the DomWalker.Visitor interface to handle node traversal events. ```java DomWalker.Visitor visitor = new DomWalker.Visitor() { @Override public boolean visit(Node n) { // Process node entry return true; // Continue to children } @Override public void exit(Node n) { // Process after children } @Override public void skip(Element e) { // Node was skipped } }; DomWalker walker = new DomWalker(visitor); ``` -------------------------------- ### Retrieve and iterate over document elements Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/WebDocument.md Fetches all elements in document order and demonstrates type checking for specific element types. ```java List elements = document.getElements(); for (WebElement elem : elements) { if (elem instanceof WebText) { WebText text = (WebText) elem; String content = text.getText(); } else if (elem instanceof WebImage) { WebImage img = (WebImage) elem; String url = img.getUrl(); } } ``` -------------------------------- ### ContentExtractor(Element root) Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/ContentExtractor.md Initializes a new ContentExtractor instance with a specified DOM root element. ```APIDOC ## Constructor ### Description Initializes a new ContentExtractor instance with a specified DOM root element for content extraction. ### Signature `public ContentExtractor(Element root)` ### Parameters - **root** (com.google.gwt.dom.client.Element) - Required - Document root element from which content will be extracted. ### Example ```java Element documentRoot = Document.get().getDocumentElement(); ContentExtractor extractor = new ContentExtractor(documentRoot); ``` ``` -------------------------------- ### Implement visit(Node) logic Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomWalker.md Use visit to process nodes and control traversal flow. Returning false skips the node's children and prevents the exit callback. ```java @Override public boolean visit(Node n) { if (n.getNodeType() == Node.ELEMENT_NODE) { Element elem = Element.as(n); // Process element if (elem.getTagName().equals("SCRIPT")) { return false; // Skip script contents } } return true; } ``` -------------------------------- ### Handle Pagination Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/README.md Attempts to detect the next page URL using different pagination algorithms. ```java // Try next-link detection first DomDistillerOptions options = DomDistillerOptions.create(); options.setPaginationAlgo("next"); options.setOriginalUrl(Document.get().getURL()); DomDistillerResult result = DomDistiller.applyWithOptions(options); String nextUrl = result.getPaginationInfo().getNextPage(); if (nextUrl.isEmpty()) { // No next link found - try page number detection options.setPaginationAlgo("pagenum"); result = DomDistiller.applyWithOptions(options); nextUrl = result.getPaginationInfo().getNextPage(); } if (!nextUrl.isEmpty()) { System.out.println("Next page: " + nextUrl); } else { System.out.println("Single-page article"); } ``` -------------------------------- ### GWT Compilation Pipeline Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/ARCHITECTURE.md Visual representation of the transformation process from Java source files to standalone JavaScript. ```text java/org/chromium/distiller/*.java ↓ [GWT Compiler] ↓ war/domdistiller/domdistiller.nocache.js ↓ [Extract JS extraction] ↓ out/domdistiller.js (standalone JavaScript) ``` -------------------------------- ### Using DomUtil Static Methods Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomUtil.md Call static methods directly from the DomUtil class for common DOM and URL utility tasks. ```java DomUtil.hasClassName(element, "content"); DomUtil.splitUrlParams("page=1&sort=date"); DomUtil.hasRootDomain(urlString, "example.com"); ``` -------------------------------- ### addTimingInfo(double startTime, TimingInfo timingInfo, String name) Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/LogUtil.md Records the elapsed time for a specific processing stage into a TimingInfo object. ```APIDOC ## addTimingInfo(double startTime, TimingInfo timingInfo, String name) ### Description Records timing for a processing stage. Updates the provided timingInfo object with the elapsed time. ### Parameters - **startTime** (double) - Required - Start time from DomUtil.getTime() - **timingInfo** (TimingInfo) - Required - Timing info object to record into - **name** (String) - Required - Name of the stage being timed ### Returns None. ``` -------------------------------- ### Recognized Query String Pagination Patterns Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/PageParameterParser.md Common query string parameter formats used to identify page numbers. ```text ?page=2 ?p=2 ?pagenum=2 ?n=2 ?page_num=2 ?start=20 (with per-page size inference) ``` -------------------------------- ### MarkupParser(Element root) Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/MarkupParser.md Constructor for the MarkupParser class. Initializes and runs all metadata parsers on the provided document root element. ```APIDOC ## Constructor ### Description Initializes a new MarkupParser instance and executes metadata extraction using OpenGraph, SchemaOrg, and IEReadingView parsers. ### Signature `public MarkupParser(Element root)` ### Parameters - **root** (com.google.gwt.dom.client.Element) - Required - The document root element to parse for metadata. ### Example ```java Element documentRoot = Document.get().getDocumentElement(); MarkupParser parser = new MarkupParser(documentRoot); ``` ``` -------------------------------- ### Configure extraction with custom options Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/README.md Customize the extraction process by setting text-only modes, debug levels, URLs, and pagination algorithms. ```java // Create options DomDistillerOptions options = DomDistillerOptions.create(); // Text-only extraction (faster, smaller output) options.setExtractTextOnly(true); // Enable debug logging options.setDebugLevel(LogUtil.DEBUG_LEVEL_BOILER_PIPE_PHASES); // Provide original URL (helps with pagination) options.setOriginalUrl(Document.get().getURL()); // Choose pagination method options.setPaginationAlgo("next"); // or "pagenum" // Extract with options DomDistillerResult result = DomDistiller.applyWithOptions(options); ``` -------------------------------- ### Sanity Check Derived Features Source: https://github.com/chromium/dom-distiller/blob/main/heuristics/distillable/README.md Compare JavaScript and native feature implementations to ensure consistency. ```bash ./check_derived_features.py --features out_dir/feature-derived ``` ```bash ./check_derived_features.py --features out_dir/mfeature-derived --from-mhtml ``` -------------------------------- ### Retrieve page description with getDescription() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/MarkupParser.md Extracts the page description from metadata. Returns an empty string if unavailable. ```java public String getDescription() ``` ```java String description = parser.getDescription(); // Use in search results, social sharing, etc. ``` -------------------------------- ### Pagination Detection Algorithms Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/ARCHITECTURE.md Describes the logic for identifying pagination links based on text patterns or URL parameters. ```text Option: pagination_algo="next" PagingLinksFinder.getPaginationInfo(originalUrl) ├─ Find all anchors in document ├─ Score by text pattern matching (next/prev) ├─ Score by URL pattern (folder structure, query params) ├─ Score by element attributes (class/id patterns) └─ Return PaginationInfo with next/prev URLs Option: pagination_algo="pagenum" PageParameterParser.parse(originalUrl, timingInfo) ├─ Collect all numeric anchors ├─ Extract page parameter patterns from original URL ├─ Group consecutive page numbers ├─ Score by URL consistency and pattern match └─ Return PageParamInfo with next page URL ``` -------------------------------- ### Generate Training CSV Source: https://github.com/chromium/dom-distiller/blob/main/heuristics/distillable/README.md Combine derived features with labels to produce training data files. ```bash ./write_features_csv.py --marked $(ls -rt out_dir/archive/*|tail -n1) --features out_dir/feature-derived --out labelled ``` ```bash ./write_features_csv.py --distilled out_dir/dfeature-derived --features out_dir/feature-derived --out labelled ``` -------------------------------- ### Enable debug logging for pagination Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/PageParameterParser.md Configures the logging level to output detailed information about page detection and scoring. ```java LogUtil.setDebugLevel(LogUtil.DEBUG_LEVEL_PAGING_INFO); PageParamInfo info = PageParameterParser.parse(currentUrl, timingInfo); // Debug logs show: // - Detected page numbers and URLs // - Grouping and scoring results // - Final next page selection ``` -------------------------------- ### Define PaginationInfo message Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/types.md Navigation URLs for multi-page articles, including next, previous, and canonical page links. ```protobuf message PaginationInfo { optional string next_page = 1; optional string prev_page = 2; optional string canonical_page = 3; } ``` -------------------------------- ### Define DomDistillerOptions message Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/types.md Configuration options for content extraction, including text-only mode and pagination algorithms. ```protobuf message DomDistillerOptions { optional bool extract_text_only = 1; optional int32 debug_level = 2; optional string original_url = 3; optional string pagination_algo = 4; } ``` -------------------------------- ### getTimingInfo() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/ContentExtractor.md Retrieves detailed performance metrics from the extraction run. ```APIDOC ## getTimingInfo() ### Description Retrieves detailed performance metrics from the extraction run. ### Method public TimingInfo getTimingInfo() ### Returns - **DomDistillerProtos.TimingInfo** - Timing data including markup_parsing_time, document_construction_time, article_processing_time, formatting_time, total_time, and other_times. ### Example ```java TimingInfo timing = extractor.getTimingInfo(); System.out.println("Total time: " + timing.getTotalTime() + "ms"); ``` ``` -------------------------------- ### Extract Metadata with MarkupParser Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/ContentExtractor.md Demonstrates retrieving the parser from an extractor instance and accessing document metadata fields. ```java MarkupParser parser = extractor.getMarkupParser(); String author = parser.getAuthor(); String description = parser.getDescription(); ``` -------------------------------- ### Manage Content Flags in WebDocument Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/WebDocument.md Use setIsContent to mark elements as article content or boilerplate, and getIsContent to retrieve the current status. ```java element.setIsContent(true); // Mark as article content element.setIsContent(false); // Mark as boilerplate boolean isContent = element.getIsContent(); ``` -------------------------------- ### Compare Distilled Content Source: https://github.com/chromium/dom-distiller/blob/main/heuristics/distillable/README.md Verify that distilled content from original pages matches MHTML archives. ```bash ./check_distilled_mhtml.py --dir out_dir ``` -------------------------------- ### Advanced Test Execution Source: https://github.com/chromium/dom-distiller/blob/main/README.md Options for running tests without windows, filtering by name, or using the swarming tool. ```bash xvfb-run out/Debug/components_browsertests ``` ```bash out/Debug/components_browsertests --gtest_filter=\*Distiller\* ``` ```bash autoninja -C out/Debug components_browsertests_run python tools/swarming_client/isolate.py run -s out/Debug/components_browsertests.isolated ``` -------------------------------- ### Retrieve copyright info with getCopyright() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/MarkupParser.md Extracts the copyright notice or holder. Returns an empty string if not specified. ```java public String getCopyright() ``` ```java String copyright = parser.getCopyright(); ``` -------------------------------- ### Configure Word Counter and Retrieve Statistics Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/configuration.md Sets the word counter based on document content before applying the distiller and retrieving the final word count. ```java StringUtil.setWordCounter( DomUtil.javascriptTextContent(Document.get().getDocumentElement())); DomDistillerResult result = DomDistiller.apply(); int wordCount = result.getStatisticsInfo().getWordCount(); ``` -------------------------------- ### Parse pagination information in Java Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/PageParameterParser.md Defines the method signature for analyzing a URL for pagination parameters. ```java public static PageParamInfo parse(String originalUrl, TimingInfo timingInfo) ``` -------------------------------- ### Access Timing Metrics in Java Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/configuration.md Retrieves and prints timing information for various stages of the extraction process. ```java TimingInfo timing = result.getTimingInfo(); System.out.println("Markup parsing: " + timing.getMarkupParsingTime() + "ms"); System.out.println("Document construction: " + timing.getDocumentConstructionTime() + "ms"); System.out.println("Article processing: " + timing.getArticleProcessingTime() + "ms"); System.out.println("Formatting: " + timing.getFormattingTime() + "ms"); System.out.println("Total: " + timing.getTotalTime() + "ms"); // Sub-stage details for (int i = 0; i < timing.getOtherTimesCount(); i++) { TimingEntry entry = timing.getOtherTimes(i); System.out.println(entry.getName() + ": " + entry.getTime() + "ms"); } ``` -------------------------------- ### Handle missing pagination in Java Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/errors.md Check for empty pagination info to identify single-page articles and disable navigation buttons. ```java PaginationInfo pagination = result.getPaginationInfo(); if (pagination.getNextPage().isEmpty()) { // Single page article - no pagination disableNextPageButton(); } ``` -------------------------------- ### Extract content with default options Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/README.md Perform a standard extraction to retrieve the title, distilled HTML content, and text direction. ```java // Get the extraction result DomDistillerResult result = DomDistiller.apply(); // Access the results String title = result.getTitle(); String content = result.getDistilledContent().getHtml(); String textDirection = result.getTextDirection(); ``` -------------------------------- ### Handle empty content extraction in Java Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/errors.md Check for empty distilled content to implement fallback logic when no article content is identified. ```java DomDistillerResult result = DomDistiller.apply(); if (result.getDistilledContent().getHtml().isEmpty()) { System.out.println("No content extracted"); // Use original content instead } ``` -------------------------------- ### Configure original_url for pagination detection Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/configuration.md Sets the original URL to enable anchor-based next-link detection and numeric page parameter extraction. ```java // For next-link detection // Finds anchors on same domain as original_url // For page number detection // Extracts parameter patterns from original_url // Matches pagination URLs against extracted patterns options.setOriginalUrl("https://example.com/news/article/page/1"); ``` -------------------------------- ### setSuppressConsoleOutput(boolean suppress) Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/LogUtil.md Configures whether logs are printed to the console, without affecting log accumulation. ```APIDOC ## setSuppressConsoleOutput(boolean suppress) ### Description Control whether logs are printed to console (only affects output, not accumulation). ### Parameters - **suppress** (boolean) - Required - True to suppress console output; logs still accumulated ### Returns None. ``` -------------------------------- ### Configure DomDistiller Debugging Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/LogUtil.md Sets the debug level in DomDistillerOptions and retrieves debug logs from the resulting DomDistillerResult. ```java DomDistillerOptions options = DomDistillerOptions.create(); options.setDebugLevel(LogUtil.DEBUG_LEVEL_TIMING_INFO); DomDistillerResult result = DomDistiller.applyWithOptions(options); // Access debug information if (result.hasDebugInfo()) { String debugLog = result.getDebugInfo().getLog(); // Process debug output } ``` -------------------------------- ### Run DOM Distiller Tests Source: https://github.com/chromium/dom-distiller/blob/main/README.md Executes all available tests within the browser console. ```javascript org.chromium.distiller.JsTestEntry.run() ``` -------------------------------- ### Record timing information with addTimingInfo Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/LogUtil.md Use this method to record the duration of a processing stage into a TimingInfo object. ```java double startTime = DomUtil.getTime(); // ... perform operation ... LogUtil.addTimingInfo(startTime, mTimingInfo, "OpenGraphProtocolParser"); ``` -------------------------------- ### Output colored status messages in Java Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/LogUtil.md Use predefined color constants to format console output for better readability. ```java LogUtil.logToConsole(LogUtil.kGreen + "✓ Content identified" + LogUtil.kReset); LogUtil.logToConsole(LogUtil.kBlue + "» Starting filter chain" + LogUtil.kReset); LogUtil.logToConsole(LogUtil.kYellow + "⚠ Unusual document structure" + LogUtil.kReset); ``` -------------------------------- ### Convert to TextDocument with createTextDocumentView() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/WebDocument.md Creates a simplified text-based representation of the document for boilerpipe processing, grouping elements by logical sections. ```java public TextDocument createTextDocumentView() ``` ```java TextDocument textDoc = webDocument.createTextDocumentView(); ArticleExtractor.INSTANCE.process(textDoc, candidateTitles); ``` -------------------------------- ### Manual Extraction Test Execution Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/ARCHITECTURE.md Command to run manual extraction tests on a specific URL using the browser test suite. ```bash # Run extraction on live URL xvfb-run out/Debug/components_browsertests \ --gtest_filter='*MANUAL_ExtractUrl' \ --run-manual \ --url=http://example.com ``` -------------------------------- ### logToConsole(String str) Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/LogUtil.md Logs a specified string message to the JavaScript console and adds it to the internal log builder. ```APIDOC ## logToConsole(String str) ### Description Logs a message to the JavaScript console. The message is also accumulated in the internal log builder. ### Method static void ### Parameters - **str** (String) - Required - The message to log. If null, it is treated as an empty string. ### Returns None. ### Example ```java LogUtil.logToConsole("Starting content extraction"); LogUtil.logToConsole(LogUtil.kGreen + "SUCCESS: Content extracted" + LogUtil.kReset); ``` ``` -------------------------------- ### DomDistillerOptions Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/README.md Configuration options for the DOM Distiller extraction process. ```APIDOC ## DomDistillerOptions ### Description Configuration object used to customize the behavior of the DOM Distiller. ### Parameters - **extract_text_only** (boolean) - Optional - Extract plain text only (faster). Default: false. - **debug_level** (int) - Optional - Debug verbosity (0-4). Default: 0. - **original_url** (string) - Optional - Page URL for pagination heuristics. Default: Document.URL. - **pagination_algo** (string) - Optional - Algorithm: "next" or "pagenum". Default: "next". ``` -------------------------------- ### TimingInfo Proto Message Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/types.md Defines the structure for tracking performance metrics across various extraction stages. ```protobuf message TimingInfo { optional double markup_parsing_time = 1; optional double document_construction_time = 2; optional double article_processing_time = 3; optional double formatting_time = 4; optional double total_time = 5; repeated TimingEntry other_times = 6; } ``` -------------------------------- ### Integrate WebDocument with Boilerpipe Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/WebDocument.md Convert a WebDocument to a text view, apply boilerpipe filters, and generate output based on classification. ```java // Convert to text view TextDocument textDoc = webDocument.createTextDocumentView(); // Apply boilerpipe filters ArticleExtractor.INSTANCE.process(textDoc, candidateTitles); // Update content flags based on boilerpipe classification // Then generate output String html = webDocument.generateOutput(false); ``` -------------------------------- ### createTextDocumentView() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/WebDocument.md Converts the document into a simplified text-based representation for boilerpipe processing. ```APIDOC ## createTextDocumentView() ### Description Convert document to text representation for boilerpipe processing. Groups text elements by their groupNumber. ### Signature `public TextDocument createTextDocumentView()` ### Returns - **TextDocument** - Simplified text-based view of the document with text blocks grouped by logical sections. ### Example ```java TextDocument textDoc = webDocument.createTextDocumentView(); ArticleExtractor.INSTANCE.process(textDoc, candidateTitles); ``` ``` -------------------------------- ### Retrieve Image URLs in Java Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/ContentExtractor.md Fetches a list of image URLs from the distilled content, returning an empty list if no images are present. ```java public List getImageUrls() ``` ```java List imageUrls = extractor.getImageUrls(); for (String url : imageUrls) { // Preload or cache images System.out.println("Image: " + url); } ``` -------------------------------- ### DomWalker(DomWalker.Visitor visitor) Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomWalker.md Constructor to initialize a new DomWalker instance with a visitor implementation. ```APIDOC ## Constructor: DomWalker(DomWalker.Visitor visitor) ### Description Initializes a new DomWalker instance configured with a visitor that receives callbacks during tree traversal. ### Parameters - **visitor** (DomWalker.Visitor) - Required - Visitor implementation receiving callbacks during tree traversal. ### Example ```java DomWalker.Visitor visitor = new DomWalker.Visitor() { @Override public boolean visit(Node n) { return true; } @Override public void exit(Node n) {} @Override public void skip(Element e) {} }; DomWalker walker = new DomWalker(visitor); ``` ``` -------------------------------- ### Implement exit(Node) logic Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomWalker.md Use exit for post-order processing or cleanup. This method is only triggered if the corresponding visit call returned true. ```java @Override public void exit(Node n) { if (n.getNodeType() == Node.ELEMENT_NODE) { Element elem = Element.as(n); // Post-process element after children finalizElement(elem); } } ``` -------------------------------- ### Enable and Retrieve Pagination Debug Logs Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/PagingLinksFinder.md Set the debug level in extraction options to capture paging information, then access the log from the resulting object. ```java // In DomDistiller extraction options options.setDebugLevel(LogUtil.DEBUG_LEVEL_PAGING_INFO); // After extraction DomDistillerResult result = DomDistiller.applyWithOptions(options); String debugLog = result.getDebugInfo().getLog(); // Log contains detailed paging link evaluation ``` -------------------------------- ### Define the Visitor interface Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomWalker.md The Visitor interface defines the contract for handling node traversal events. ```java public interface Visitor { public boolean visit(Node n); public void exit(Node n); public void skip(Element e); } ``` -------------------------------- ### Retrieve image metadata with getImages() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/MarkupParser.md Returns an array of image objects containing URL, type, dimensions, and caption information. The first element in the returned array represents the primary image. ```java public MarkupParser.Image[] getImages() ``` ```java MarkupParser.Image[] images = parser.getImages(); if (images.length > 0) { MarkupParser.Image primaryImage = images[0]; String imageUrl = primaryImage.url; int width = primaryImage.width; int height = primaryImage.height; } ``` -------------------------------- ### getElements() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/WebDocument.md Retrieves all elements currently in the document in their document order. ```APIDOC ## getElements() ### Description Get all elements in the document. ### Method public List getElements() ### Returns - **List** - List of all elements in document order. Includes text blocks, tables, images, videos, embeds, and tags. ### Example ```java List elements = document.getElements(); for (WebElement elem : elements) { if (elem instanceof WebText) { WebText text = (WebText) elem; String content = text.getText(); } else if (elem instanceof WebImage) { WebImage img = (WebImage) elem; String url = img.getUrl(); } } ``` ``` -------------------------------- ### getDebugLevel() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/LogUtil.md Retrieves the current debug level setting for the application. ```APIDOC ## getDebugLevel() ### Description Get the current debug level setting. ### Signature `static int getDebugLevel()` ### Returns - **int** - Current debug level (0-4). ``` -------------------------------- ### Retrieve performance timing metrics Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/ContentExtractor.md Returns a TimingInfo object containing detailed performance breakdowns for various stages of the extraction process. ```java public TimingInfo getTimingInfo() ``` ```java TimingInfo timing = extractor.getTimingInfo(); System.out.println("Total time: " + timing.getTotalTime() + "ms"); System.out.println("Document construction: " + timing.getDocumentConstructionTime() + "ms"); System.out.println("Article processing: " + timing.getArticleProcessingTime() + "ms"); ``` -------------------------------- ### getDescription() Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/MarkupParser.md Extracts the page description from the page metadata. ```APIDOC ## getDescription() ### Description Extract the page description. ### Signature `public String getDescription()` ### Returns - **String** - Page description from metadata. Empty string if unavailable. ### Example ```java String description = parser.getDescription(); ``` ``` -------------------------------- ### Re-extract Features from MHTML Source: https://github.com/chromium/dom-distiller/blob/main/heuristics/distillable/README.md Run feature extraction using xvfb to process MHTML archives. ```bash xvfb-run -a -s "-screen 0 1600x5000x24" ./get_screenshots.py --out out_dir --urls-file urls.txt --load-mhtml --skip-distillation ``` -------------------------------- ### Visitor Interface Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/api-reference/DomWalker.md The Visitor interface defines the contract for handling DOM node traversal events. ```APIDOC ## Visitor Interface ### Description Interface for defining custom traversal logic during DOM tree walking. ### Methods - **visit(Node n)**: Called when reaching a node. Return true to process children, false to skip. - **exit(Node n)**: Called after a node's subtree is processed. Only invoked if visit() returned true. - **skip(Element e)**: Called when an element is skipped via a false return from visit(). ``` -------------------------------- ### MarkupArticle Proto Message Source: https://github.com/chromium/dom-distiller/blob/main/_autodocs/types.md Defines the structure for article-specific metadata. ```protobuf message MarkupArticle { optional string published_time = 1; optional string modified_time = 2; optional string expiration_time = 3; optional string section = 4; repeated string authors = 5; } ```

Heading