A Text
", markupever.HtmlOptions(profile=True)) # # Tokenizer profile, in nanoseconds # ``` -------------------------------- ### Parse HTML with httpx (traditional) Source: https://awolverp.github.io/markupever/more-examples Demonstrates parsing HTML content obtained from an httpx GET request using markupever. It creates an httpx Client, sends a GET request, and then parses the response content. ```python import markupever import httpx # Create a Client instance with httpx.Client() as client: # Send a GET request to google response = client.get("https://www.example.com/") # Parse the result using markupever dom = markupever.parse(response.content, markupever.HtmlOptions()) ``` -------------------------------- ### HTML Parsing Options Source: https://awolverp.github.io/markupever/parser Demonstrates the use of HtmlOptions for parsing HTML content. Key options include `profile` and `drop_doctype`. ```APIDOC ## HTML Parsing Options ### Description This section details the `HtmlOptions` available for parsing HTML content with MarkupEver. It covers options like `profile` for performance profiling and `drop_doctype` for controlling DOCTYPE handling. ### Method N/A (Configuration options for the `markupever.parse` function) ### Endpoint N/A ### Parameters #### Query Parameters - **`profile`** (boolean) - Optional - Keep a record of how long we spent in each state? Printed when `finish()` is called. default: False. - **`drop_doctype`** (boolean) - Optional - Should we drop the DOCTYPE (if any) from the tree? default: False. ### Request Example ```python import markupever # Example with profile=False (default) markupever.parse("A Text
", markupever.HtmlOptions(profile=False)) # Example with drop_doctype=True dom = markupever.parse("A Text
", markupever.HtmlOptions(drop_doctype=True)) dom.serialize() # Expected output: "A Text
" # Example with drop_doctype=False (default) dom = markupever.parse("A Text
", markupever.HtmlOptions(drop_doctype=False)) dom.serialize() # Expected output: "A Text
" ``` ### Response #### Success Response (200) N/A (This describes configuration options, not a direct API response) #### Response Example N/A ``` -------------------------------- ### Navigate and Query HTML DOM Source: https://awolverp.github.io/markupever/parser_q= Demonstrates how to navigate and query a parsed HTML Document Object Model (DOM) using MarkupEver. It shows how to get the root node, select elements using CSS selectors, access element attributes, and serialize elements back to HTML strings. ```python root = dom.root() # Get root node title = root.select_one("title") # Accepts CSS selectors print(title.name) # QualName(local="title", ns="http://www.w3.org/1999/xhtml", prefix=None) print(title.serialize()) # 'A Text
") p.finish() p.errors() # Expected output: ["Unexpected token TagToken(Tag { kind: StartTag, name: Atom(\'p\' type=inline), self_closing: false, attrs: [] }) in insertion mode Initial"] # Example with exact_errors=False (default) p = markupever.Parser(markupever.XmlOptions(exact_errors=False)) p.process("A Text
") p.finish() p.errors() # Expected output: ["Unexpected token"] # Example with profile=True markupever.parse("A Text
", markupever.XmlOptions(profile=True)) # Expected output includes tokenizer profile information # Example with profile=False (default) markupever.parse("A Text
", markupever.XmlOptions(profile=False)) ``` ### Response #### Success Response (200) N/A (This describes configuration options, not a direct API response) #### Response Example N/A ``` -------------------------------- ### Retrieve Parsing Errors from MarkupEver Parser Source: https://awolverp.github.io/markupever/parser Demonstrates how to use the `errors()` method to get a list of any parsing errors detected during the process. This is crucial for debugging malformed input. ```python import markupever parser = markupever.Parser(options=markupever.HtmlOptions()) parser.process("... content 1 ...") parser.process("... content 2 ...") parser.process("... content 3 ...") parser.finish() print(parser.errors()) # ['Unexpected token'] ``` -------------------------------- ### Instantiate and Use MarkupEver Parser for HTML Source: https://awolverp.github.io/markupever/parser_q= Demonstrates how to create a Parser instance with HTML options, process content in chunks, and finalize the parsing. This is useful for handling large inputs efficiently. ```python import markupever # Create Parser parser = markupever.Parser(options=markupever.HtmlOptions()) # Process contents parser.process("... content 1 ...") parser.process("... content 2 ...") parser.process("... content 3 ...") # Mark as finished parser.finish() ``` -------------------------------- ### Profile XML Parsing in MarkupEver Source: https://awolverp.github.io/markupever/parser_q= Illustrates how to profile XML parsing using `markupever.XmlOptions(profile=True)`. The output includes detailed timing information for tokenizer states after `finish()` is called. ```python import markupever markupever.parse("A Text
", markupever.XmlOptions(profile=True)) # # Tokenizer profile, in nanoseconds # # 93331 total in token sink # # 46121 total in tokenizer # 17651 38.3% Data # 13640 29.6% TagName # 11768 25.5% TagOpen # 3062 6.6% EndTagOpen ``` -------------------------------- ### Parse HTML with MarkupEver Source: https://awolverp.github.io/markupever/parser_q= Demonstrates parsing an HTML string using the `markupever.parse` function. The `HtmlOptions` can be used to configure parsing behavior, such as disabling profiling. ```python import markupever markupever.parse("A Text
", markupever.HtmlOptions(profile=False)) ``` -------------------------------- ### Parse HTML with httpx (streaming) Source: https://awolverp.github.io/markupever/more-examples Shows how to parse HTML content from an httpx streaming GET request using markupever. This method is recommended for larger responses as it processes content in chunks. ```python import markupever import httpx # Create a Client instance with httpx.Client() as client: # Stream a GET request to google with client.stream( "GET", "https://www.example.com/", ) as stream: # Parse the result using markupever parser = markupever.Parser(markupever.HtmlOptions()) for content in stream.iter_bytes(): parser.process(content) dom = parser.finish().into_dom() ``` -------------------------------- ### Parse HTML with requests Source: https://awolverp.github.io/markupever/more-examples Illustrates parsing HTML content fetched using the 'requests' library with markupever. It sends a GET request and then uses markupever to parse the response content. ```python import markupever import requests # Send a GET request to google response = requests.get("https://www.example.com/") # Parse the result using markupever dom = markupever.parse(response.content, markupever.HtmlOptions()) ``` -------------------------------- ### Extract all links from an HTML page Source: https://awolverp.github.io/markupever/parser A common task is to extract specific elements, such as all anchor tags with an 'href' attribute starting with 'https://'. This demonstrates using the select method with attribute selectors to filter and retrieve desired elements. ```python for tag in root.select("a[href^='https://']"): print(tag.attrs["href"]) ``` -------------------------------- ### Extract Links from HTML Source: https://awolverp.github.io/markupever/parser_q= Shows a common task of extracting all hyperlink URLs from an HTML document using MarkupEver. It iterates through all anchor tags (``) with an `href` attribute starting with 'https://' and prints the URLs. ```python for tag in root.select("a[href^='https://']"): print(tag.attrs["href"]) # https://www.example.com # https://www.wikipedia.org # https://www.bbc.com # https://www.microsoft.com ``` -------------------------------- ### Profile HTML Parsing Performance Source: https://awolverp.github.io/markupever/parser_q= Enables performance profiling during HTML parsing. The results, showing time spent in different states, are printed when the `finish()` method is called. ```python import markupever markupever.parse("A Text
", markupever.HtmlOptions(profile=True)) # Output will include: # Tokenizer profile, in nanoseconds # ``` -------------------------------- ### Parse HTML with full_document=True Source: https://awolverp.github.io/markupever/parser_q= Parses an HTML snippet as a full document, automatically adding ``, ``, and `` tags if they are missing. This is the default behavior. ```python import markupever dom = markupever.parse("A Text
", markupever.HtmlOptions(full_document=True)) print(dom.serialize()) #A Text
``` -------------------------------- ### Navigate and Query XML DOM Source: https://awolverp.github.io/markupever/parser_q= Demonstrates navigating and querying a parsed XML Document Object Model (DOM) using MarkupEver. It shows how to select elements, including those within specific namespaces, using the `select()` method. ```python root = dom.root() # Get root node print(root.select_one("bookstore")) # Element(name=QualName(local="bookstore"), attrs=[], template=false, mathml_annotation_xml_integration_point=false) for i in root.select("mag|*"): # get all elements which has namespace 'mag' print(i) # Element(name=QualName(local="magazine", ns="http://www.example.com/magazines", prefix=Some("mag")), attrs=[], template=false, mathml_annotation_xml_integration_point=false) ``` -------------------------------- ### Access Parser Information (lineno, quirks_mode, errors) Source: https://awolverp.github.io/markupever/parser_q= Shows how to retrieve parsing details such as the line count, quirks mode status, and any detected errors after parsing is complete but before converting to a DOM. ```python import markupever parser = markupever.Parser(options=markupever.HtmlOptions()) parser.process("... content 1 ...") parser.process("... content 2 ...") parser.process("... content 3 ...") parser.finish() # Use `.errors()`, `.lineno`, or `.quirks_mode` if you want print(parser.lineno) # 56 print(parser.quirks_mode) # 2 print(parser.errors()) # ['Unexpected token'] dom = parser.into_dom() ``` -------------------------------- ### Parse HTML with full_document=False Source: https://awolverp.github.io/markupever/parser_q= Parses an HTML snippet without enforcing a full document structure. This option allows for more direct serialization of the provided content. ```python import markupever dom = markupever.parse("A Text
", markupever.HtmlOptions(full_document=False)) print(dom.serialize()) #A Text
``` -------------------------------- ### Convert Parser to TreeDom Source: https://awolverp.github.io/markupever/parser_q= Illustrates the final step of converting the parsed content into a `TreeDom` object using the `into_dom()` method, which also releases the parser's memory. ```python import markupever parser = markupever.Parser(options=markupever.HtmlOptions()) parser.process("... content 1 ...") parser.process("... content 2 ...") parser.process("... content 3 ...") parser.finish() dom = parser.into_dom() ``` -------------------------------- ### Parse HTML with exact_errors=False Source: https://awolverp.github.io/markupever/parser_q= Disables detailed error reporting for HTML parsing, providing only a general indication of errors encountered. ```python import markupever p = markupever.Parser(markupever.HtmlOptions(exact_errors=False)) p.process("A Text
") p.finish() print(p.errors()) # ["Unexpected token"] ``` -------------------------------- ### Parse HTML with exact_errors=True Source: https://awolverp.github.io/markupever/parser_q= Enables detailed error reporting for HTML parsing, providing specific details about unexpected tokens and their context within the parsing process. ```python import markupever p = markupever.Parser(markupever.HtmlOptions(exact_errors=True)) p.process("A Text
") p.finish() print(p.errors()) # ["Unexpected token TagToken(Tag { kind: StartTag, name: Atom(\'p\' type=inline), self_closing: false, attrs: [] }) in insertion mode Initial"] ``` -------------------------------- ### Retrieve Quirks Mode from MarkupEver Parser Source: https://awolverp.github.io/markupever/parser Illustrates how to check the `quirks_mode` property of the Parser object. This property indicates the parsing mode, which is typically OFF for XML. ```python import markupever parser = markupever.Parser(options=markupever.HtmlOptions()) parser.process("... content 1 ...") parser.process("... content 2 ...") parser.process("... content 3 ...") parser.finish() print(parser.quirks_mode) # 2 ``` -------------------------------- ### Install MarkupEver using Pip Source: https://awolverp.github.io/markupever/index_q= Installs the MarkupEver library using pip, the Python package installer. It is recommended to use virtual environments for managing Python packages. ```bash $ pip3 install markupever ``` -------------------------------- ### Parse XML file using parse() Source: https://awolverp.github.io/markupever/parser_q= Parses an XML file by reading its entire content into memory using the `parse()` function. It utilizes `XmlOptions` for parsing configuration. This method is suitable for smaller XML files. ```python import markupever with open("file.xml", "rb") as fd: dom = markupever.parse(fd.read(), markupever.XmlOptions()) ``` -------------------------------- ### Repair Incomplete HTML Source: https://awolverp.github.io/markupever/parser_q= Demonstrates MarkupEver's ability to automatically repair incomplete HTML documents during parsing. Serializing the parsed DOM of an incomplete HTML file shows the added closing tags and structure, resulting in a valid HTML document. ```python print(root.serialize()) # #A Text
") p.finish() p.errors() # ["Unexpected token TagToken(Tag { kind: StartTag, name: Atom(\'p\' type=inline), self_closing: false, attrs: [] }) in insertion mode Initial"] ``` -------------------------------- ### Parse XML with Profile - Python Source: https://awolverp.github.io/markupever/parser Parses an XML string using MarkupEver with XML options, enabling profiling to track parsing time. This is useful for performance analysis. The `profile` option is set to `True`. ```python import markupever markupever.parse("A Text
", markupever.XmlOptions(profile=True)) ``` -------------------------------- ### Parse HTML file using parse() Source: https://awolverp.github.io/markupever/parser_q= Parses an HTML file by reading its entire content into memory using the `parse()` function. It utilizes `HtmlOptions` for parsing configuration. This method is suitable for smaller files where memory is not a concern. ```python import markupever with open("index.html", "rb") as fd: dom = markupever.parse(fd.read(), markupever.HtmlOptions()) ``` -------------------------------- ### Parse HTML with pycurl (recommended) Source: https://awolverp.github.io/markupever/more-examples Shows how to parse HTML content fetched using the 'pycurl' library with markupever. This example utilizes pycurl's WRITEDATA option to directly feed response data to the markupever parser. ```python import pycurl import certifi from io import BytesIO # Create a PyCURL instance c = pycurl.Curl() # Define Options ... c.setopt(c.URL, 'https://www.google.com/') c.setopt(c.CAINFO, certifi.where()) # Setup markupever to recieve response parser = markupever.Parser() c.setopt(c.WRITEDATA, parser) # Send Request c.perform() # Close Connection c.close() # Use the parsed DOM dom = parser.finish().into_dom() ``` -------------------------------- ### Parse HTML and Drop DOCTYPE with MarkupEver Source: https://awolverp.github.io/markupever/parser_q= Shows how to parse an HTML document and optionally drop the DOCTYPE declaration using `markupever.HtmlOptions(drop_doctype=True)`. The `serialize()` method is then used to output the resulting DOM. ```python import markupever dom = markupever.parse("A Text
", markupever.HtmlOptions(drop_doctype=True)) dom.serialize() #A Text
``` -------------------------------- ### Parse HTML file using parse_file() Source: https://awolverp.github.io/markupever/parser_q= Parses an HTML file efficiently by reading it chunk by chunk using the `parse_file()` function. This method is recommended for large files as it minimizes memory usage. It accepts a file path and `HtmlOptions` for customization. ```python import markupever dom = markupever.parse_file("index.html", markupever.HtmlOptions()) ``` -------------------------------- ### Retrieve Line Number from MarkupEver Parser Source: https://awolverp.github.io/markupever/parser Shows how to access the `lineno` property of the Parser object after parsing content. This property returns the total line count of the parsed input. ```python import markupever parser = markupever.Parser(options=markupever.HtmlOptions()) parser.process("... content 1 ...") parser.process("... content 2 ...") parser.process("... content 3 ...") parser.finish() print(parser.lineno) # 56 ``` -------------------------------- ### Create a TreeDom Instance in Python Source: https://awolverp.github.io/markupever/treedom_q= Initializes a new document structure using the TreeDom class from the markupever.dom module. This is the starting point for building any document programmatically. ```python from markupever import dom tree = dom.TreeDom() ``` -------------------------------- ### Repair and serialize incomplete HTML Source: https://awolverp.github.io/markupever/parser MarkupEver automatically repairs invalid HTML during parsing. Serializing the parsed DOM will output the corrected HTML structure, ensuring well-formedness. ```python print(root.serialize()) ``` -------------------------------- ### Parse XML file using parse_file() Source: https://awolverp.github.io/markupever/parser_q= Parses an XML file efficiently by reading it chunk by chunk using the `parse_file()` function. This method is recommended for large XML files to minimize memory usage. It accepts a file path and `XmlOptions` for customization. ```python import markupever dom = markupever.parse_file("file.xml", markupever.XmlOptions()) ``` -------------------------------- ### Navigate and query parsed XML DOM Source: https://awolverp.github.io/markupever/parser After parsing XML, you can navigate the TreeDom object. This includes accessing the root node and selecting elements using methods like select_one and select. Namespace-aware selection is also supported. ```python root = dom.root() # Get root node print(root.select_one("bookstore")) for i in root.select("mag|*"): # get all elements which has namespace 'mag' print(i) ``` -------------------------------- ### Parse XML without Profile - Python Source: https://awolverp.github.io/markupever/parser Parses an XML string using MarkupEver with XML options, disabling profiling. This is the default behavior for parsing. The `profile` option is set to `False`. ```python import markupever markupever.parse("A Text
", markupever.XmlOptions(profile=False)) ``` -------------------------------- ### Parse XML with Exact Errors Disabled in MarkupEver Source: https://awolverp.github.io/markupever/parser_q= Shows parsing XML with `exact_errors=False` in `markupever.XmlOptions`, resulting in more general error messages. The `errors()` method is used to inspect the parsing issues. ```python import markupever p = markupever.Parser(markupever.XmlOptions(exact_errors=False)) p.process("A Text
") p.finish() p.errors() # ["Unexpected token"] ``` -------------------------------- ### Navigate and query parsed HTML DOM Source: https://awolverp.github.io/markupever/parser After parsing HTML, you can navigate the resulting TreeDom object. This includes accessing the root node, selecting elements using CSS selectors (e.g., 'title', 'ul'), retrieving element names, serializing elements back to HTML strings, and accessing parent nodes. ```python root = dom.root() # Get root node title = root.select_one("title") # Accepts CSS selectors print(title.name) print(title.serialize()) print(title.text()) print(title.parent.name) ul = root.select_one("ul") print(ul.serialize()) ``` -------------------------------- ### Parse HTML without Dropping DOCTYPE with MarkupEver Source: https://awolverp.github.io/markupever/parser_q= Illustrates parsing an HTML document without dropping the DOCTYPE declaration using `markupever.HtmlOptions(drop_doctype=False)`. The `serialize()` method outputs the DOM including the DOCTYPE. ```python import markupever dom = markupever.parse("A Text
", markupever.HtmlOptions(drop_doctype=False)) dom.serialize() #A Text
``` -------------------------------- ### Parse HTML and Keep DOCTYPE - Python Source: https://awolverp.github.io/markupever/parser Parses an HTML string with MarkupEver while preserving the DOCTYPE declaration. This is the default behavior if `drop_doctype` is not explicitly set to `True`. The `drop_doctype` option is set to `False`. ```python import markupever dom = markupever.parse("A Text
", markupever.HtmlOptions(drop_doctype=False)) dom.serialize() ``` -------------------------------- ### Parse HTML and Drop DOCTYPE - Python Source: https://awolverp.github.io/markupever/parser Parses an HTML string with MarkupEver, specifically dropping the DOCTYPE declaration. This is useful when you want a cleaner DOM representation without the DOCTYPE. The `drop_doctype` option is set to `True`. ```python import markupever dom = markupever.parse("A Text
", markupever.HtmlOptions(drop_doctype=True)) dom.serialize() ``` -------------------------------- ### Parse XML without Exact Errors - Python Source: https://awolverp.github.io/markupever/parser Parses an XML string using MarkupEver with XML options, disabling exact error reporting. This provides more general error messages, which might be faster. The `exact_errors` option is set to `False`. ```python import markupever p = markupever.Parser(markupever.XmlOptions(exact_errors=False)) p.process("A Text
") p.finish() p.errors() ``` -------------------------------- ### Parse XML with Exact Errors - Python Source: https://awolverp.github.io/markupever/parser Parses an XML string using MarkupEver with XML options, enabling exact error reporting. This provides detailed parse errors as described in the specification, potentially at a performance cost. The `exact_errors` option is set to `True`. ```python import markupever p = markupever.Parser(markupever.XmlOptions(exact_errors=True)) p.process("A Text
") p.finish() p.errors() ``` -------------------------------- ### Parse XML Document and Get Root - Python Source: https://awolverp.github.io/markupever/treedom_q= Parses an XML string into a TreeDom object and retrieves the root element. This is the initial step for DOM manipulation in MarkupEver. It requires the 'markupever' library. ```python import markupever dom: markupever.dom.TreeDom = markupever.parse( """CSS Selector Example
I wish you a good day
I wish you a good day
""", markupever.HtmlOptions() ) ``` -------------------------------- ### Prepend Child Node in Python Source: https://awolverp.github.io/markupever/treedom_q= Adds a new child node to the beginning of a parent node's children list. This is achieved by specifying dom.Ordering.PREPEND in the create_element method. ```python from markupever import dom tree = dom.TreeDom() root = tree.root() root.create_element("child1") root.create_element("child2", ordering=dom.Ordering.PREPEND) ``` -------------------------------- ### Parse HTML Content and Extract Links (Python) Source: https://awolverp.github.io/markupever/index_q= Parses HTML content from a file and extracts 'href' attributes from all anchor () tags using MarkupEver. It reads the file in binary mode and uses HtmlOptions for parsing. ```python import markupever with open("index.html", "rb") as fd: dom = markupever.parse(fd.read(), markupever.HtmlOptions()) for element in dom.select("a[href]"): print(element.attrs["href"]) ``` -------------------------------- ### Insert Sibling Node Before in Python Source: https://awolverp.github.io/markupever/treedom_q= Adds a new node as the previous sibling of an existing node. This is done by calling create_element on the existing node and specifying dom.Ordering.BEFORE. ```python from markupever import dom tree = dom.TreeDom() root = tree.root() child = root.create_element("child") child.create_element("sibling", ordering=dom.Ordering.BEFORE) ``` -------------------------------- ### Serialize Document to HTML String in Python Source: https://awolverp.github.io/markupever/treedom_q= Converts the current document structure into an HTML-formatted string. The is_html=True argument ensures proper HTML serialization. ```python tree.serialize(is_html=True) ``` -------------------------------- ### Iterate Over DOM Nodes - Python Source: https://awolverp.github.io/markupever/treedom_q= Provides an overview of various iterator methods in MarkupEver for navigating DOM nodes, including children(), ancestors(), prev_siblings(), next_siblings(), first_children(), last_children(), traverse(), and descendants(). These methods offer flexible ways to explore the document tree. ```python # Example usage of iterator methods (conceptual, not runnable code) # node.children() # node.ancestors() # node.prev_siblings() # node.next_siblings() # node.first_children() # node.last_children() # node.traverse() # node.descendants() ``` -------------------------------- ### Parse HTML File and Extract Links (Python) Source: https://awolverp.github.io/markupever/index_q= Parses an HTML file directly using MarkupEver's parse_file function and extracts 'href' attributes from anchor () tags. This method is convenient for parsing files. ```python import markupever dom = markupever.parse_file("index.html", markupever.HtmlOptions()) for element in dom.select("a[href]"): print(element.attrs["href"]) ``` -------------------------------- ### Parse XML Document into TreeDom - Python Source: https://awolverp.github.io/markupever/treedom Demonstrates parsing an XML string into a TreeDom object using MarkupEver. This is the foundational step for DOM manipulation. It shows the structure of the parsed document, including elements and text nodes. ```python import markupever dom: markupever.dom.TreeDom = markupever.parse( """