### Set up virtual environment and install dev dependencies Source: https://github.com/emilstenstrom/justhtml/blob/main/CONTRIBUTING.md Creates a virtual environment, activates it, and installs the package with development dependencies. ```bash python -m venv .venv source .venv/bin/activate pip install -e ".[dev]" ``` -------------------------------- ### Install pre-commit hooks Source: https://github.com/emilstenstrom/justhtml/blob/main/CONTRIBUTING.md Installs the pre-commit hooks configured in the repository. ```bash pre-commit install ``` -------------------------------- ### Install JustHTML benchmark dependencies Source: https://github.com/emilstenstrom/justhtml/blob/main/PERFORMANCE.md Installs the extra dependencies needed to run the benchmarks, including html5-parser and lxml. Must be run before executing the benchmark. ```bash pip install -e ".[benchmark]" ``` -------------------------------- ### Quick example with SetAttrs and Drop Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/transforms.md Demonstrates a quick example of using SetAttrs and Drop transforms. The tree is mutated in memory and output remains safe by default. ```python from justhtml import JustHTML, Drop, SetAttrs doc = JustHTML( "
Hello
", transforms=[ SetAttrs("p", id="greeting"), Drop("script"), ], ) # The tree is transformed in memory print(doc.to_html()) # Output is still safe by default print(doc.to_html(pretty=False)) ``` -------------------------------- ### Practical selector examples Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/selectors.md A collection of example queries for common tasks: navigation links, active menu items, form inputs excluding hidden, alternating table rows, first paragraphs, and external links. ```python # Navigation links doc.query("nav a") # Active menu item doc.query("nav li.active > a") # Form inputs except hidden doc.query("form input:not([type='hidden'])") # Alternating table rows doc.query("tr:nth-child(even)") # First paragraph in each container doc.query("div > p:first-of-type") # External links doc.query('a[href^="http"]:not([href*="mysite.com"])') ``` -------------------------------- ### Documentation example with output marker Source: https://github.com/emilstenstrom/justhtml/blob/main/CONTRIBUTING.md Shows the documentation example format: a Python fence followed by an explicit output marker and an expected-output fence. The docs test executes the Python block and compares its stdout with the following fence. ```text ```python from justhtml import JustHTML print(JustHTML("x
", fragment=True).to_html()) ``` ```htmlx
``` ``` -------------------------------- ### justhtml CLI usage examples Source: https://github.com/emilstenstrom/justhtml/blob/main/README.md Command-line usage examples for the justhtml CLI tool, showing how to pretty-print an HTML file, parse from stdin, extract text from selected nodes, and convert selected HTML to Markdown. ```bash justhtml index.html ``` ```bash curl -s https://example.com | justhtml - ``` ```bash justhtml index.html --selector "main p" --format text ``` ```bash justhtml index.html --selector "article" --format markdown ``` -------------------------------- ### Install JustHTML with pip Source: https://github.com/emilstenstrom/justhtml/blob/main/README.md Install the JustHTML package. Requires Python 3.10 or later. ```bash pip install justhtml ``` -------------------------------- ### Clone test suites and set up symlinks Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/correctness.md One-time setup: clone the WPT and html5lib test suites, then create symlinks in the tests directory. Run from the parent directory of justhtml. ```bash # Clone the test suites (one-time setup) cd .. git clone --filter=blob:none --sparse https://github.com/web-platform-tests/wpt.git cd wpt git sparse-checkout set html/syntax/parsing/resources cd .. git clone https://github.com/html5lib/html5lib-tests.git cd justhtml # Create symlinks cd tests ln -s ../../wpt/html/syntax/parsing/resources html5lib-tests-tree ln -s ../../html5lib-tests/serializer html5lib-tests-serializer ln -s ../../html5lib-tests/encoding html5lib-tests-encoding cd .. # Run all tests python run_tests.py ``` -------------------------------- ### Usage example for clean() wrapper Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/bleach-migration.md Shows how to replace existing bleach.clean(user_html) calls with the provided clean() wrapper. The comment indicates the direct substitution. ```python # Replace existing bleach.clean(user_html) calls with clean(user_html). safe_html = clean(user_html) ``` -------------------------------- ### to_markdown() output for complex HTML example Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/text.md The Markdown output for the preceding example. Shows headings, bold text, links, list items, and an indented code block. ```markdown # Title Hello **world** and [links](https://example.com). - First item - Second item ``` code block ``` ``` -------------------------------- ### Fuzz testing output Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/correctness.md Example output from the fuzz benchmark, showing no crashes or hangs. ```text ============================================================ FUZZING RESULTS: justhtml ============================================================ Total tests: 3000000 Successes: 3000000 Crashes: 0 Hangs (>5s): 0 Total time: 928s Tests/second: 3232 ``` -------------------------------- ### Sample HTML input for conversion Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/migration-examples.md Sample HTML input used in the conversion example. It contains paragraphs and a link with class 'source'. ```html
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa
Consectetuer adipiscing elit. Some Link Aenean commodo ligula eget dolor. Aenean massa
Aenean massa.Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa
Consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa
Hello world
Hello world
") doc.to_text() # => Hello world ``` -------------------------------- ### to_markdown() example Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/api.md Creates a JustHTML document and calls to_markdown() to get GitHub Flavored Markdown. Tables and images are preserved as raw HTML; script, style, and textarea tags are dropped by default unless html_passthrough=True. Sanitization happens at construction time; the returned Markdown is not escaped HTML and should be rendered with a compliant Markdown renderer. ```python doc = JustHTML("Hello world
") doc.to_markdown() # => # Title # => # => Hello **world** doc.to_markdown(html_passthrough=True) ``` -------------------------------- ### Get all text inside an element - JustHTML Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/migration-examples.md JustHTML code to get all text inside an element using to_text(). ```python from justhtml import JustHTML doc = JustHTML(html, fragment=True, sanitize=False) matches = doc.root.query("a.someclass") print(matches[0].to_text() if matches else "") # => Wie Messi einen kleinen Jungen stehen lässt ``` -------------------------------- ### Get all text inside an element - lxml Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/migration-examples.md lxml code to get all text inside an element using itertext(). ```python from lxml import html as lxml_html root = lxml_html.fromstring(html) el = root.cssselect("a.someclass") print("".join(el[0].itertext()) if el else "") ``` -------------------------------- ### Initialize FragmentContext Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/fragments.md Shows how to initialize FragmentContext for HTML, SVG, and MathML namespaces. ```python from justhtml.parser.context import FragmentContext # HTML namespace (default) ctx = FragmentContext("div") ctx = FragmentContext("div", namespace=None) # SVG namespace ctx = FragmentContext("svg", namespace="svg") # MathML namespace ctx = FragmentContext("math", namespace="math") ``` -------------------------------- ### Benchmark and profile commands Source: https://github.com/emilstenstrom/justhtml/blob/main/CONTRIBUTING.md Run a quick benchmark with the justhtml parser for one iteration, then profile hotspots with the profile script. Use these commands to verify performance impact after making changes. ```bash # Quick benchmark python benchmarks/performance.py --parsers justhtml --iterations 1 ``` ```bash # Profile hotspots python benchmarks/profile.py ``` -------------------------------- ### Allow your main host and a CDN host Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/url-cleaning.md Allows HTTPS URLs from example.com and static.example.com. Use when content is served from a primary domain and a CDN. ```python UrlRule( allowed_schemes={"https"}, allowed_hosts={"example.com", "static.example.com"}, ) ``` -------------------------------- ### Get all text inside an element Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/migration-examples.md Shows how to get all text inside an element, including text from child elements. The lxml version uses itertext() and the JustHTML version uses to_text(). ```html Wie Messi einen kleinen Jungen stehen lässt ``` -------------------------------- ### Run the full test suite Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/correctness.md Executes the complete html5lib test suite on every commit. ```bash python run_tests.py ``` -------------------------------- ### Get source locations (line/col/offset) Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/migration-examples.md Shows how to get source location information (line, column, offset) for elements. The original uses lxml's sourceline; JustHTML provides origin_line, origin_col, and origin_offset when track_node_locations=True. ```python from lxml import html as lxml_html root = lxml_html.fromstring(html) for el in root.xpath("//p"): print(el.sourceline) ``` ```python from justhtml import JustHTML doc = JustHTML(html, fragment=True, sanitize=False, track_node_locations=True) locs = [(el.origin_line, el.origin_col, el.origin_offset) for el in doc.root.query("p")] print(locs) # => [(1, 1, 0)] ``` -------------------------------- ### Run tests with various options Source: https://github.com/emilstenstrom/justhtml/blob/main/CONTRIBUTING.md Commands for running the test suite, individual suites, coverage, specific test files, and quick snippet testing. Coverage is required to be 100%. ```bash # Run all tests python run_tests.py # Run one suite (faster iteration) python run_tests.py --suite tree python run_tests.py --suite justhtml python run_tests.py --suite tokenizer # Run with coverage report coverage run run_tests.py && coverage report # Run specific test file python run_tests.py --test-specs test2.test:5,10 -v # Quick iteration - test a snippet python -c 'from justhtml import JustHTML, to_test_format; print(to_test_format(JustHTML("").root))' ``` -------------------------------- ### Get source locations (line/col/offset) Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/migration-examples.md Shows how to get source location information (line, column, offset) for elements. The original uses lxml's sourceline; JustHTML provides origin_line, origin_col, and origin_offset when track_node_locations=True. ```htmlsome text ``` -------------------------------- ### Create test symlinks Source: https://github.com/emilstenstrom/justhtml/blob/main/tests/README.md Creates symlinks in the `tests/` directory to the test repositories. Run from `tests/` after cloning. ```bash cd tests ln -s ../../wpt/html/syntax/parsing/resources html5lib-tests-tree ln -s ../../html5lib-tests/serializer html5lib-tests-serializer ln -s ../../html5lib-tests/encoding html5lib-tests-encoding ``` -------------------------------- ### Basic selectors with doc.query and doc.query_one Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/selectors.md Shows how to use tag, class, ID, and universal selectors. doc.query returns all matching elements, while doc.query_one returns the first match or None. ```python doc.query("div") # All
tags. The output is printed with pretty=False and results inHello world
a b. ```python from justhtml import CollapseWhitespace, JustHTML doc = JustHTML( "\n Hello \n\t world\n
a b", fragment=True, transforms=[CollapseWhitespace()], ) print(doc.to_html(pretty=False)) # =>Hello world
a b``` -------------------------------- ### Build first, query later with query_one() Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/building.md Builds a DOM first and then queries it with query_one() to retrieve a specific element. ```python from justhtml import JustHTML from justhtml.dom.builder import element doc = JustHTML( element( "div", element("p", {"class": "lead"}, "Hello"), element("p", "World"), ), fragment=True, ) lead = doc.query_one("p.lead") lead.to_html(pretty=False) # =>Hello
``` -------------------------------- ### Disable sanitization for raw output Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/transforms.md Shows how to get raw output by disabling sanitization with sanitize=False. The script tag is preserved in the output. ```python doc = JustHTML("Hello
", fragment=True, sanitize=False) print(doc.to_html(pretty=False)) # =>Hello
``` -------------------------------- ### query_one() example Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/api.md Calls query_one() with a CSS selector to return the first matching descendant or None. The return type is QueryMatch | None. ```python node = doc.query_one("div.container > p") ``` -------------------------------- ### Copyable clean() compatibility wrapper for Bleach migration Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/bleach-migration.md Provides a drop-in replacement for bleach.clean() and bleach.linkify() using JustHTML, reproducing Bleach 6.x default tags, attributes, and protocols. Includes build_url_policy() helper and handles strip_comments behavior. Use as a compatibility baseline, not a recommended final policy. ```python import re from collections.abc import Collection, Mapping from justhtml import JustHTML, Linkify, SanitizationPolicy, SetAttrs, UrlPolicy, UrlRule URL_LIKE_ATTRS = { "href", "src", "srcset", "poster", "action", "formaction", "data", "cite", "background", "ping", } BLEACH_DEFAULT_TAGS = frozenset( {"a", "abbr", "acronym", "b", "blockquote", "code", "em", "i", "li", "ol", "strong", "ul"} ) BLEACH_DEFAULT_ATTRIBUTES = { "a": {"href", "title"}, "abbr": {"title"}, "acronym": {"title"}, } BLEACH_DEFAULT_PROTOCOLS = frozenset({"http", "https", "mailto"}) def build_url_policy( allowed_tags: Collection[str], allowed_attributes: Mapping[str, Collection[str]], protocols: Collection[str], ) -> UrlPolicy: rules = {} global_attrs = allowed_attributes.get("*", ()) for tag in allowed_tags: for attr in global_attrs: if attr in URL_LIKE_ATTRS: rules[(tag, attr)] = UrlRule( allowed_schemes=protocols, allow_relative=True, ) for tag, attrs in allowed_attributes.items(): if tag == "*": continue for attr in attrs: if attr in URL_LIKE_ATTRS: rules[(tag, attr)] = UrlRule( allowed_schemes=protocols, allow_relative=True, ) return UrlPolicy(allow_rules=rules) def clean( html: str, *, tags: Collection[str] = BLEACH_DEFAULT_TAGS, attributes: Mapping[str, Collection[str]] = BLEACH_DEFAULT_ATTRIBUTES, protocols: Collection[str] = BLEACH_DEFAULT_PROTOCOLS, css_properties: Collection[str] = (), strip: bool = False, strip_comments: bool = True, ) -> str: attrs = {tag: set(values) for tag, values in attributes.items()} disallowed_tag_handling = "unwrap" if strip else "escape" # Bleach defaults to strip_comments=True. If you use escape mode # (`strip=False`) and still want comments removed rather than displayed, # remove comments before parsing. if strip_comments and disallowed_tag_handling == "escape": html = re.sub(r"", "", html, flags=re.DOTALL) policy = SanitizationPolicy( allowed_tags=tags, allowed_attributes=attrs, allowed_css_properties=css_properties, url_policy=build_url_policy(tags, attrs, protocols), drop_comments=strip_comments, disallowed_tag_handling=disallowed_tag_handling, ) return JustHTML(html, fragment=True, policy=policy).to_html(pretty=False) def linkify(text: str, *, nofollow: bool = False) -> str: transforms = [Linkify()] if nofollow: transforms.append(SetAttrs("a", rel="nofollow")) return JustHTML( text, fragment=True, sanitize=False, transforms=transforms, ).to_html(pretty=False) ``` -------------------------------- ### Clone test repositories with sparse checkout Source: https://github.com/emilstenstrom/justhtml/blob/main/tests/README.md Clones wpt with sparse checkout and html5lib-tests. Requires that `justhtml` is already cloned and you are in its parent directory. ```bash cd .. git clone --filter=blob:none --sparse https://github.com/web-platform-tests/wpt.git cd wpt git sparse-checkout set html/syntax/parsing/resources cd .. git clone https://github.com/html5lib/html5lib-tests.git cd justhtml ``` -------------------------------- ### JustHTML constructor parameters Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/api.md Shows the signature of the JustHTML constructor with all parameters and keyword-only arguments. The html parameter is required; all others default as documented. ```python JustHTML( html, *, sanitize=True, policy=None, collect_errors=False, track_node_locations=False, debug=False, encoding=None, fragment=False, fragment_context=None, iframe_srcdoc=False, strict=False, transforms=None, ) ``` -------------------------------- ### Structural pseudo-class selectors Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/selectors.md Examples of structural pseudo-class selectors for querying list items by position, including first, last, nth-child with formulas and odd/even. ```python doc.query("li:first-child") # First list item doc.query("li:last-child") # Last list item doc.query("li:nth-child(2)") # Second list item doc.query("li:nth-child(odd)") # Odd list items (1st, 3rd, 5th...) doc.query("li:nth-child(even)") # Even list items (2nd, 4th, 6th...) doc.query("li:nth-child(3n)") # Every 3rd item doc.query("li:nth-child(3n+1)") # 1st, 4th, 7th... (formula: An+B) ``` -------------------------------- ### Escape element tags Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/transforms.md Escapes the start and end tags of matching elements into text nodes while keeping children. The output shows the escaped tags. ```python from justhtml import Escape, JustHTML doc = JustHTML( "", fragment=True, sanitize=False, transforms=[Escape("x")], ) print(doc.to_html(pretty=False)) # =>
hi <x>hi</x>
``` -------------------------------- ### UrlRule constructor with all parameters Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/api.md Creates a UrlRule instance to control URL validation for attributes like a[href] and img[src]. Parameters include allow_fragment, resolve_protocol_relative, allowed_schemes, allowed_hosts, handling, and proxy. Defaults are shown; allowed_schemes defaults to an empty set, allowed_hosts to None, handling to None (uses UrlPolicy.default_handling), and proxy to None. ```python UrlRule( allow_fragment=True, resolve_protocol_relative="https", allowed_schemes=set(), allowed_hosts=None, handling=None, proxy=None, ) ``` -------------------------------- ### Raise unsafe handling with UnsafeHtmlError Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/unsafe-handling.md Configure SanitizationPolicy with unsafe_handling='raise' to raise UnsafeHtmlError at the first unsafe construct. The example shows catching the exception and printing its message. ```python from justhtml import JustHTML, SanitizationPolicy, UnsafeHtmlError, UrlPolicy doc = JustHTML("ok
", fragment=True) policy = SanitizationPolicy( allowed_tags=["p"], allowed_attributes={"*": []}, url_policy=UrlPolicy(allow_rules={}), unsafe_handling="raise", ) try: _ = JustHTML("ok
", fragment=True, policy=policy) except UnsafeHtmlError as e: print(e) ``` ```text Unsafe tag 'script' (dropped content) ``` -------------------------------- ### Initialize JustHTML and sanitize input Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/comparison.md Demonstrates initializing a JustHTML document from an HTML string and serializing it. Sanitization is enabled by default, removing elements like script tags and unsafe links. ```python from justhtml import JustHTML doc = JustHTML("Hellolink
", fragment=True) print(doc.to_html(pretty=False)) #Hellolink
``` -------------------------------- ### FragmentContext Constructor Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/api.md Initialize a new FragmentContext with a tag name and optional namespace. ```python FragmentContext(tag_name, namespace=None) ``` -------------------------------- ### Streaming API with stream() Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/quickstart.md Use the streaming API for large files or when the full DOM is not needed. The output shows the sequence of start, text, and end events. ```python from justhtml import stream html = "Hello
world
" for event, data in stream(html): if event == "start": tag, attrs = data print(f"Start: {tag}") elif event == "text": print(f"Text: {data}") elif event == "end": print(f"End: {data}") ``` ```html Start: p Text: Hello End: p Start: p Text: world End: p ``` -------------------------------- ### Count elements Source: https://github.com/emilstenstrom/justhtml/blob/main/docs/streaming.md Counts occurrences of each start tag using a Counter and prints the 10 most common tags. Requires html to be defined (e.g., from prior code). ```python from justhtml import stream from collections import Counter counts = Counter() for event, data in stream(html): if event == "start": tag, attrs = data counts[tag] += 1 print(counts.most_common(10)) ```