nanohtml2text

https://docs.rs/nanohtml2text/0.2.1
Admin
A Rust crate that converts HTML strings to plain text while handling basic HTML tags, entities, and...
Tokens:6,007
Snippets:58
Trust Score:9.1
Update:4 months ago
Show doc for...
Tokens:
Raw
### Link with Href Equal to Content Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests an anchor tag where the link URL is the same as the displayed text.

```Rust
    link_href_equal_to_content: "click <a href=\"test\">test</a>" to "click test",

```

--------------------------------

### Link with Absolute URL Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests anchor tag conversion with a full absolute URL.

```Rust
    link_absolute_url: "click <a href=\"http://bit.ly/2n4wXRs\">news</a>" to "click news (http://bit.ly/2n4wXRs)",

```

--------------------------------

### Test Macro for HTML to Text Conversion

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

A Rust macro to define multiple test cases for the html2text function, comparing input HTML with expected plain text output.

```Rust
    macro_rules! test {
        ($name:ident, $from:literal, $to:literal $(,)?) => {
        #[test]
            fn $name() {
                assert_eq!(html2text($from), $to);
                }
            };
        ($($name:ident: $from:literal to $to:literal,)*) => {
        $(test!{$name, $from, $to})*
        };
    }

```

--------------------------------

### Link Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of an HTML anchor tag with a link and descriptive text.

```Rust
    link: "click <a href=\"test\">here</a>" to "click here (test)",

```

--------------------------------

### Handle Paragraph and Heading Tags

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Converts paragraph (<p>) and heading (<h1> to <h6>) tags, including their closing tags, into a double newline sequence '\r\n\r\n'.

```rust
        // Paragraphs and headings
        "p" | "h1" | "h2" | "h3" | "h4" | "h5" | "h6" | "/h1" | "/h2" | "/h3" | "/h4" | "/h5" | "/h6" => ("\r\n\r\n".to_string(), tag_content.len() + 1),
```

--------------------------------

### Handle Specific HTML Tags

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Parses and handles specific HTML tags like 'head', 'script', and 'style' by skipping their content and finding the corresponding closing tag. Also handles HTML comments.

```Rust
            // Search for the closing tag

            let closing_tag = format!("</{}>", name);
            let lower_rest = rest.to_ascii_lowercase();
            let end_tag_start = lower_rest.find(&closing_tag).unwrap_or(lower_rest.len());
            let closing_tag_len = if end_tag_start < lower_rest.len() {
                closing_tag.len()
            } else {
                0
            };

            let total_skip = tag_content.len() + 1 + end_tag_start + closing_tag_len;

            (String::new(), total_skip)
        }

        // HTML comments
        "!--" => {
            let end = s.find("-->").map_or(s.len(), |n| n + 3);

            (String::new(), end)
        }

        // Discard other tags but keep their content
        _ => (String::new(), tag_content.len() + 1),
    }
}

```

--------------------------------

### Tag with Content Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of an HTML div tag containing simple text.

```Rust
    tag_contents: "<div>simple text</div>" to "simple text",

```

--------------------------------

### Link with JavaScript URL Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests that anchor tags with 'javascript:' URLs are converted without the URL.

```Rust
    link_javascript: "click <a href=\"javascript:void(0)\">here</a>" to "click here",

```

--------------------------------

### Link with Entities in URL Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests anchor tag conversion when the URL contains HTML entities.

```Rust
    link_entities_in_url: "click <a href=\"ents/&apos;x&apos;">here</a>" to "click here (ents/'x')",

```

--------------------------------

### Basic Plaintext Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of a simple string without any HTML tags.

```Rust
    test! {
    plaintext: "blah" to "blah",

```

--------------------------------

### Paragraph Tag Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of multiple paragraph tags into text separated by double newlines.

```Rust
    paragraph: "<p>two</p><p>paragraphs</p>" to "two\r\n\r\nparagraphs",

```

--------------------------------

### Empty Tag Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of an empty HTML div tag, expecting an empty string.

```Rust
    tag: "<div></div>" to "",

```

--------------------------------

### H2 Header Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of an H2 header tag at the beginning of a section.

```Rust
    h2: "<h2>Second</h2>next section" to "Second\r\n\r\nnext section",

```

--------------------------------

### Handle Anchor Tag Conversion

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Processes an anchor tag ('<a>'), extracting the href attribute (if not a javascript: link) and the link text. Formats the output as 'text (href)' if text and href differ, otherwise just the text.

```rust
    // Handle anchor tags
    "a" => {
        // Extract href attribute
        let href = attribs
            .split_ascii_whitespace()
            .find_map(|attr| {
                let mut parts = attr.splitn(2, '=');

                if let (Some(key), Some(value)) = (parts.next(), parts.next()) {
                    if key.eq_ignore_ascii_case("href") {
                        Some(value.trim_matches(['"', '\'' ].as_ref()))
                    } else {
                        None
                    }
                } else {
                    None
                }
            })
            .filter(|href| !href.starts_with("javascript:"))
            .map(html_entities_to_text);

        // Search for closing </a> tag
        let lower_rest = rest.to_ascii_lowercase();
        let end_tag_start = lower_rest.find("</a>").unwrap_or(lower_rest.len());
        let content = &rest[..end_tag_start];

        // Calculate the total length to skip
        let closing_tag_len = if end_tag_start < lower_rest.len() {
            4
        } else {
            0
        };
        // Length of "</a>"

        let total_skip = tag_content.len() + 1 + end_tag_start + closing_tag_len;
        let content_text = html2text(content.trim());
        let link = match (href, content_text.is_empty()) {
            (Some(href_value), false) if content_text != href_value => {
                format!("{} ({})", content_text, href_value)
            }

            (Some(href_value), _) => href_value,

            (_, false) => content_text,

            _ => String::new(),
        };

        (link, total_skip)
    }
```

--------------------------------

### H3 Header Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of an H3 header tag at the beginning of a section.

```Rust
    h3: "<h3>Third</h3>next section" to "Third\r\n\r\nnext section",

```

--------------------------------

### HTML to Text Conversion Function

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

The main function to convert an HTML string to plain text. It collapses whitespace and processes tags and entities.

```Rust
/// Convert an HTML string to plain text.
/// Handles basic HTML tags and entities, and collapses whitespace.
pub fn html2text(html: &str) -> String {
    // Collapse multiple whitespace characters into a single space
    let html = html.split_whitespace().collect::<Vec<_>>().join(" ");
    let mut out = String::new();
    let mut index = 0;
    while index < html.len() {
        if let Some(pos) = html[index..].find('<') {
            if pos > 0 {
                out.push_str(&html_entities_to_text(&html[index..index + pos]));
                index += pos;
            }
            index += 1; // Skip the '<'
            let (parsed_text, advance) = handle_tag(&html[index..]);
            if !parsed_text.is_empty() {
                if out.ends_with("\r\n\r\n") || out.is_empty() {
                    out.push_str(&parsed_text.trim_start());
                } else {
                    out.push_str(&parsed_text);
                }
            }
            index += advance;
        } else {
            // No more tags, process the remaining text
            out.push_str(&html_entities_to_text(&html[index..]));
            break;
        }
    }

    out
}

```

--------------------------------

### Collapse Mixed Whitespace Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of mixed whitespace characters (spaces, newlines, carriage returns) into single spaces.

```Rust
    collapse_mixed: "should \nignore \r\nnew lines" to "should ignore new lines",

```

--------------------------------

### Collapse Spaces Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests that multiple spaces in the HTML are collapsed into a single space in the output.

```Rust
    collapse_spaces: "should ignore more spaces" to "should ignore more spaces",

```

--------------------------------

### H1 Header Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of an H1 header tag, followed by main text, ensuring proper separation with double newlines.

```Rust
    h1: "<h1>First</h1>main text" to "First\r\n\r\nmain text",

```

--------------------------------

### Handle Line Break and List Item Tags

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Converts specific HTML tags like <br>, <br/>, <li>, </ol>, and </ul> into a newline character sequence '\r\n'.

```rust
        // Line breaks and list items
        "br" | "br/" | "li" | "/ol" | "/ul" => ("\r\n".to_string(), tag_content.len() + 1),
```

--------------------------------

### H3 Inline Header Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests an H3 header tag that appears inline with preceding text.

```Rust
    h3_inline: "Second<h3>Third</h3>next section" to "Second\r\n\r\nThird\r\n\r\nnext section",

```

--------------------------------

### Collapse Line Breaks Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests that newline characters in the HTML are converted to spaces in the output.

```Rust
    collapse_linebreaks: "a\nb\nc" to "a b c",

```

--------------------------------

### BR Tag Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of '<br>' and '<br/>' tags into newline characters ('\r\n').

```Rust
    br_tag: "two<br>line<br/>breaks" to "two\r\nline\r\nbreaks",

```

--------------------------------

### H4 Header Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests the conversion of an H4 header tag at the beginning of a section.

```Rust
    h4: "<h4>Fourth</h4>next section" to "Fourth\r\n\r\nnext section",

```

--------------------------------

### Multiple Links Ignoring Attributes Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests conversion of multiple anchor tags, ensuring attributes are ignored and URLs are correctly appended.

```Rust
    link_ignore_attributes_2: "<a rel=\"mw:WikiLink\" href=\"/wiki/yet#English\" title=\"yet\">yet</a>, <a rel=\"mw:WikiLink\" href=\"/wiki/not_yet#English\" title=\"not yet\">not yet</a>" to "yet (/wiki/yet#English), not yet (/wiki/not_yet#English)",

```

--------------------------------

### H2 Inline Header Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests an H2 header tag that appears inline with preceding text, ensuring correct newline separation.

```Rust
    h2_inline: "First<h2>Second</h2>next section" to "First\r\n\r\nSecond\r\n\r\nnext section",

```

--------------------------------

### H4 Inline Header Conversion Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests an H4 header tag that appears inline with preceding text.

```Rust
    h4_inline: "Third<h4>Fourth</h4>next section" to "Third\r\n\r\nFourth\r\n\r\nnext section",

```

--------------------------------

### HTML Entity Definitions

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/entity.rs.html

A static array defining common HTML entities and their corresponding Unicode characters. This is used for converting HTML entities to their character representations.

```rust
pub static ENTITIES: &'static [(&'static str, char)] = &[
    ("AElig", '\u{00C6}'),
    ("AMP", '\u{000026}'),
    ("Aacute", '\u{0000C1}'),
    ("Abreve", '\u{000102}'),
    ("Acirc", '\u{0000C2}'),
    ("Acy", '\u{000410}'),
    ("Afr", '\u{01D504}'),
    ("Agrave", '\u{0000C0}'),
    ("Alpha", '\u{000391}'),
    ("Amacr", '\u{000100}'),
    ("And", '\u{002A53}'),
    ("Aogon", '\u{000104}'),
    ("Aopf", '\u{01D538}'),
    ("ApplyFunction", '\u{002061}'),
    ("Aring", '\u{0000C5}'),
    ("Ascr", '\u{01D49C}'),
    ("Assign", '\u{002254}'),
    ("Atilde", '\u{0000C3}'),
    ("Auml", '\u{0000C4}'),
    ("Backslash", '\u{002216}'),
    ("Barv", '\u{002AE7}'),
    ("Barwed", '\u{002306}'),
    ("Bcy", '\u{000411}'),
    ("Because", '\u{002235}'),
    ("Bernoullis", '\u{00212C}'),
    ("Beta", '\u{000392}'),
    ("Bfr", '\u{01D505}'),
    ("Bopf", '\u{01D539}'),
    ("Breve", '\u{0002D8}'),
    ("Bscr", '\u{00212C}'),
    ("Bumpeq", '\u{00224E}'),
    ("CHcy", '\u{000427}'),
    ("COPY", '\u{0000A9}'),
    ("Cacute", '\u{000106}'),
    ("Cap", '\u{0022D2}'),
    ("CapitalDifferentialD", '\u{002145}'),
    ("Cayleys", '\u{00212D}'),
    ("Ccaron", '\u{00010C}'),
    ("Ccedil", '\u{0000C7}'),
    ("Ccirc", '\u{000108}'),
    ("Cconint", '\u{002230}'),
    ("Cdot", '\u{00010A}'),
    ("Cedilla", '\u{0000B8}'),
    ("CenterDot", '\u{0000B7}'),
    ("Cfr", '\u{00212D}'),
    ("Chi", '\u{0003A7}'),
    ("CircleDot", '\u{002299}'),
    ("CircleMinus", '\u{002296}'),
    ("CirclePlus", '\u{002295}'),
    ("CircleTimes", '\u{002297}'),
    ("ClockwiseContourIntegral", '\u{002232}'),
    ("CloseCurlyDoubleQuote", '\u{00201D}'),
    ("CloseCurlyQuote", '\u{002019}'),
    ("Colon", '\u{002237}'),
    ("Colone", '\u{002A74}'),
    ("Congruent", '\u{002261}'),
    ("Conint", '\u{00222F}'),
    ("ContourIntegral", '\u{00222E}'),
    ("Copf", '\u{002102}'),
    ("Coproduct", '\u{002210}'),
    ("CounterClockwiseContourIntegral", '\u{002233}'),
    ("Cross", '\u{002A2F}'),
    ("Cscr", '\u{01D49E}'),
    ("Cup", '\u{0022D3}'),
    ("CupCap", '\u{00224D}'),
    ("DD", '\u{002145}'),
    ("DDotrahd", '\u{002911}'),
    ("DJcy", '\u{000402}'),
    ("DScy", '\u{000405}'),
    ("DZcy", '\u{00040F}'),
    ("Dagger", '\u{002021}'),
    ("Darr", '\u{0021A1}'),
    ("Dashv", '\u{002AE4}'),
    ("Dcaron", '\u{00010E}'),
    ("Dcy", '\u{000414}'),
    ("Del", '\u{002207}'),
    ("Delta", '\u{000394}'),
    ("Dfr", '\u{01D507}'),
    ("DiacriticalAcute", '\u{0000B4}'),
    ("DiacriticalDot", '\u{0002D9}'),
    ("DiacriticalDoubleAcute", '\u{0002DD}'),
    ("DiacriticalGrave", '\u{000060}'),
    ("DiacriticalTilde", '\u{0002DC}'),
    ("Diamond", '\u{0022C4}'),
    ("DifferentialD", '\u{002146}'),
    ("Dopf", '\u{01D53B}'),
    ("Dot", '\u{0000A8}'),
    ("DotDot", '\u{0020DC}'),
    ("DotEqual", '\u{002250}'),
    ("DoubleContourIntegral", '\u{00222F}'),
    ("DoubleDot", '\u{0000A8}'),
    ("DoubleDownArrow", '\u{0021D3}'),
    ("DoubleLeftArrow", '\u{0021D0}'),
    ("DoubleLeftRightArrow", '\u{0021D4}'),
    ("DoubleLeftTee", '\u{002AE4}'),
    ("DoubleLongLeftArrow", '\u{0027F8}'),
    ("DoubleLongLeftRightArrow", '\u{0027FA}'),
    ("DoubleLongRightArrow", '\u{0027F9}'),
    ("DoubleRightArrow", '\u{0021D2}'),
    ("DoubleRightTee", '\u{0022A8}'),
    ("DoubleUpArrow", '\u{0021D1}'),
    ("DoubleUpDownArrow", '\u{0021D5}'),
    ("DoubleVerticalBar", '\u{002225}'),
    ("DownArrow", '\u{002193}')
];
```

--------------------------------

### Convert HTML Entities to Text

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Converts all HTML entities within a string to their corresponding characters. It iterates through the string, parses entities, and reconstructs the text.

```rust
fn html_entities_to_text(s: &str) -> String {
    let mut out = String::new();
    let mut parts = s.split('&');

    // Add the first part (before any '&')
    out.push_str(parts.next().unwrap_or_default());

    for part in parts {
        let end = part
            .find(|c: char| c.is_whitespace() || c == ';')
            .unwrap_or_else(|| part.len());

        if let Some(entity) = parse_html_entity(&part[..end]) {
            out.push(entity);

            // Advance past the entity and any following semicolon or whitespace
            let next_char_len = part[end..].chars().next().map_or(0, |c| c.len_utf8());
            let remaining = &part[end + next_char_len..];

            out.push_str(remaining);
        } else {
            out.push('&');

            out.push_str(part);
        }
    }

    out
}
```

--------------------------------

### html2text Function Signature

Source: https://docs.rs/nanohtml2text/0.2.1/nanohtml2text/fn.html2text.html

This is the function signature for html2text. It takes an HTML string slice as input and returns a plain text String.

```rust
pub fn html2text(html: &str) -> String
```

--------------------------------

### HTML Entity to Unicode Mapping

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/entity.rs.html

This snippet shows a Rust array mapping HTML entity names (like 'yacy', 'ycirc') to their Unicode character codes. This is used for converting HTML entities into their text equivalents.

```rust
("yacy", '\u{00044F}'),
    ("ycirc", '\u{000177}'),
    ("ycy", '\u{00044B}'),
    ("yen", '\u{0000A5}'),
    ("yfr", '\u{01D536}'),
    ("yicy", '\u{000457}'),
    ("yopf", '\u{01D56A}'),
    ("yscr", '\u{01D4CE}'),
    ("yucy", '\u{00044E}'),
    ("yuml", '\u{0000FF}'),
    ("zacute", '\u{00017A}'),
    ("zcaron", '\u{00017E}'),
    ("zcy", '\u{000437}'),
    ("zdot", '\u{00017C}'),
    ("zeetrf", '\u{002128}'),
    ("zeta", '\u{0003B6}'),
    ("zfr", '\u{01D537}'),
    ("zhcy", '\u{000436}'),
    ("zigrarr", '\u{0021DD}'),
    ("zopf", '\u{01D56B}'),
    ("zscr", '\u{01D4CF}'),
    ("zwj", '\u{00200D}'),
    ("zwnj", '\u{00200C}')
```

--------------------------------

### html2text

Source: https://docs.rs/nanohtml2text/0.2.1/nanohtml2text/fn.html2text.html

Converts an HTML string to plain text. It handles basic HTML tags and entities, and collapses whitespace.

```APIDOC
## html2text

### Description
Converts an HTML string to plain text. Handles basic HTML tags and entities, and collapses whitespace.

### Signature
```rust
pub fn html2text(html: &str) -> String
```

### Parameters
* **html** (*&str*) - The HTML string to convert.
```

--------------------------------

### Link Ignoring Nested Tags Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests anchor tag conversion when the link text contains nested HTML tags.

```Rust
    link_ignore_content_tags: "click <a href=\"test\"><span>here</span> or here</a>" to "click here or here (test)",

```

--------------------------------

### Ignoring Inline Element Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests that inline HTML elements like 'strong' are ignored, and their content is preserved.

```Rust
    ignore_inline: "strong <strong>text</strong>" to "strong text",

```

--------------------------------

### Decode Named HTML Entity

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Decodes a named HTML entity into its corresponding character. Uses a binary search on a predefined list of entities.

```rust
fn decode_named_entity(entity: &str) -> Option<char> {
    entity::ENTITIES
        .binary_search_by_key(&entity, |&(name, _)| name)
        .ok()
        .map(|idx| entity::ENTITIES[idx].1)
}
```

--------------------------------

### Parse HTML Entity (Named or Numeric)

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Parses a given string as either a named or numeric HTML entity and returns the corresponding character. Handles hexadecimal and decimal numeric entities.

```rust
fn parse_html_entity(entity: &str) -> Option<char> {
    if let Some(c) = decode_named_entity(entity) {
        return Some(c);
    }

    let num = entity.strip_prefix('#')?;

    let code_point = if let Some(hex) = num.strip_prefix(|c| c == 'x' || c == 'X') {
        u32::from_str_radix(hex, 16).ok()? 
    } else {
        u32::from_str_radix(num, 10).ok()? 
    };

    // Exclude control characters and ensure valid Unicode code point
    if matches!(code_point, 0x09 | 0x0A | 0x0D | 0x20..) {
        char::from_u32(code_point)
    } else {
        None
    }
}
```

--------------------------------

### Link Ignoring Attributes Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests that extra attributes in an anchor tag are ignored during conversion.

```Rust
    links_ignore_attributes: "click <a class=\"x\" href=\"test\">here</a>" to "click here (test)",

```

--------------------------------

### Ignoring Inline Element with Attributes Test Case

Source: https://docs.rs/nanohtml2text/0.2.1/src/nanohtml2text/lib.rs.html

Tests that inline HTML elements with attributes, like a 'div', are ignored, and their content is preserved.

```Rust
    ignore_inline_attributes: "some <div id=\"a\" class=\"b\">div</div>" to "some div",

```
Tokens:
Raw