html5gum

https://docs.rs/html5gum/0.8.3/
Admin
A WHATWG-compliant HTML tokenizer for Rust that efficiently tokenizes HTML documents and passes...
Tokens:170,491
Snippets:1,008
Trust Score:9.1
Update:6 months ago
Show doc for...
Tokens:
Raw
### Initialize start tag token

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/callback/struct.CallbackEmitter.html

Sets the current token to be a start tag, preparing for tag name and attribute emission.

```rust
fn init_start_tag(&mut self)
```

--------------------------------

### Tokenizer example with StartTag

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/reader.rs.html

This example shows how to use the `Tokenizer` to process an HTML string and match `StartTag` tokens. It demonstrates basic tokenization and handling of tag attributes.

```rust
use std::fmt::Write;
use html5gum::{Tokenizer, Token};

let html = "<title   >hello world</title>";
let mut new_html = String::new();

for Ok(token) in Tokenizer::new(html) {
    match token {
        Token::StartTag(tag) => {

```

--------------------------------

### read_until Method Implementation Example

Source: https://docs.rs/html5gum/0.8.3/html5gum/trait.Reader.html

Demonstrates reading characters until a delimiter is found. This example uses `StringReader` and collects chunks of data.

```rust
use html5gum::{Reader, Readable};

let mut reader = "hello world".to_reader();
let mut eof = false;
let mut chunks = Vec::new();
while !eof {
    let mut char_buf = [0; 4];
    let xs = reader.read_until(&[b' ', b'r'], &mut char_buf).unwrap();
    if let Some(xs) = xs {
        chunks.push(std::str::from_utf8(xs).unwrap().to_owned());
    } else {
        eof = true;
    }
}

assert_eq!(chunks, &["hello", " ", "wo", "r", "ld"]);
```

--------------------------------

### Initialize Attribute

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/emitters/callback.rs.html

Flushes the open start tag to prepare for attribute processing.

```rust
fn init_attribute(&mut self) {
        self.flush_open_start_tag();
```

--------------------------------

### starts_with

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Checks if a slice starts with a specific prefix.

```APIDOC
## starts_with

### Description
Returns true if needle is a prefix of the slice or equal to the slice.

### Parameters
- **needle** (&[T]) - Required - The prefix to check.
```

--------------------------------

### Start Open Tag

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/trait.ForwardingEmitter.html

Initiates the process of parsing an open tag.

```rust
fn start_open_tag(&mut self) { ... }
```

--------------------------------

### Start Open Tag

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/emitters/callback.rs.html

Records the starting position of an open tag. This is used to track the span of the tag for potential future use, like error reporting or reconstruction.

```rust
fn start_open_tag(&mut self) {
        self.emitter_state.current_taglike_span = self.emitter_state.position.offset(-1);
    }
```

--------------------------------

### Set last start tag name

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/callback/struct.CallbackEmitter.html

Records the name of the most recently encountered start tag.

```rust
fn set_last_start_tag(&mut self, last_start_tag: Option<&[u8]>)
```

--------------------------------

### Get naive next state from tag name

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/fn.naive_next_state.html

Use this function to guess the next state based on a start tag name. It's useful for HTML scraping but not for full browser emulation.

```rust
pub fn naive_next_state(tag_name: &[u8]) -> Option<State>
```

--------------------------------

### Initialize Comment

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/emitters/callback.rs.html

Prepares the tokenizer to start a new comment by flushing existing characters and clearing the current comment buffer.

```rust
fn init_comment(&mut self) {
        self.flush_current_characters();
        self.emitter_state.current_comment.clear();
    }
```

--------------------------------

### Check slice prefix

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Verify if a slice starts with a specific sequence. Empty slices always return true.

```rust
let v = [10, 40, 30];
assert!(v.starts_with(&[10]));
assert!(v.starts_with(&[10, 40]));
assert!(v.starts_with(&v));
assert!(!v.starts_with(&[50]));
assert!(!v.starts_with(&[10, 50]));
```

```rust
let v = &[10, 40, 30];
assert!(v.starts_with(&[]));
let v: &[u8] = &[];
assert!(v.starts_with(&[]));
```

--------------------------------

### Configure html5gum crate root

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/lib.rs.html

Sets up crate-level attributes, including forbidding unsafe code and configuring documentation links for examples.

```rust
1#![warn(missing_docs)]
2// This is an HTML parser. HTML can be untrusted input from the internet.
3#![forbid(unsafe_code)]
4//
5// Relative links in the README.md don't work in rustdoc, so we have to override them.
6#![doc = concat!("[LICENSE]: ", blob_url_prefix!(), "LICENSE")]
7#![doc = concat!("[examples/tokenize_with_state_switches.rs]: ", blob_url_prefix!(), "examples/tokenize_with_state_switches.rs")]
8#![doc = concat!("[examples/custom_emitter.rs]: ", blob_url_prefix!(), "examples/custom_emitter.rs")]
9#![doc = concat!("[examples/callback_emitter.rs]: ", blob_url_prefix!(), "examples/callback_emitter.rs")]
10#![doc = concat!("[examples/scraper.rs]: ", blob_url_prefix!(), "examples/scraper.rs")]
11#![cfg_attr(doc, doc = include_str!("../README.md"))]
12//
13#![warn(clippy::all)]
14#![warn(
15    absolute_paths_not_starting_with_crate,
16    rustdoc::invalid_html_tags,
17    missing_copy_implementations,
18    missing_debug_implementations,
19    semicolon_in_expressions_from_macros,
20    unreachable_pub,
21    unused_extern_crates,
22    variant_size_differences
23)]
24#![allow(clippy::module_name_repetitions)]
25#![allow(clippy::missing_errors_doc)]
26
27macro_rules! blob_url_prefix {
28    () => {
29        concat!(
30            "https://github.com/untitaker/html5gum/blob/",
31            env!("CARGO_PKG_VERSION"),
32            "/"
33        )
34    };
35}
36
37// miraculously makes warnings disappear as blob_url_prefix is used in #![doc]
38use blob_url_prefix;
39
40mod arrayvec;
41mod char_validator;
42pub mod emitters;
43mod entities;
44mod error;
45mod htmlstring;
46mod machine;
47mod machine_helper;
48mod read_helper;
49mod reader;
50mod span;
51mod state;
52mod tokenizer;
53mod utils;
54
55#[cfg(debug_assertions)]
56#[doc(hidden)]
57pub mod testutils;
58
59pub use emitters::default::{DefaultEmitter, Doctype, EndTag, StartTag, Token};
60pub use emitters::{naive_next_state, Emitter, ForwardingEmitter};
61pub use error::Error;
62pub use htmlstring::HtmlString;
63pub use reader::{IoReader, Readable, Reader, StringReader};
64pub use span::{Span, SpanBound, Spanned};
65pub use state::State;
66pub use tokenizer::Tokenizer;
```

--------------------------------

### StartTag Struct Definition

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/default/struct.StartTag.html

The definition of the StartTag struct, which holds information about an HTML start tag.

```rust
pub struct StartTag<S: SpanBound> {
    pub self_closing: bool,
    pub name: HtmlString,
    pub attributes: BTreeMap<HtmlString, Spanned<HtmlString, S>>,
    pub span: Span<S>,
}
```

--------------------------------

### Split slices into arrays with as_chunks

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Splits a slice into fixed-size arrays from the start, returning the chunks and any remaining elements.

```rust
let slice = ['l', 'o', 'r', 'e', 'm'];
let (chunks, remainder) = slice.as_chunks();
assert_eq!(chunks, &[['l', 'o'], ['r', 'e']]);
assert_eq!(remainder, &['m']);
```

```rust
let slice = ['R', 'u', 's', 't'];
let (chunks, []) = slice.as_chunks::<2>() else {
    panic!("slice didn't have even length")
};
assert_eq!(chunks, &[['R', 'u'], ['s', 't']]);
```

--------------------------------

### Function: naive_next_state

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/fn.naive_next_state.html

Provides an educated guess for the next parsing state based on the emitted start tag name.

```APIDOC
## FUNCTION naive_next_state

### Description
Take an educated guess at the next state using the name of a just-now emitted start tag. This can be used to implement Emitter::emit_current_tag for most HTML scraping applications, but is unsuitable for implementing a browser.

### Signature
`pub fn naive_next_state(tag_name: &[u8]) -> Option<State>`

### Parameters
- **tag_name** (&[u8]) - Required - The name of the just-now emitted start tag.

### Returns
- **Option<State>** - The guessed next parsing state, or None if no state could be determined.
```

--------------------------------

### Demonstrate Borrowing with HtmlString

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/htmlstring.rs.html

Demonstrates a use case for Borrow/BorrowMut with HtmlString, specifically checking for attribute existence in a StartTag.

```rust
#[test]
fn test_borrowing() {
    use crate::StartTag;
    // demonstrate a usecase for Borrow/BorrowMut
    let tag = StartTag::<()>::default();
    assert!(!tag.attributes.contains_key(b"href".as_slice()));
}
```

--------------------------------

### Set self-closing flag for start tag

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/callback/struct.CallbackEmitter.html

Marks the current start tag as self-closing.

```rust
fn set_self_closing(&mut self)
```

--------------------------------

### Initialize Attribute Value

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/emitters/callback.rs.html

Sets the start position for an attribute value. This is typically called before pushing characters that form the attribute's value.

```rust
fn init_attribute_value(&mut self) {
        self.emitter_state.current_attribute_value_start = self.emitter_state.position;
    }
```

--------------------------------

### IoReader Initialization

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/reader.rs.html

Methods to instantiate a new IoReader with default or custom buffer configurations.

```APIDOC
## IoReader::new

### Description
Constructs a new IoReader with a default heap-allocated buffer of 16kB.

### Parameters
- **reader** (R) - Required - The reader instance implementing the Read trait.

## IoReader::new_with_buffer_size

### Description
Constructs a new IoReader with a specific internal buffer size.

### Parameters
- **reader** (R) - Required - The reader instance.
- **BUF_SIZE** (usize) - Required - The size of the internal buffer.

## IoReader::new_with_buffer

### Description
Instantiates IoReader with a custom provided buffer.

### Parameters
- **reader** (R) - Required - The reader instance.
- **buf** (&'a mut [u8]) - Required - A mutable slice to be used as the buffer.
```

--------------------------------

### Tokenizer Initialization

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/tokenizer.rs.html

Methods to instantiate a new Tokenizer with default or custom emitters.

```APIDOC
## Tokenizer::new

### Description
Creates a new tokenizer from input using the DefaultEmitter.

### Parameters
- **input** (S: Readable) - Required - The input source to tokenize.

## Tokenizer::new_with_emitter

### Description
Constructs a new tokenizer from input and a custom emitter for advanced control over token allocation.

### Parameters
- **input** (S: Readable) - Required - The input source to tokenize.
- **emitter** (E: Emitter) - Required - The custom emitter implementation.
```

--------------------------------

### Initialize Doctype

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/trait.ForwardingEmitter.html

Sets the current token to be a doctype, preparing for doctype name and identifier emission.

```rust
fn init_doctype(&mut self) { ... }
```

--------------------------------

### Doctype Examples

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/default/struct.Doctype.html

Illustrates different formats for HTML doctype declarations. These examples show how the `name`, `public_identifier`, and `system_identifier` fields are used.

```text
<!DOCTYPE {name}>
```

```text
<!DOCTYPE {name} PUBLIC '{public_identifier}'>
```

```text
<!DOCTYPE {name} SYSTEM '{system_identifier}'>
```

```text
<!DOCTYPE {name} PUBLIC '{public_identifier}' '{system_identifier}'>
```

--------------------------------

### Get Last Element of Slice

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Use `last` to get an `Option` containing a reference to the last element. Returns `None` for empty slices.

```rust
let v = [10, 40, 30];
assert_eq!(Some(&30), v.last());

let w: &[i32] = &[];
assert_eq!(None, w.last());
```

--------------------------------

### Set Last Start Tag

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/trait.ForwardingEmitter.html

Sets the name of the last start tag encountered. Primarily for testing purposes and does not override the current tag's name.

```rust
fn set_last_start_tag(&mut self, last_start_tag: Option<&[u8]>) { ... }
```

--------------------------------

### IoReader Constructor Methods

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.IoReader.html

Methods for initializing an IoReader instance with various buffer configurations.

```APIDOC
## IoReader::new

### Description
Construct a new IoReader from any type that implements Read. Defaults to a heap-allocated buffer of size 16kB.

### Parameters
- **reader** (R: Read) - Required - The input source to read from.

## IoReader::new_with_buffer_size

### Description
Construct a new IoReader with a specific internal buffer size.

### Parameters
- **reader** (R: Read) - Required - The input source to read from.
- **BUF_SIZE** (usize) - Required - The size of the internal buffer.

## IoReader::new_with_buffer

### Description
Instantiate IoReader with a custom, pre-allocated buffer.

### Parameters
- **reader** (R: Read) - Required - The input source to read from.
- **buf** (&'a mut [u8]) - Required - A mutable slice to use as the internal buffer.
```

--------------------------------

### Get Mutable Reference to Last Element of Slice

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Use `last_mut` to get an `Option` containing a mutable reference to the last element. Returns `None` for empty slices.

```rust
let x = &mut [0, 1, 2];

if let Some(last) = x.last_mut() {
    *last = 10;
}
assert_eq!(x, &[0, 1, 10]);

let y: &mut [i32] = &mut [];
assert_eq!(None, y.last_mut());
```

--------------------------------

### Initialize ReadHelper

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/read_helper.rs.html

Creates a new instance of `ReadHelper` with the provided reader.

```rust
pub(crate) fn new(reader: R) -> Self {
        ReadHelper {
            reader,
            last_character_was_cr: false,
            to_reconsume: None,
        }
    }
```

--------------------------------

### Experimental API: CloneToUninit

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Nightly-only experimental API for cloning data to an uninitialized memory location.

```APIDOC
## impl<T> CloneToUninit for T

### Description
🔬 This is a nightly-only experimental API.
Provides a way to clone data directly into uninitialized memory.

#### unsafe fn clone_to_uninit(&self, dest: *mut u8)

### Description
Performs copy-assignment from `self` to `dest`.

### Method
N/A (Trait Implementation)

### Endpoint
N/A

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
None (This is an unsafe operation that modifies memory directly)
```

--------------------------------

### Parse Unicode Character References starting with 'p' (continued)

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/entities.rs.html

This snippet continues the parsing logic for character references starting with 'p', specifically handling the 'recnapprox;' and 'recapprox;' cases.

```Rust
                    if try_read(other_chars)? {
                        return Ok(Some(CharRef {
                            name: other_chars,
                            characters,
                        }));
                    }
                }

                Ok(None)
            }

            branch_p(try_read)
        }

        'q' => {
            #[allow(non_snake_case)]
            fn branch_q<E>(
                mut try_read: impl FnMut(&str) -> Result<bool, E>,
            ) -> Result<Option<CharRef>, E> {
                for (other_chars, characters) in &[
                    ("uaternions;", "\u{210d}"),
                    ("uatint;", "\u{2a16}"),
                    ("uesteq;", "\u{225f}"),
                    ("prime;", "\u{2057}"),
                    ("uest;", "\u{3f}"),
                    ("int;", "\u{2a0c}"),
                    ("opf;", "\u{1d562}"),
                    ("scr;", "\u{1d4c6}"),
                    ("uot;", "\u{22}"),
                    ("fr;", "\u{1d52e}"),
                    ("uot", "\u{22}"),
                ] {
                    if try_read(other_chars)? {
                        return Ok(Some(CharRef {
                            name: other_chars,
                            characters,
                        }));
                    }
                }

                Ok(None)
            }

            branch_q(try_read)
        }

        'r' => {
            #[allow(non_snake_case)]
            fn branch_r<E>(
                mut try_read: impl FnMut(&str) -> Result<bool, E>,
            ) -> Result<Option<CharRef>, E> {
                for (other_chars, characters) in &[
                    ("ecnapprox;", "\u{2ab9}"),
                    ("ecapprox;", "\u{2ab7}"),
                    ("itchfork;", "\u{22d4}"),
                    ("arallel;", "\u{2225}"),
                    ("lusacir;", "\u{2a23}"),
                    ("ointint;", "\u{2a15}"),
                    ("recneqq;", "\u{2ab5}"),
                    ("recnsim;", "\u{22e8}"),
                    ("rofalar;", "\u{232e}"),
                    ("rofline;", "\u{2312}"),
                    ("rofsurf;", "\u{2313}"),
                    ("ertenk;", "\u{2031}"),
                    ("lanckh;", "\u{210e}"),
                    ("luscir;", "\u{2a22}"),
                    ("lussim;", "\u{2a26}"),
                    ("lustwo;", "\u{2a27}"),
                    ("recsim;", "\u{227e}"),
                    ("arsim;", "\u{2af3}"),
                    ("ercnt;", "\u{25}"),
                    ("eriod;", "\u{2e}"),
                    ("ermil;", "\u{2030}"),
                    ("hmmat;", "\u{2133}"),
                    ("lanck;", "\u{210f}"),
                    ("lankv;", "\u{210f}"),
                    ("lusdo;", "\u{2214}"),
                    ("lusdu;", "\u{2a25}"),
                    ("lusmn;", "\u{b1}"),
                    ("receq;", "\u{2aaf}"),
                    ("rimes;", "\u{2119}"),
                    ("rnsim;", "\u{22e8}"),
                    ("ropto;", "\u{221d}"),
                    ("rurel;", "\u{22b0}"),
                    ("uncsp;", "\u{2008}"),
                    ("arsl;", "\u{2afd}"),
                    ("hone;", "\u{260e}"),
                    ("lusb;", "\u{229e}"),
                    ("luse;", "\u{2a72}"),
                    ("lusmn", "\u{b1}"),
                    ("ound;", "\u{a3}"),
                    ("rcue;", "\u{227c}"),
                    ("rime;", "\u{2032}"),
                    ("rnap;", "\u{2ab9}"),
                    ("rsim;", "\u{227e}"),
                    ("ara;", "\u{b6}"),
                    ("art;", "\u{2202}"),
                    ("erp;", "\u{22a5}"),
                    ("hiv;", "\u{3d5}"),
                    ("lus;", "\u{2b}"),
                    ("opf;", "\u{1d561}"),
                    ("ound", "\u{a3}"),
                    ("rap;", "\u{2ab7}"),
                    ("rec;", "\u{227a}"),
                    ("rnE;", "\u{2ab5}"),
                    ("rod;", "\u{220f}"),
                    ("rop;", "\u{221d}"),
                    ("scr;", "\u{1d4c5}"),
                    ("ar;", "\u{2225}"),
                    ("ara", "\u{b6}"),
                    ("cy;", "\u{43f}"),
                    ("fr;", "\u{1d52d}"),
                    ("hi;", "\u{3c6}"),
                    ("iv;", "\u{3d6}"),
                    ("rE;", "\u{2ab3}"),
                    ("re;", "\u{2aaf}"),
                    ("si;", "\u{3c8}"),
                    ("i;", "\u{3c0}"),
                    ("m;", "\u{b1}"),
                    ("r;", "\u{227a}")
                ] {

```

--------------------------------

### Get First N Elements as Array

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Use `first_chunk` with a const generic `N` to get a reference to the first `N` elements as an array. Returns `None` if the slice is shorter than `N`.

```rust
let u = [10, 40, 30];
assert_eq!(Some(&[10, 40]), u.first_chunk::<2>());

let v: &[i32] = &[10];
assert_eq!(None, v.first_chunk::<2>());

let w: &[i32] = &[];
assert_eq!(Some(&[]), w.first_chunk::<0>());
```

--------------------------------

### Tokenize input using IoReader

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/reader.rs.html

Demonstrates how to use IoReader to tokenize input from types implementing std::io::Read.

```rust
use std::fmt::Write;
use html5gum::{Token, IoReader, Tokenizer};

let tokenizer = Tokenizer::new(IoReader::new("<title>hello world</title>".as_bytes()));
// more realistically: Tokenizer::new(File::open("index.html")?)
// long-form: Tokenizer::new(IoReader::new(File::open("index.html")?))

let mut new_html = String::new();

for token in tokenizer {
    let token = token.unwrap();

    match token {
        Token::StartTag(tag) => {
            write!(new_html, "<{}>", String::from_utf8_lossy(&tag.name)).unwrap();
        }
        Token::String(hello_world) => {
            write!(new_html, "{}", String::from_utf8_lossy(&hello_world)).unwrap();
        }
        Token::EndTag(tag) => {
            write!(new_html, "</{}>", String::from_utf8_lossy(&tag.name)).unwrap();
        }
        _ => panic!("unexpected input"),
    }

}
```

--------------------------------

### Parse HTML Entities Starting with 'e'

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/entities.rs.html

Handles HTML entities that start with 'e', mapping names like 'xponentiale;' to their Unicode characters. Requires a function to attempt reading the entity name.

```rust
fn branch_e<E>(
                mut try_read: impl FnMut(&str) -> Result<bool, E>,
            ) -> Result<Option<CharRef>, E> {
                for (other_chars, characters) in &[
                    ("xponentiale;", "\u{2147}"),
                    ("qslantless;", "\u{2a95}"),
                    ("xpectation;", "\u{2130}"),
                    ("qslantgtr;", "\u{2a96}"),
                    ("linters;", "\u{23e7}"),
                    ("mptyset;", "\u{2205}"),
                    ("qvparsl;", "\u{29e5}"),
                    ("psilon;", "\u{3b5}"),
                    ("qcolon;", "\u{2255}"),
                    ("quivDD;", "\u{2a78}"),
                    ("acute;", "\u{e9}"),
                    ("aster;", "\u{2a6e}"),
                    ("caron;", "\u{11b}"),
                    ("colon;", "\u{2255}"),
                    ("grave;", "\u{e8}"),
                ] {
                    if try_read(other_chars)? {
                        return Ok(Some(CharRef {
                            name: other_chars,
                            characters,
                        }));
                    }
                }

                Ok(None)
            }
```

--------------------------------

### Parse HTML Entities Starting with 'd'

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/entities.rs.html

Handles HTML entities that start with 'd', mapping names like 'ownharpoonright;' to their Unicode characters. Requires a function to attempt reading the entity name.

```rust
fn branch_d<E>(
                mut try_read: impl FnMut(&str) -> Result<bool, E>,
            ) -> Result<Option<CharRef>, E> {
                for (other_chars, characters) in &[
                    ("ownharpoonright;", "\u{21c2}"),
                    ("ownharpoonleft;", "\u{21c3}"),
                    ("oublebarwedge;", "\u{2306}"),
                    ("owndownarrows;", "\u{21ca}"),
                    ("ivideontimes;", "\u{22c7}"),
                    ("iamondsuit;", "\u{2666}"),
                    ("otsquare;", "\u{22a1}"),
                    ("ownarrow;", "\u{2193}"),
                    ("oteqdot;", "\u{2251}"),
                    ("otminus;", "\u{2238}"),
                    ("rbkarow;", "\u{2910}"),
                    ("zigrarr;", "\u{27ff}"),
                    ("bkarow;", "\u{290f}"),
                    ("dagger;", "\u{2021}"),
                    ("dotseq;", "\u{2a77}"),
                    ("emptyv;", "\u{29b1}"),
                    ("iamond;", "\u{22c4}"),
                    ("igamma;", "\u{3dd}"),
                    ("otplus;", "\u{2214}"),
                    ("wangle;", "\u{29a6}"),
                    ("agger;", "\u{2020}"),
                    ("aleth;", "\u{2138}"),
                    ("caron;", "\u{10f}"),
                    ("fisht;", "\u{297f}"),
                    ("ivide;", "\u{f7}"),
                    ("ivonx;", "\u{22c7}"),
                    ("lcorn;", "\u{231e}"),
                    ("lcrop;", "\u{230d}"),
                    ("ollar;", "\u{24}"),
                    ("rcorn;", "\u{231f}"),
                    ("rcrop;", "\u{230c}"),
                    ("strok;", "\u{111}"),
                    ("ashv;", "\u{22a3}"),
                    ("blac;", "\u{2dd}"),
                    ("darr;", "\u{21ca}"),
                    ("elta;", "\u{3b4}"),
                    ("harl;", "\u{21c3}"),
                    ("harr;", "\u{21c2}"),
                    ("iams;", "\u{2666}"),
                    ("isin;", "\u{22f2}"),
                    ("ivide", "\u{f7}"),
                    ("oteq;", "\u{2250}"),
                    ("tdot;", "\u{22f1}"),
                    ("trif;", "\u{25be}"),
                    ("uarr;", "\u{21f5}"),
                    ("uhar;", "\u{296f}"),
                    ("Arr;", "\u{21d3}"),
                    ("Har;", "\u{2965}"),
                    ("arr;", "\u{2193}"),
                    ("ash;", "\u{2010}"),
                    ("iam;", "\u{22c4}"),
                    ("jcy;", "\u{452}"),
                    ("opf;", "\u{1d555}"),
                    ("scr;", "\u{1d4b9}"),
                    ("scy;", "\u{455}"),
                    ("sol;", "\u{29f6}"),
                    ("tri;", "\u{25bf}"),
                    ("zcy;", "\u{45f}"),
                    ("cy;", "\u{434}"),
                    ("eg;", "\u{b0}"),
                    ("fr;", "\u{1d521}"),
                    ("ie;", "\u{a8}"),
                    ("iv;", "\u{f7}"),
                    ("ot;", "\u{2d9}"),
                    ("d;", "\u{2146}"),
                    ("eg", "\u{b0}"),
                ] {
                    if try_read(other_chars)? {
                        return Ok(Some(CharRef {
                            name: other_chars,
                            characters,
                        }));
                    }
                }

                Ok(None)
            }

            branch_d(try_read)
```

--------------------------------

### Parse HTML5 Character References starting with 'I'

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/entities.rs.html

Parses HTML5 character references starting with 'I'. It checks against a list of known entities like 'nvisibleComma', 'nvisibleTimes', and 'ntegral', returning the matched character reference or None.

```rust
                for (other_chars, characters) in &[
                    ("nvisibleComma;", "\u{2063}"),
                    ("nvisibleTimes;", "\u{2062}"),
                    ("ntersection;", "\u{22c2}"),
                    ("maginaryI;", "\u{2148}"),
                    ("ntegral;", "\u{222b}"),
                    ("mplies;", "\u{21d2}"),
                    ("acute;", "\u{cd}"),
                    ("grave;", "\u{cc}"),
                    ("tilde;", "\u{128}"),
                    ("Jlig;", "\u{132}"),
                    ("acute", "\u{cd}"),
                    ("circ;", "\u{ce}"),
                    ("grave", "\u{cc}"),
                    ("macr;", "\u{12a}"),
                    ("ogon;", "\u{12e}"),
                    ("ukcy;", "\u{406}"),
                    ("Ecy;", "\u{415}"),
                    ("Ocy;", "\u{401}"),
                    ("circ", "\u{ce}"),
                    ("dot;", "\u{130}"),
                    ("opf;", "\u{1d540}"),
                    ("ota;", "\u{399}"),
                    ("scr;", "\u{2110}"),
                    ("uml;", "\u{cf}"),
                    ("cy;", "\u{418}"),
                    ("fr;", "\u{2111}"),
                    ("nt;", "\u{222c}"),
                    ("uml", "\u{cf}"),
                    ("m;", "\u{2111}"),
                ] {
                    if try_read(other_chars)? {
                        return Ok(Some(CharRef {
                            name: other_chars,
                            characters,
                        }));
                    }
                }

                Ok(None)
```

--------------------------------

### Initialize string token

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/callback/struct.CallbackEmitter.html

Prepares the emitter to receive a sequence of plain characters as a character token.

```rust
fn init_string(&mut self)
```

--------------------------------

### Tokenizer::new

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.Tokenizer.html

Creates a new tokenizer instance from a readable input source.

```APIDOC
## pub fn new<'a, S: Readable<'a, Reader = R>>(input: S) -> Self

### Description
Create a new tokenizer from some input. The input can be &String or &str.

### Parameters
#### Request Body
- **input** (S) - Required - The input source to tokenize.
```

--------------------------------

### Get First N Elements Mutably as Array

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Use `first_chunk_mut` with a const generic `N` to get a mutable reference to the first `N` elements as an array. Returns `None` if the slice is shorter than `N`.

```rust
let x = &mut [0, 1, 2];

if let Some(first) = x.first_chunk_mut::<2>() {
    first[0] = 5;
    first[1] = 4;
}
assert_eq!(x, &[5, 4, 2]);

assert_eq!(None, x.first_chunk_mut::<4>());
```

--------------------------------

### Check for Appropriate End Tag

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/emitters/emitter.rs.html

Determines if the current token is an appropriate end tag for the last start tag. This checks if the current token is an end tag, if a last start tag exists, and if their names match.

```rust
fn current_is_appropriate_end_tag_token(&mut self) -> bool
```

--------------------------------

### Initialize Vec with set_len using FFI

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Demonstrates safe initialization of a Vec's length when used with FFI, ensuring the length does not exceed capacity. Requires `deflateGetDictionary` and `Z_OK` to be defined.

```rust
pub fn get_dictionary(&self) -> Option<Vec<u8>> {
    // Per the FFI method's docs, "32768 bytes is always enough".
    let mut dict = Vec::with_capacity(32_768);
    let mut dict_length = 0;
    // SAFETY: When `deflateGetDictionary` returns `Z_OK`, it holds that:
    // 1. `dict_length` elements were initialized.
    // 2. `dict_length` <= the capacity (32_768)
    // which makes `set_len` safe to call.
    unsafe {
        // Make the FFI call...
        let r = deflateGetDictionary(self.strm, dict.as_mut_ptr(), &mut dict_length);
        if r == Z_OK {
            // ...and update the length to what was initialized.
            dict.set_len(dict_length);
            Some(dict)
        } else {
            None
        }
    }
}
```

--------------------------------

### Define ScriptDataEscapeStart State

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/machine.rs.html

Handles the initial character after encountering '<script' or '<textarea' followed by an escape sequence. It specifically looks for a '-' character to enter the next state.

```Rust
define_state!(ScriptDataEscapeStart, slf, {
        slow_read_byte!(
            slf,
            match c {
                Some(b'-') => {
                    slf.emitter.emit_string(b"-");
                    switch_to!(slf, ScriptDataEscapeStartDash)
                }
                c => {
                    reconsume_in!(slf, c, ScriptData)
                }
            }
        )
    });
```

--------------------------------

### CallbackEmitter Usage Example

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/emitters/callback.rs.html

This example demonstrates how to use the CallbackEmitter to extract text content specifically from within `<span>` tags in an HTML string. It defines a callback function that identifies `<span>` tags and captures their inner text.

```APIDOC
## CallbackEmitter Usage Example

### Description
This example demonstrates how to use the `CallbackEmitter` to extract text content specifically from within `<span>` tags in an HTML string. It defines a callback function that identifies `<span>` tags and captures their inner text.

### Method
N/A (Illustrative Example)

### Endpoint
N/A (Illustrative Example)

### Parameters
N/A

### Request Example
```rust
use html5gum::{Span, Tokenizer};
use html5gum::emitters::callback::{CallbackEvent, CallbackEmitter};

let mut is_in_span = false;
let emitter = CallbackEmitter::new(move |event: CallbackEvent<'_>, _span: Span<()>| -> Option<Vec<u8>> {
    match event {
        CallbackEvent::OpenStartTag { name } => {
            is_in_span = name == b"span";
        },
        CallbackEvent::String { value } if is_in_span => {
            return Some(value.to_vec());
        }
        CallbackEvent::EndTag { .. } => {
            is_in_span = false;
        }
        _ => {}
    }

    None
});

let input = r"<h1><span class=hello>Hello</span> world!</h1>";
let Ok(text_fragments) = Tokenizer::new_with_emitter(input, emitter)
    .collect::<Result<Vec<_>, _>>();

assert_eq!(text_fragments, vec![b"Hello".to_vec()]);
```

### Response
N/A (Illustrative Example)
```

--------------------------------

### Initialize doctype token

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/callback/struct.CallbackEmitter.html

Sets the current token to be a new doctype token, preparing for its name and identifiers.

```rust
fn init_doctype(&mut self)
```

--------------------------------

### Parse HTML5 Character References starting with 'K'

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/entities.rs.html

Handles parsing of HTML5 character references that start with 'K'. This function iterates through a list of entities like 'cedil', 'appa', and 'Hcy', returning the matched character reference or None.

```rust
                for (other_chars, characters) in &[
                    ("cedil;", "\u{136}"),
                    ("appa;", "\u{39a}"),
                    ("Hcy;", "\u{425}"),
                    ("Jcy;", "\u{40c}"),
                    ("opf;", "\u{1d542}"),
                    ("scr;", "\u{1d4a6}"),
                    ("cy;", "\u{41a}"),
                    ("fr;", "\u{1d50e}"),
                ] {
                    if try_read(other_chars)? {
                        return Ok(Some(CharRef {
                            name: other_chars,
                            characters,
                        }));
                    }
                }

                Ok(None)
```

--------------------------------

### Get Mutable Slice of Spare Capacity in Rust Vec

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Use `spare_capacity_mut` to get a mutable slice of the vector's unused capacity. This allows direct initialization of new elements before updating the vector's length with `set_len`.

```rust
let mut v = Vec::with_capacity(10);
let uninit = v.spare_capacity_mut();
uninit[0].write(0);
uninit[1].write(1);
uninit[2].write(2);
unsafe {
    v.set_len(3);
}
assert_eq!(&v, &[0, 1, 2]);
```

--------------------------------

### Token Initialization and Emission

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/emitters/emitter.rs.html

Methods for initializing and emitting various types of HTML tokens.

```APIDOC
## POST /tokenizer/init_string

### Description
Initializes a new string token. This is typically called when the tokenizer encounters the start of a string.

### Method
POST

### Endpoint
/tokenizer/init_string

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **status** (string) - Indicates successful initialization.

#### Response Example
```json
{
  "status": "initialized"
}
```
```

```APIDOC
## POST /tokenizer/emit_string

### Description
Emits a sequence of characters as a character token. This method appends the provided byte slice to the current string token.

### Method
POST

### Endpoint
/tokenizer/emit_string

### Parameters
#### Request Body
- **c** (bytes) - Required - The byte slice representing the characters to emit.

### Request Example
```json
{
  "c": "SGVsbG8gV29ybGQ="
}
```

### Response
#### Success Response (200)
- **status** (string) - Indicates successful emission.

#### Response Example
```json
{
  "status": "emitted"
}
```
```

```APIDOC
## POST /tokenizer/init_start_tag

### Description
Initializes the tokenizer to begin processing a start tag.

### Method
POST

### Endpoint
/tokenizer/init_start_tag

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **status** (string) - Indicates successful initialization.

#### Response Example
```json
{
  "status": "initialized"
}
```
```

```APIDOC
## POST /tokenizer/init_end_tag

### Description
Initializes the tokenizer to begin processing an end tag.

### Method
POST

### Endpoint
/tokenizer/init_end_tag

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **status** (string) - Indicates successful initialization.

#### Response Example
```json
{
  "status": "initialized"
}
```
```

```APIDOC
## POST /tokenizer/init_comment

### Description
Initializes the tokenizer to begin processing a comment.

### Method
POST

### Endpoint
/tokenizer/init_comment

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **status** (string) - Indicates successful initialization.

#### Response Example
```json
{
  "status": "initialized"
}
```
```

```APIDOC
## POST /tokenizer/emit_current_tag

### Description
Emits the current token, assuming it is a tag (start or end). This method also handles appending attributes and updating the last start tag if necessary. It returns an optional `State` which may be used to switch the tokenizer's state.

### Method
POST

### Endpoint
/tokenizer/emit_current_tag

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **next_state** (Option<State>) - The next state for the tokenizer, or `None` if the state should not change.

#### Response Example
```json
{
  "next_state": "InBody"
}
```
```

```APIDOC
## POST /tokenizer/emit_current_comment

### Description
Emits the current token, assuming it is a comment. This method may panic if the current token is not a comment.

### Method
POST

### Endpoint
/tokenizer/emit_current_comment

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **status** (string) - Indicates successful emission.

#### Response Example
```json
{
  "status": "emitted"
}
```
```

```APIDOC
## POST /tokenizer/emit_current_doctype

### Description
Emits the current token, assuming it is a doctype. This method may panic if the current token is not a doctype.

### Method
POST

### Endpoint
/tokenizer/emit_current_doctype

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **status** (string) - Indicates successful emission.

#### Response Example
```json
{
  "status": "emitted"
}
```
```

--------------------------------

### Tokenizer::new_with_emitter

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/tokenizer.rs.html

Initializes a new Tokenizer with a given input string and an emitter.

```APIDOC
## Tokenizer::new_with_emitter

### Description
Creates a new tokenizer instance with the provided input string and an emitter.

### Method
Associated function (constructor-like)

### Parameters
- **input** (string) - The HTML string to tokenize.
- **emitter** (Emitter) - The emitter to use for generating tokens.

### Request Example
```rust
let emitter = Emitter::new(); // Assuming Emitter has a ::new() constructor
let tokenizer = Tokenizer::new_with_emitter("<html>", emitter);
```

### Response
- **Tokenizer** - A new instance of the Tokenizer.

### Response Example
```rust
// The tokenizer instance itself is the return value.
```
```

--------------------------------

### take

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.Tokenizer.html

Creates an iterator that yields the first `n` elements, or fewer if the underlying iterator ends sooner.

```APIDOC
## take(self, n: usize)

### Description
Creates an iterator that yields the first `n` elements, or fewer if the underlying iterator ends sooner.

### Method
`take`

### Parameters
- **n** (usize) - Description: The maximum number of elements to yield.
```

--------------------------------

### Struct Span

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.Span.html

Represents a span with start and end boundaries.

```APIDOC
## Struct Span

### Description
Position/ boundary `start..end` in the input. The position will mostly be a byte offset, but depending on the crate::Reader it originates from, it can be something entirely else.

### Fields
- **start** (B) - Start position (inclusive) of the span.
- **end** (B) - End position (exclusive) of the span.

### Trait Implementations
- **Clone**: Allows creating a duplicate of the Span.
- **Debug**: Enables formatting the Span for debugging.
- **Default**: Provides a default value for Span.
- **Hash**: Allows hashing the Span.
- **PartialEq**: Enables comparison for equality between Spans.
- **Copy**: Allows the Span to be copied implicitly.
- **Eq**: Indicates that equality is reflexive, symmetric, and transitive.

### Auto Trait Implementations
- **Freeze**, **RefUnwindSafe**, **Send**, **Sync**, **Unpin**, **UnwindSafe**
```

--------------------------------

### Tokenize HTML with Tokenizer

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.StringReader.html

Demonstrates using the Tokenizer to process an HTML string and extract tags and content.

```rust
use std::fmt::Write;
use html5gum::{Tokenizer, Token};

let html = "<title   >hello world</title>";
let mut new_html = String::new();

for Ok(token) in Tokenizer::new(html) {
    match token {
        Token::StartTag(tag) => {
            write!(new_html, "<{}>", String::from_utf8_lossy(&tag.name)).unwrap();
        }
        Token::String(hello_world) => {
            write!(new_html, "{}", String::from_utf8_lossy(&hello_world)).unwrap();
        }
        Token::EndTag(tag) => {
            write!(new_html, "</{}>", String::from_utf8_lossy(&tag.name)).unwrap();
        }
        _ => panic!("unexpected input"),
    }
}

assert_eq!(new_html, "<title>hello world</title>");
```

--------------------------------

### Get vector length

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Returns the number of elements currently in the vector.

```rust
let a = vec![1, 2, 3];
assert_eq!(a.len(), 3);
```

--------------------------------

### Enter State Macro

Source: https://docs.rs/html5gum/0.8.3/src/html5gum/machine_helper.rs.html

Enters a specified state, optionally marking it as an attribute context. Use when entering a new parsing state.

```rust
macro_rules! enter_state {
    ($slf:expr, $state:ident, $is_attribute:expr) => {{ $slf.machine_helper
            .enter_state($crate::machine_helper::state_ref!($state), $is_attribute);
        Ok(ControlToken::Continue)
    }};
}
```

--------------------------------

### pub fn rchunks

Source: https://docs.rs/html5gum/0.8.3/html5gum/struct.HtmlString.html

Returns an iterator over chunk_size elements starting from the end.

```APIDOC
## fn rchunks

### Description
Returns an iterator over `chunk_size` elements of the slice at a time, starting at the end of the slice. The chunks are slices and do not overlap.

### Parameters
#### Arguments
- **chunk_size** (usize) - The size of the chunks.

### Panics
- Panics if `chunk_size` is zero.
```

--------------------------------

### Emitter Token Management

Source: https://docs.rs/html5gum/0.8.3/html5gum/emitters/trait.ForwardingEmitter.html

Methods for emitting and initializing various HTML token types.

```APIDOC
## emit_current_tag

### Description
Emits the current token as a tag and appends the current attribute. Updates the last start tag if a start tag is emitted.

### Method
POST

### Endpoint
/emit_current_tag

## emit_current_comment

### Description
Emits the current token as a comment.

### Method
POST

### Endpoint
/emit_current_comment

## emit_current_doctype

### Description
Emits the current token as a doctype.

### Method
POST

### Endpoint
/emit_current_doctype

## init_doctype

### Description
Initializes a new doctype token with default values.

### Method
POST

### Endpoint
/init_doctype
```
Tokens:
Raw