utf8parse

https://docs.rs/utf8parse/
Admin
utf8parse is a table-driven UTF-8 parser in Rust designed to parse UTF-8 byte streams with minimal...
Tokens:5,841
Snippets:29
Trust Score:9.1
Update:4 weeks ago
Show doc for...
Tokens:
Raw
### Define State enum with UTF-8 parser states

Source: https://docs.rs/utf8parse/latest/src/utf8parse/types.rs.html

Defines the State enum with states for ground, tail bytes, and special starting bytes (E0, ED, F0, F4). The default ground state is marked with #[default].

```rust
/// States the parser can be in.
///
/// There is a state for each initial input of the 3 and 4 byte sequences since
/// the following bytes are subject to different conditions than a tail byte.
#[allow(non_camel_case_types)]
#[derive(Debug, Default, Copy, Clone, PartialEq, Eq)]
pub enum State {
    /// Ground state; expect anything
    #[default]
    Ground = 0,
    /// 3 tail bytes
    Tail3 = 1,
    /// 2 tail bytes
    Tail2 = 2,
    /// 1 tail byte
    Tail1 = 3,
    /// UTF8-3 starting with E0
    U3_2_e0 = 4,
    /// UTF8-3 starting with ED
    U3_2_ed = 5,
    /// UTF8-4 starting with F0
    Utf8_4_3_f0 = 6,
    /// UTF8-4 starting with F4
    Utf8_4_3_f4 = 7,
}
```

--------------------------------

### Parser::new

Source: https://docs.rs/utf8parse/latest/utf8parse/struct.Parser.html

Creates a new Parser instance.

```APIDOC
## Parser::new

### Description
Creates a new Parser instance.

### Method
`pub fn new() -> Parser`

### Parameters
None.

### Returns
- **Parser** - A new Parser instance.

### Example
```rust
let mut parser = Parser::new();
```
```

--------------------------------

### Implement State::advance method for UTF-8 state transition

Source: https://docs.rs/utf8parse/latest/src/utf8parse/types.rs.html

This method takes the current state and an input byte and returns the next state and an action. It handles all valid UTF-8 lead and continuation byte ranges; invalid bytes produce (State::Ground, Action::InvalidSequence). Used by the parser to process each byte.

```rust
impl State {
    /// Advance the parser state.
    ///
    /// This takes the current state and input byte into consideration, to determine the next state
    /// and any action that should be taken.
    #[inline]
    pub fn advance(self, byte: u8) -> (State, Action) {
        match self {
            State::Ground => match byte {
                0x00..=0x7f => (State::Ground, Action::EmitByte),
                0xc2..=0xdf => (State::Tail1, Action::SetByte2Top),
                0xe0 => (State::U3_2_e0, Action::SetByte3Top),
                0xe1..=0xec => (State::Tail2, Action::SetByte3Top),
                0xed => (State::U3_2_ed, Action::SetByte3Top),
                0xee..=0xef => (State::Tail2, Action::SetByte3Top),
                0xf0 => (State::Utf8_4_3_f0, Action::SetByte4),
                0xf1..=0xf3 => (State::Tail3, Action::SetByte4),
                0xf4 => (State::Utf8_4_3_f4, Action::SetByte4),
                _ => (State::Ground, Action::InvalidSequence),
            },
            State::U3_2_e0 => match byte {
                0xa0..=0xbf => (State::Tail1, Action::SetByte2),
                _ => (State::Ground, Action::InvalidSequence),
            },
            State::U3_2_ed => match byte {
                0x80..=0x9f => (State::Tail1, Action::SetByte2),
                _ => (State::Ground, Action::InvalidSequence),
            },
            State::Utf8_4_3_f0 => match byte {
                0x90..=0xbf => (State::Tail2, Action::SetByte3),
                _ => (State::Ground, Action::InvalidSequence),
            },
            State::Utf8_4_3_f4 => match byte {
                0x80..=0x8f => (State::Tail2, Action::SetByte3),
                _ => (State::Ground, Action::InvalidSequence),
            },
            State::Tail3 => match byte {
                0x80..=0xbf => (State::Tail2, Action::SetByte3),
                _ => (State::Ground, Action::InvalidSequence),
            },
            State::Tail2 => match byte {
                0x80..=0xbf => (State::Tail1, Action::SetByte2),
                _ => (State::Ground, Action::InvalidSequence),
            },
            State::Tail1 => match byte {
                0x80..=0xbf => (State::Ground, Action::SetByte1),
                _ => (State::Ground, Action::InvalidSequence),
            },
        }
    }
}
```

--------------------------------

### State::advance

Source: https://docs.rs/utf8parse/latest/src/utf8parse/types.rs.html

Advances the parser state based on the current state and an input byte, returning the next state and the action to take. This is the primary method for driving the UTF-8 parser.

```APIDOC
## State::advance

### Description
Advances the parser state given the current state and an input byte, returning the next state and the action to take. This is the core method for driving the UTF-8 parser.

### Method
`pub fn advance(self, byte: u8) -> (State, Action)`

### Parameters
- **self** (State) - The current parser state.
- **byte** (u8) - The input byte to process.

### Returns
- **(State, Action)** - A tuple containing the next state and the action to take.

### Behavior
- When in `Ground` state:
  - `0x00..=0x7f` -> `(Ground, EmitByte)`
  - `0xc2..=0xdf` -> `(Tail1, SetByte2Top)`
  - `0xe0` -> `(U3_2_e0, SetByte3Top)`
  - `0xe1..=0xec` -> `(Tail2, SetByte3Top)`
  - `0xed` -> `(U3_2_ed, SetByte3Top)`
  - `0xee..=0xef` -> `(Tail2, SetByte3Top)`
  - `0xf0` -> `(Utf8_4_3_f0, SetByte4)`
  - `0xf1..=0xf3` -> `(Tail3, SetByte4)`
  - `0xf4` -> `(Utf8_4_3_f4, SetByte4)`
  - `_` -> `(Ground, InvalidSequence)`
- When in `U3_2_e0` state:
  - `0xa0..=0xbf` -> `(Tail1, SetByte2)`
  - `_` -> `(Ground, InvalidSequence)`
- When in `U3_2_ed` state:
  - `0x80..=0x9f` -> `(Tail1, SetByte2)`
  - `_` -> `(Ground, InvalidSequence)`
- When in `Utf8_4_3_f0` state:
  - `0x90..=0xbf` -> `(Tail2, SetByte3)`
  - `_` -> `(Ground, InvalidSequence)`
- When in `Utf8_4_3_f4` state:
  - `0x80..=0x8f` -> `(Tail2, SetByte3)`
  - `_` -> `(Ground, InvalidSequence)`
- When in `Tail3` state:
  - `0x80..=0xbf` -> `(Tail2, SetByte3)`
  - `_` -> `(Ground, InvalidSequence)`
- When in `Tail2` state:
  - `0x80..=0xbf` -> `(Tail1, SetByte2)`
  - `_` -> `(Ground, InvalidSequence)`
- When in `Tail1` state:
  - `0x80..=0xbf` -> `(Ground, SetByte1)`
  - `_` -> `(Ground, InvalidSequence)`

### Example
```rust
let (next_state, action) = State::Ground.advance(0x41);
// next_state = State::Ground, action = Action::EmitByte
```
```

--------------------------------

### Parser

Source: https://docs.rs/utf8parse/latest/utf8parse

A parser for UTF-8 characters. This struct is used to parse UTF-8 encoded input and emit events for codepoints and invalid sequences.

```APIDOC
## Parser

### Description
A parser for UTF-8 characters. This struct is used to parse UTF-8 encoded input and emit events for codepoints and invalid sequences.

### Usage
Create a new Parser instance and feed it bytes to parse. The parser will invoke the Receiver trait methods for each codepoint or invalid sequence.

### Methods
- `new()` - Creates a new Parser instance.
- `advance(&mut self, input: u8, receiver: &mut R)` - Advances the parser with the next byte of input, calling the receiver's methods as needed.

### Example
```rust
use utf8parse::{Parser, Receiver};

struct MyReceiver;
impl Receiver for MyReceiver {
    fn codepoint(&mut self, c: char) {
        // handle codepoint
    }
    fn invalid_sequence(&mut self) {
        // handle invalid sequence
    }
}

let mut parser = Parser::new();
let mut receiver = MyReceiver;
parser.advance(b'a', &mut receiver);
```
```

--------------------------------

### utf8parse lib.rs source code

Source: https://docs.rs/utf8parse/latest/src/utf8parse/lib.rs.html

The complete source code of the utf8parse library, including the Parser struct, Receiver trait, and benchmark modules. It implements a table-driven UTF-8 parser with minimal branching, using a state machine and continuation byte masking.

```rust
1//! A table-driven UTF-8 Parser
2//!
3//! This module implements a table-driven UTF-8 parser which should
4//! theoretically contain the minimal number of branches (1). The only branch is
5//! on the `Action` returned from unpacking a transition.
6#![deny(clippy::all, clippy::if_not_else, clippy::enum_glob_use)]
7#![cfg_attr(all(feature = "nightly", test), feature(test))]
8#![no_std]
9
10use core::char;
11
12mod types;
13
14use types::{Action, State};
15
16/// Handles codepoint and invalid sequence events from the parser.
17pub trait Receiver {
18    /// Called whenever a codepoint is parsed successfully
19    fn codepoint(&mut self, _: char);
20
21    /// Called when an invalid_sequence is detected
22    fn invalid_sequence(&mut self);
23}
24
25/// A parser for Utf8 Characters
26///
27/// Repeatedly call `advance` with bytes to emit Utf8 characters
28#[derive(Clone, Default, PartialEq, Eq, Debug)]
29pub struct Parser {
30    point: u32,
31    state: State,
32}
33
34/// Continuation bytes are masked with this value.
35const CONTINUATION_MASK: u8 = 0b0011_1111;
36
37impl Parser {
38    /// Create a new Parser
39    pub fn new() -> Parser {
40        Parser { point: 0, state: State::Ground }
41    }
42
43    /// Advance the parser
44    ///
45    /// The provider receiver will be called whenever a codepoint is completed or an invalid
46    /// sequence is detected.
47    pub fn advance<R>(&mut self, receiver: &mut R, byte: u8)
48    where
49        R: Receiver,
50    {
51        let (state, action) = self.state.advance(byte);
52        self.perform_action(receiver, byte, action);
53        self.state = state;
54    }
55
56    fn perform_action<R>(&mut self, receiver: &mut R, byte: u8, action: Action)
57    where
58        R: Receiver,
59    {
60        match action {
61            Action::InvalidSequence => {
62                self.point = 0;
63                receiver.invalid_sequence();
64            },
65            Action::EmitByte => {
66                receiver.codepoint(byte as char);
67            },
68            Action::SetByte1 => {
69                let point = self.point | ((byte & CONTINUATION_MASK) as u32);
70                let c = unsafe { char::from_u32_unchecked(point) };
71                self.point = 0;
72
73                receiver.codepoint(c);
74            },
75            Action::SetByte2 => {
76                self.point |= ((byte & CONTINUATION_MASK) as u32) << 6;
77            },
78            Action::SetByte2Top => {
79                self.point |= ((byte & 0b0001_1111) as u32) << 6;
80            },
81            Action::SetByte3 => {
82                self.point |= ((byte & CONTINUATION_MASK) as u32) << 12;
83            },
84            Action::SetByte3Top => {
85                self.point |= ((byte & 0b0000_1111) as u32) << 12;
86            },
87            Action::SetByte4 => {
88                self.point |= ((byte & 0b0000_0111) as u32) << 18;
89            },
90        }
91    }
92}
93
94#[cfg(all(feature = "nightly", test))]
95mod benches {
96    extern crate std;
97    extern crate test;
98
    use super::{Parser, Receiver};

    use self::test::{black_box, Bencher};

    static UTF8_DEMO: &[u8] = include_bytes!("../tests/UTF-8-demo.txt");

    impl Receiver for () {
        fn codepoint(&mut self, c: char) {
            black_box(c);
        }

        fn invalid_sequence(&mut self) {}
    }

    #[bench]
    fn parse_bench_utf8_demo(b: &mut Bencher) {
        let mut parser = Parser::new();

        b.iter(|| {
            for byte in UTF8_DEMO {
                parser.advance(&mut (), *byte);
            }
        })
    }

    #[bench]
    fn std_string_parse_utf8(b: &mut Bencher) {
        b.iter(|| {
            for c in std::str::from_utf8(UTF8_DEMO).unwrap().chars() {
                black_box(c);
            }
        });
    }
}
```

--------------------------------

### Parser::advance

Source: https://docs.rs/utf8parse/latest/utf8parse/struct.Parser.html

Advances the parser with a byte, invoking the receiver when a codepoint is completed or an invalid sequence is detected.

```APIDOC
## Parser::advance

### Description
Advances the parser with a byte, invoking the receiver when a codepoint is completed or an invalid sequence is detected.

### Method
`pub fn advance<R>(&mut self, receiver: &mut R, byte: u8) where R: Receiver`

### Parameters
- **receiver** (`&mut R`) - The receiver to be called when a codepoint is completed or an invalid sequence is detected.
- **byte** (`u8`) - The byte to process.

### Returns
`()` - No return value.

### Example
```rust
let mut parser = Parser::new();
let mut receiver = MyReceiver;
parser.advance(&mut receiver, b'a');
```
```

--------------------------------

### trait Receiver

Source: https://docs.rs/utf8parse/latest/utf8parse/trait.Receiver.html

Defines the required methods for handling codepoint and invalid sequence events from the UTF-8 parser.

```APIDOC
## Trait Receiver

### Description
Handles codepoint and invalid sequence events from the parser.

### Required Methods

#### fn codepoint(&mut self, _: char)
Called whenever a codepoint is parsed successfully.

#### fn invalid_sequence(&mut self)
Called when an invalid_sequence is detected.

### Dyn Compatibility
This trait is dyn compatible.

### Implementors
(Not listed in source)
```

--------------------------------

### Define Action enum with UTF-8 parser actions

Source: https://docs.rs/utf8parse/latest/src/utf8parse/types.rs.html

Defines the Action enum with variants for invalid sequences, byte emission, and setting continuation bytes. Each variant has an implicit discriminant value (0-7) used internally.

```rust
//! Types supporting the UTF-8 parser

/// Action to take when receiving a byte
#[derive(Debug, Copy, Clone)]
pub enum Action {
    /// Unexpected byte; sequence is invalid
    InvalidSequence = 0,
    /// Received valid 7-bit ASCII byte which can be directly emitted.
    EmitByte = 1,
    /// Set the bottom continuation byte
    SetByte1 = 2,
    /// Set the 2nd-from-last continuation byte
    SetByte2 = 3,
    /// Set the 2nd-from-last byte which is part of a two byte sequence
    SetByte2Top = 4,
    /// Set the 3rd-from-last continuation byte
    SetByte3 = 5,
    /// Set the 3rd-from-last byte which is part of a three byte sequence
    SetByte3Top = 6,
    /// Set the top byte of a four byte sequence.
    SetByte4 = 7,
}
```
Tokens:
Raw