Unicode Segmentation

https://docs.rs/unicode-segmentation/latest
Admin
Provides iterators that split strings on Grapheme Cluster, Word, or Sentence boundaries, adhering to...
Tokens:112,696
Snippets:431
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### Test GraphemeCursor Chunk Start Pre-context

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=std%3A%3Avec

Ensures that CRLF sequences at chunk boundaries trigger the correct pre-context request.

```rust
#[test]
fn test_grapheme_cursor_chunk_start_require_precontext() {
    let s = "\r\n";
    let mut c = GraphemeCursor::new(1, s.len(), true);
    assert_eq!(
        c.is_boundary(&s[1..], 1),
        Err(GraphemeIncomplete::PreContext(1))
    );
    c.provide_context(&s[..1], 0);
    assert_eq!(c.is_boundary(&s[1..], 1), Ok(false));
}
```

--------------------------------

### Segment Grapheme Clusters

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/lib.rs.html

Examples of using the graphemes method to split strings into extended grapheme clusters.

```rust
# use self::unicode_segmentation::UnicodeSegmentation;
let gr1 = UnicodeSegmentation::graphemes("a\u{310}e\u{301}o\u{308}\u{332}", true)
          .collect::<Vec<&str>>();
let b: &[_] = &["a\u{310}", "e\u{301}", "o\u{308}\u{332}"];

assert_eq!(&gr1[..], b);

let gr2 = UnicodeSegmentation::graphemes("a
b🇷🇺🇸🇹", true).collect::<Vec<&str>>();
let b: &[_] = &["a", "
", "b", "🇷🇺", "🇸🇹"];

assert_eq!(&gr2[..], b);
```

--------------------------------

### Retrieve Grapheme Indices

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/lib.rs.html

Example of using the grapheme_indices method to get both the cluster and its byte offset.

```rust
# use self::unicode_segmentation::UnicodeSegmentation;
let gr_inds = UnicodeSegmentation::grapheme_indices("a̐éö̲
", true)
              .collect::<Vec<(usize, &str)>>();
let b: &[_] = &[(0, "a̐"), (3, "é"), (6, "ö̲"), (11, "
")];

assert_eq!(&gr_inds[..], b);
```

--------------------------------

### Initialize GraphemeCursor

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Creates a new cursor instance with specified offset, length, and extension mode. The offset must be on a valid codepoint boundary.

```rust
# use unicode_segmentation::GraphemeCursor;
let s = "हिन्दी";
let mut legacy = GraphemeCursor::new(0, s.len(), false);
assert_eq!(legacy.next_boundary(s, 0), Ok(Some("ह".len())));
let mut extended = GraphemeCursor::new(0, s.len(), true);
assert_eq!(extended.next_boundary(s, 0), Ok(Some("हि".len())));
```

--------------------------------

### provide_context()

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html

Provides additional pre-context to the cursor to resolve boundary ambiguities.

```APIDOC
## fn provide_context(&mut self, chunk: &str, chunk_start: usize)

### Description
Provides additional pre-context when it is needed to decide a boundary. The end of the chunk must coincide with the value given in the `GraphemeIncomplete::PreContext` error.

### Parameters
- **chunk** (&str) - The string slice containing the required pre-context.
- **chunk_start** (usize) - The starting byte offset of the provided chunk.
```

--------------------------------

### Property-Based Testing for ASCII Path

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/word.rs.html

Proptest configuration to ensure the optimized ASCII path produces identical results to the general Unicode path.

```rust
/// Strategy that yields every code-point from NUL (0) to DEL (127).
fn ascii_char() -> impl Strategy<Value = char> {
    (0u8..=127).prop_map(|b| b as char)
}

proptest! {
    #![proptest_config(ProptestConfig::with_cases(10000))]
    /// Fast path must equal general path for any ASCII input.
    #[test]
    fn proptest_ascii_matches_unicode_word_indices(
        // Vec<char> → String, length 0‒99
        s in proptest::collection::vec(ascii_char(), 0..100)
               .prop_map(|v| v.into_iter().collect::<String>())
    ) {
        let fast: Vec<(usize, &str)> = new_ascii_word_bound_indices(&s).collect();
        let uni:  Vec<(usize, &str)> = new_word_bound_indices(&s).collect();

```

--------------------------------

### Provide Pre-context for Boundary Detection

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Shows how to supply additional context to the cursor when GraphemeIncomplete::PreContext is returned, allowing the cursor to resolve boundary decisions.

```rust
# use unicode_segmentation::{GraphemeCursor, GraphemeIncomplete};
let flags = "\u{1F1F7}\u{1F1F8}\u{1F1EE}\u{1F1F4}";
let mut cursor = GraphemeCursor::new(8, flags.len(), false);
// Not enough pre-context to decide if there's a boundary between the two flags.
assert_eq!(cursor.is_boundary(&flags[8..], 8), Err(GraphemeIncomplete::PreContext(8)));
// Provide one more Regional Indicator Symbol of pre-context
cursor.provide_context(&flags[4..8], 4);
// Still not enough context to decide.
assert_eq!(cursor.is_boundary(&flags[8..], 8), Err(GraphemeIncomplete::PreContext(4)));
// Provide additional requested context.
cursor.provide_context(&flags[0..4], 0);
// That's enough to decide (it always is when context goes to the start of the string)
assert_eq!(cursor.is_boundary(&flags[8..], 8), Ok(true));
```

--------------------------------

### GraphemeCursor::new

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Creates a new GraphemeCursor instance. The offset must be on a valid codepoint boundary.

```APIDOC
## GraphemeCursor::new(offset: usize, len: usize, is_extended: bool) -> GraphemeCursor

### Description
Creates a new cursor for grapheme segmentation. The `is_extended` parameter determines whether extended grapheme clusters are used.

### Parameters
- **offset** (usize) - Required - The initial byte offset in the string.
- **len** (usize) - Required - The total length of the string in bytes.
- **is_extended** (bool) - Required - Whether to use extended grapheme cluster rules.
```

--------------------------------

### Implement State Transition Logic

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/sentence.rs.html?search=

Methods for advancing the state machine based on character categories and checking state patterns.

```rust
impl SentenceBreaksState {
        // Attempt to advance the internal state by one part
        // Whitespace and some punctutation will be collapsed
        fn next(&self, cat: SentenceCat) -> SentenceBreaksState {
            let &SentenceBreaksState(parts) = self;
            let parts = match (parts[3], cat) {
                (StatePart::ClosePlus, SentenceCat::SC_Close) => parts,
                (StatePart::SpPlus, SentenceCat::SC_Sp) => parts,
                _ => [
                    parts[1],
                    parts[2],
                    parts[3],
                    match cat {
                        SentenceCat::SC_CR => StatePart::CR,
                        SentenceCat::SC_LF => StatePart::LF,
                        SentenceCat::SC_Sep => StatePart::Sep,
                        SentenceCat::SC_ATerm => StatePart::ATerm,
                        SentenceCat::SC_Upper | SentenceCat::SC_Lower => StatePart::UpperLower,
                        SentenceCat::SC_Close => StatePart::ClosePlus,
                        SentenceCat::SC_Sp => StatePart::SpPlus,
                        SentenceCat::SC_STerm => StatePart::STerm,
                        _ => StatePart::Other,
                    },
                ],
            };
            SentenceBreaksState(parts)
        }

        fn end(&self) -> SentenceBreaksState {
            let &SentenceBreaksState(parts) = self;
            SentenceBreaksState([parts[1], parts[2], parts[3], StatePart::Eot])
        }

        // Helper function to check if state head matches a single `StatePart`
        fn match1(&self, part: StatePart) -> bool {
            let &SentenceBreaksState(parts) = self;
            part == parts[3]
        }

        // Helper function to check if first two `StateParts` in state match
        // the given two
        fn match2(&self, part1: StatePart, part2: StatePart) -> bool {
            let &SentenceBreaksState(parts) = self;
            part1 == parts[2] && part2 == parts[3]
        }
    }
```

--------------------------------

### provide_context()

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=

Provides additional pre-context required to determine a grapheme boundary.

```APIDOC
## provide_context(chunk: &str, chunk_start: usize)

### Description
Provides additional pre-context when it is needed to decide a boundary. The end of the chunk must coincide with the value given in the `GraphemeIncomplete::PreContext` request.

### Parameters
- **chunk** (&str) - The string slice containing the additional context.
- **chunk_start** (usize) - The starting byte offset of the provided chunk.
```

--------------------------------

### provide_context()

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Provides additional pre-context to the cursor when required to resolve boundary ambiguities, specifically for complex Unicode sequences.

```APIDOC
## fn provide_context(&mut self, chunk: &str, chunk_start: usize)

### Description
Supplies additional pre-context to the cursor. The end of the provided chunk must match the offset requested by a previous GraphemeIncomplete::PreContext error.

### Parameters
- **chunk** (&str) - The string slice containing the additional context.
- **chunk_start** (usize) - The starting byte offset of the provided chunk.
```

--------------------------------

### Perform String Segmentation

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/lib.rs.html?search=

Demonstrates basic usage of grapheme, word, and word boundary iterators on a string.

```rust
use unicode_segmentation::UnicodeSegmentation;

fn main() {
    let s = "a̐éö̲
";
    let g = s.graphemes(true).collect::<Vec<&str>>();
    let b: &[_] = &["a̐", "é", "ö̲", "
"];
    assert_eq!(g, b);

    let s = "The quick ("brown") fox can't jump 32.3 feet, right?";
    let w = s.unicode_words().collect::<Vec<&str>>();
    let b: &[_] = &["The", "quick", "brown", "fox", "can't", "jump", "32.3", "feet", "right"];
    assert_eq!(w, b);

    let s = "The quick ("brown")  fox";
    let w = s.split_word_bounds().collect::<Vec<&str>>();
    let b: &[_] = &["The", " ", "quick", " ", "(", "\"", "brown", "\"", ")", "  ", "fox"];
    assert_eq!(w, b);
}
```

--------------------------------

### Word Boundary Helpers

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/word.rs.html?search=

Low-level constructors for word boundaries and indices.

```rust
#[inline]
pub fn new_word_bounds(s: &str) -> UWordBounds<'_> {
    UWordBounds {
        string: s,
        cat: None,
        catb: None,
    }
}

#[inline]
pub fn new_word_bound_indices(s: &str) -> UWordBoundIndices<'_> {
    UWordBoundIndices {
        start_offset: s.as_ptr() as usize,
        iter: new_word_bounds(s),
    }
}

#[inline]
fn new_ascii_word_bound_indices(s: &str) -> AsciiWordBoundIter<'_> {
    AsciiWordBoundIter::new(s)
}
```

--------------------------------

### Find Grapheme Boundaries with GraphemeCursor

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Demonstrates finding boundaries in a full string and handling partial string chunks.

```rust
let flags = "\u{1F1F7}\u{1F1F8}\u{1F1EE}\u{1F1F4}";
let mut cursor = GraphemeCursor::new(4, flags.len(), false);
assert_eq!(cursor.next_boundary(flags, 0), Ok(Some(8)));
assert_eq!(cursor.next_boundary(flags, 0), Ok(Some(16)));
assert_eq!(cursor.next_boundary(flags, 0), Ok(None));
```

```rust
let s = "abcd";
let mut cursor = GraphemeCursor::new(0, s.len(), false);
assert_eq!(cursor.next_boundary(&s[..2], 0), Ok(Some(1)));
assert_eq!(cursor.next_boundary(&s[..2], 0), Err(GraphemeIncomplete::NextChunk));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(Some(2)));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(Some(3)));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(Some(4)));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(None));
```

--------------------------------

### Find previous boundaries with partial strings

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html

Shows how to handle partial string chunks when searching for boundaries, which may return incomplete chunk errors.

```rust
# use unicode_segmentation::{GraphemeCursor, GraphemeIncomplete};
let s = "abcd";
let mut cursor = GraphemeCursor::new(4, s.len(), false);
assert_eq!(cursor.prev_boundary(&s[2..4], 2), Ok(Some(3)));
assert_eq!(cursor.prev_boundary(&s[2..4], 2), Err(GraphemeIncomplete::PrevChunk));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(Some(2)));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(Some(1)));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(Some(0)));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(None));
```

--------------------------------

### provide_context()

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=std%3A%3Avec

Provides additional pre-context to the cursor when required to resolve boundary decisions.

```APIDOC
## fn provide_context(&mut self, chunk: &str, chunk_start: usize)

### Description
Provides additional pre-context when it is needed to decide a boundary. The end of the chunk must coincide with the value given in the `GraphemeIncomplete::PreContext` request.

### Parameters
- **chunk** (&str) - The string slice containing the additional context.
- **chunk_start** (usize) - The starting byte offset of the provided chunk.
```

--------------------------------

### GraphemeCursor Unit Tests

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

A collection of unit tests verifying boundary detection logic, including handling of RIS sequences, pre-context requirements, and emoji sequences with ZWJ.

```rust
#[test]
fn test_grapheme_cursor_ris_precontext() {
    let s = "\u{1f1fa}\u{1f1f8}\u{1f1fa}\u{1f1f8}\u{1f1fa}\u{1f1f8}";
    let mut c = GraphemeCursor::new(8, s.len(), true);
    assert_eq!(
        c.is_boundary(&s[4..], 4),
        Err(GraphemeIncomplete::PreContext(4))
    );
    c.provide_context(&s[..4], 0);
    assert_eq!(c.is_boundary(&s[4..], 4), Ok(true));
}
```

```rust
#[test]
fn test_grapheme_cursor_chunk_start_require_precontext() {
    let s = "\r\n";
    let mut c = GraphemeCursor::new(1, s.len(), true);
    assert_eq!(
        c.is_boundary(&s[1..], 1),
        Err(GraphemeIncomplete::PreContext(1))
    );
    c.provide_context(&s[..1], 0);
    assert_eq!(c.is_boundary(&s[1..], 1), Ok(false));
}
```

```rust
#[test]
fn test_grapheme_cursor_prev_boundary() {
    let s = "abcd";
    let mut c = GraphemeCursor::new(3, s.len(), true);
    assert_eq!(
        c.prev_boundary(&s[2..], 2),
        Err(GraphemeIncomplete::PrevChunk)
    );
    assert_eq!(c.prev_boundary(&s[..2], 0), Ok(Some(2)));
}
```

```rust
#[test]
fn test_grapheme_cursor_prev_boundary_chunk_start() {
    let s = "abcd";
    let mut c = GraphemeCursor::new(2, s.len(), true);
    assert_eq!(
        c.prev_boundary(&s[2..], 2),
        Err(GraphemeIncomplete::PrevChunk)
    );
    assert_eq!(c.prev_boundary(&s[..2], 0), Ok(Some(1)));
}
```

```rust
#[test]
fn test_grapheme_cursor_boundary_with_zwj_on_chunk_start() {
    use GraphemeIncomplete::*;

    let chunk0 = "👩"; // 4 bytes
    let chunk1 = "\u{200d}🔬"; // 3 bytes + 4 bytes

    let full_len = chunk0.len() + chunk1.len();

    let mut cur = GraphemeCursor::new(0, full_len, true);
    assert_eq!(cur.next_boundary(chunk0, 0), Err(NextChunk));
    match cur.next_boundary(chunk1, chunk0.len()) {
        Ok(res) => assert_eq!(res, Some(11)),
        Err(PreContext(_)) => {
            cur.provide_context(chunk0, 0);
            assert_eq!(cur.next_boundary(chunk1, chunk0.len()), Ok(Some(11)));
        }
        _ => unreachable!(),
    }
}
```

```rust
#[test]
fn test_grapheme_cursor_emoji_no_zwj() {
    use GraphemeIncomplete::*;
    let chunk0 = "🍒"; // 4 bytes
    let chunk1 = "🥑"; // 4 bytes
    let full_len = chunk0.len() + chunk1.len();

    let mut c = GraphemeCursor::new(0, full_len, true);
    assert_eq!(c.next_boundary(chunk0, 0), Err(NextChunk));
    assert_eq!(
        c.next_boundary(chunk1, chunk0.len()),
        Err(PreContext(chunk0.len()))
    );
    c.provide_context(chunk0, 0);
    assert_eq!(c.next_boundary(chunk1, chunk0.len()), Ok(Some(4)));
    assert_eq!(c.next_boundary(chunk1, chunk0.len()), Ok(Some(8)));
    assert_eq!(c.next_boundary(chunk1, chunk0.len()), Ok(None));
}
```

```rust
#[test]
fn test_grapheme_cursor_emoji_chunk_boundary_before_zwj() {
    use GraphemeIncomplete::*;
    let chunk0 = "🍒"; // 4 bytes
    let chunk1 = "\u{200d}🥑"; // 3 + 4 bytes
    let full_len = chunk0.len() + chunk1.len(); // 11

    let mut c = GraphemeCursor::new(0, full_len, true);
    assert_eq!(c.next_boundary(chunk0, 0), Err(NextChunk));
    assert_eq!(
        c.next_boundary(chunk1, chunk0.len()),
        Err(PreContext(chunk0.len()))
    );
    c.provide_context(chunk0, 0);
    assert_eq!(c.next_boundary(chunk1, chunk0.len()), Ok(Some(11)));
    assert_eq!(c.next_boundary(chunk1, chunk0.len()), Ok(None));
}
```

--------------------------------

### Create a new GraphemeCursor

Source: https://docs.rs/unicode-segmentation/latest/unicode_segmentation/struct.GraphemeCursor.html

Initializes a cursor with a specified offset and length, using the is_extended flag to toggle between legacy and extended grapheme cluster segmentation.

```rust
let s = "हिन्दी";
let mut legacy = GraphemeCursor::new(0, s.len(), false);
assert_eq!(legacy.next_boundary(s, 0), Ok(Some("ह".len())));
let mut extended = GraphemeCursor::new(0, s.len(), true);
assert_eq!(extended.next_boundary(s, 0), Ok(Some("हि".len())));
```

--------------------------------

### Access the UNICODE_VERSION constant

Source: https://docs.rs/unicode-segmentation/latest/unicode_segmentation/constant.UNICODE_VERSION.html

Represents the Unicode version as a tuple of three u64 integers.

```rust
pub const UNICODE_VERSION: (u64, u64, u64);
```

--------------------------------

### Add dependency to Cargo.toml

Source: https://docs.rs/unicode-segmentation/latest/index.html

Configuration required to include the crate in a Rust project.

```toml
[dependencies]
unicode-segmentation = "1"
```

--------------------------------

### GraphemeCursor::provide_context

Source: https://docs.rs/unicode-segmentation/latest/unicode_segmentation/struct.GraphemeCursor.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Provides additional pre-context required to determine a boundary.

```APIDOC
## GraphemeCursor::provide_context(&mut self, chunk: &str, chunk_start: usize)

### Description
Provide additional pre-context when it is needed to decide a boundary. The end of the chunk must coincide with the value given in the GraphemeIncomplete::PreContext request.
```

--------------------------------

### GraphemeCursor::provide_context

Source: https://docs.rs/unicode-segmentation/latest/unicode_segmentation/struct.GraphemeCursor.html?search=

Provides additional pre-context required to determine a boundary.

```APIDOC
## pub fn provide_context(&mut self, chunk: &str, chunk_start: usize)

### Description
Supplies additional pre-context when needed to decide a boundary. The end of the chunk must coincide with the value given in the `GraphemeIncomplete::PreContext` request.
```

--------------------------------

### Find previous grapheme boundaries

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html

Demonstrates finding previous boundaries in a full string using GraphemeCursor.

```rust
# use unicode_segmentation::GraphemeCursor;
let flags = "\u{1F1F7}\u{1F1F8}\u{1F1EE}\u{1F1F4}";
let mut cursor = GraphemeCursor::new(12, flags.len(), false);
assert_eq!(cursor.prev_boundary(flags, 0), Ok(Some(8)));
assert_eq!(cursor.prev_boundary(flags, 0), Ok(Some(0)));
assert_eq!(cursor.prev_boundary(flags, 0), Ok(None));
```

--------------------------------

### Find Previous Grapheme Boundary

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Demonstrates finding previous grapheme boundaries using GraphemeCursor, including handling of full and partial string chunks.

```rust
# use unicode_segmentation::GraphemeCursor;
let flags = "\u{1F1F7}\u{1F1F8}\u{1F1EE}\u{1F1F4}";
let mut cursor = GraphemeCursor::new(12, flags.len(), false);
assert_eq!(cursor.prev_boundary(flags, 0), Ok(Some(8)));
assert_eq!(cursor.prev_boundary(flags, 0), Ok(Some(0)));
assert_eq!(cursor.prev_boundary(flags, 0), Ok(None));
```

```rust
# use unicode_segmentation::{GraphemeCursor, GraphemeIncomplete};
let s = "abcd";
let mut cursor = GraphemeCursor::new(4, s.len(), false);
assert_eq!(cursor.prev_boundary(&s[2..4], 2), Ok(Some(3)));
assert_eq!(cursor.prev_boundary(&s[2..4], 2), Err(GraphemeIncomplete::PrevChunk));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(Some(2)));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(Some(1)));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(Some(0)));
assert_eq!(cursor.prev_boundary(&s[0..2], 0), Ok(None));
```

--------------------------------

### View remaining string in Graphemes

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Demonstrates how to inspect the remaining portion of the string during standard grapheme cluster iteration.

```rust
# use unicode_segmentation::UnicodeSegmentation;
let mut iter = "abc".graphemes(true);
assert_eq!(iter.as_str(), "abc");
iter.next();
assert_eq!(iter.as_str(), "bc");
iter.next();
iter.next();
assert_eq!(iter.as_str(), "");
```

--------------------------------

### Initialize Sentence Iterators

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/sentence.rs.html?search=u32+-%3E+bool

Functions to create iterators for sentence boundaries, indices, and filtered alphanumeric sentences.

```rust
pub fn new_sentence_breaks(source: &str) -> SentenceBreaks<'_> {
    SentenceBreaks {
        string: source,
        pos: 0,
        state: INITIAL_STATE,
    }
}
```

```rust
#[inline]
pub fn new_sentence_bounds(source: &str) -> USentenceBounds<'_> {
    USentenceBounds {
        iter: fwd::new_sentence_breaks(source),
        sentence_start: None,
    }
}
```

```rust
#[inline]
pub fn new_sentence_bound_indices(source: &str) -> USentenceBoundIndices<'_> {
    USentenceBoundIndices {
        start_offset: source.as_ptr() as usize,
        iter: new_sentence_bounds(source),
    }
}
```

```rust
#[inline]
pub fn new_unicode_sentences(s: &str) -> UnicodeSentences<'_> {
    use super::UnicodeSegmentation;
    use crate::tables::util::is_alphanumeric;

    fn has_alphanumeric(s: &&str) -> bool {
        s.chars().any(is_alphanumeric)
    }
    let has_alphanumeric: fn(&&str) -> bool = has_alphanumeric; // coerce to fn pointer

    UnicodeSentences {
        inner: s.split_sentence_bounds().filter(has_alphanumeric),
    }
}
```

--------------------------------

### ASCII Word Boundary Iterator Implementation

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/word.rs.html?search=std%3A%3Avec

Logic for forward and backward iteration over ASCII word boundaries, handling spaces, core runs, and CRLF sequences.

```rust
            let mut i = 1;
            while i < len && bytes[i] == b' ' {
                i += 1;
            }
            let word = &self.rest[..i];
            let pos = self.offset;
            self.rest = &self.rest[i..];
            self.offset += i;
            return Some((pos, word));
        }

        // 2) Core-run (letters/digits/underscore + infix)
        // Spec: ALetter × ALetter, Numeric × Numeric etc. (WB5–WB13b)
        if Self::is_core(bytes[0]) {
            let mut i = 1;
            while i < len {
                let b = bytes[i];
                if Self::is_core(b)
                    || (i + 1 < len && Self::is_infix(b, bytes[i - 1], bytes[i + 1]))
                {
                    i += 1;
                } else {
                    break;
                }
            }
            let word = &self.rest[..i];
            let pos = self.offset;
            self.rest = &self.rest[i..];
            self.offset += i;
            return Some((pos, word));
        }

        // 3) Do not break within CRLF.
        // Spec: WB3 treats CR+LF as a single non‑breaking pair.
        if bytes[0] == b'\r' && len >= 2 && bytes[1] == b'\n' {
            let word = &self.rest[..2];
            let pos = self.offset;
            self.rest = &self.rest[2..];
            self.offset += 2;
            Some((pos, word))
        } else {
            // 4) Otherwise, break everywhere
            // Spec: the catch‑all rule WB999.
            let word = &self.rest[..1];
            let pos = self.offset;
            self.rest = &self.rest[1..];
            self.offset += 1;
            Some((pos, word))
        }
    }
}

impl<'a> DoubleEndedIterator for AsciiWordBoundIter<'a> {
    fn next_back(&mut self) -> Option<(usize, &'a str)> {
        let rest = self.rest;
        if rest.is_empty() {
            return None;
        }
        let bytes = rest.as_bytes();
        let len = bytes.len();

        // 1) Group runs of spaces
        // Spec: WB3d joins adjacent *WSegSpace* into a single segment.
        if bytes[len - 1] == b' ' {
            // find start of this last run of spaces
            let mut start = len - 1;
            while start > 0 && bytes[start - 1] == b' ' {
                start -= 1;
            }
            let word = &rest[start..];
            let pos = self.offset + start;
            self.rest = &rest[..start];
            return Some((pos, word));
        }

        // 2) Trailing Core-run (letters/digits/underscore + infix)
        // Spec: ALetter × ALetter, Numeric × Numeric etc. (WB5–WB13b)
        if Self::is_core(bytes[len - 1]) {
            // scan backwards as long as we see `is_core` or an `is_infix`
            let mut start = len - 1;
            while start > 0 {
                let b = bytes[start - 1];
                let prev = if start >= 2 { bytes[start - 2] } else { b };
                let next = bytes[start]; // the byte we just included
                if Self::is_core(b) || Self::is_infix(b, prev, next) {
                    start -= 1;
                } else {
                    break;
                }
            }
            let word = &rest[start..];
            let pos = self.offset + start;
            self.rest = &rest[..start];
            return Some((pos, word));
        }

        // 3) Non-core: CR+LF as one token, otherwise single char
        // Spec: WB3 treats CR+LF as a single non‑breaking pair.
        if len >= 2 && bytes[len - 2] == b'\r' && bytes[len - 1] == b'\n' {
            let start = len - 2;
            let word = &rest[start..];
            let pos = self.offset + start;
            self.rest = &rest[..start];
            return Some((pos, word));
        }

        // 4) Fallback – every other byte is its own segment
        // Spec: the catch‑all rule WB999.
        let start = len - 1;
        let word = &rest[start..];
        let pos = self.offset + start;
        self.rest = &rest[..start];
        Some((pos, word))
    }
}
```

--------------------------------

### Test GraphemeCursor with ZWJ Across Multiple Chunks

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=std%3A%3Avec

Verifies that the cursor correctly handles ZWJ sequences split across three distinct chunks.

```rust
#[test]
fn test_grapheme_cursor_emoji_zwj_across_chunks() {
    use GraphemeIncomplete::*;
    let chunk0 = "🍒"; // 4 bytes
    let chunk1 = "\u{200d}"; // 3 bytes
    let chunk2 = "🥑"; // 4 bytes
    let full_len = chunk0.len() + chunk1.len() + chunk2.len(); // 11
    let chunk2_start = chunk0.len() + chunk1.len();

    let mut c = GraphemeCursor::new(0, full_len, true);
    assert_eq!(c.next_boundary(chunk0, 0), Err(NextChunk));
    assert_eq!(c.next_boundary(chunk1, chunk0.len()), Err(NextChunk));
    assert_eq!(
        c.next_boundary(chunk2, chunk2_start),
        Err(PreContext(chunk2_start))
    );
    c.provide_context(chunk1, chunk0.len());
    assert_eq!(
        c.next_boundary(chunk2, chunk2_start),
        Err(PreContext(chunk0.len()))
    );
    c.provide_context(chunk0, 0);
    assert_eq!(c.next_boundary(chunk2, chunk2_start), Ok(Some(11)));
    assert_eq!(c.next_boundary(chunk2, chunk2_start), Ok(None));
}
```

--------------------------------

### Word Segmentation State Machine Transitions

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/word.rs.html?search=u32+-%3E+bool

Handles character category transitions and rule-based state updates for word boundary detection.

```rust
            if prev_zwj && is_emoji(ch) {
                state = Emoji;
                continue;
            }
            // Don't use `continue` in this match without updating `cat`
            state = match state {
                Start if cat == wd::WC_CR => {
                    idx += match self.get_next_cat(idx) {
                        Some(wd::WC_LF) => 1, // rule WB3
                        _ => 0,
                    };
                    break; // rule WB3a
                }
                Start => match cat {
                    wd::WC_ALetter => Letter,            // rule WB5, WB6, WB9, WB13a
                    wd::WC_Hebrew_Letter => HLetter,     // rule WB5, WB6, WB7a, WB7b, WB9, WB13a
                    wd::WC_Numeric => Numeric,           // rule WB8, WB10, WB12, WB13a
                    wd::WC_Katakana => Katakana,         // rule WB13, WB13a
                    wd::WC_ExtendNumLet => ExtendNumLet, // rule WB13a, WB13b
                    wd::WC_Regional_Indicator => Regional(RegionalState::Half), // rule WB13c
                    wd::WC_LF | wd::WC_Newline => break, // rule WB3a
                    wd::WC_ZWJ => Zwj,                   // rule WB3c
                    wd::WC_WSegSpace => WSegSpace,       // rule WB3d
                    _ => {
                        if let Some(ncat) = self.get_next_cat(idx) {
                            // rule WB4
                            if ncat == wd::WC_Format || ncat == wd::WC_Extend || ncat == wd::WC_ZWJ
                            {
                                state = FormatExtend(AcceptNone);
                                self.cat = Some(ncat);
                                continue;
                            }
                        }
                        break; // rule WB999
                    }
                },
                WSegSpace => match cat {
                    wd::WC_WSegSpace if !skipped_format_extend => WSegSpace,
                    _ => {
                        take_curr = false;
                        break;
                    }
                },
                Zwj => {
                    // We already handle WB3c above.
                    take_curr = false;
                    break;
                }
                Letter | HLetter => match cat {
                    wd::WC_ALetter => Letter,            // rule WB5
                    wd::WC_Hebrew_Letter => HLetter,     // rule WB5
                    wd::WC_Numeric => Numeric,           // rule WB9
                    wd::WC_ExtendNumLet => ExtendNumLet, // rule WB13a
                    wd::WC_Double_Quote if state == HLetter => {
                        savecat = cat;
                        saveidx = idx;
                        FormatExtend(RequireHLetter) // rule WB7b
                    }
                    wd::WC_Single_Quote if state == HLetter => {
                        FormatExtend(AcceptQLetter) // rule WB7a
                    }
                    wd::WC_MidLetter | wd::WC_MidNumLet | wd::WC_Single_Quote => {
                        savecat = cat;
                        saveidx = idx;
                        FormatExtend(RequireLetter) // rule WB6
                    }
                    _ => {
                        take_curr = false;
                        break;
                    }
                },
                Numeric => match cat {
                    wd::WC_Numeric => Numeric,           // rule WB8
                    wd::WC_ALetter => Letter,            // rule WB10
                    wd::WC_Hebrew_Letter => HLetter,     // rule WB10
                    wd::WC_ExtendNumLet => ExtendNumLet, // rule WB13a
                    wd::WC_MidNum | wd::WC_MidNumLet | wd::WC_Single_Quote => {
                        savecat = cat;
                        saveidx = idx;
                        FormatExtend(RequireNumeric) // rule WB12
                    }
                    _ => {
                        take_curr = false;
                        break;
                    }
                },
                Katakana => match cat {
                    wd::WC_Katakana => Katakana,         // rule WB13
                    wd::WC_ExtendNumLet => ExtendNumLet, // rule WB13a
                    _ => {

```

--------------------------------

### GraphemeCursor::new

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html

Creates a new GraphemeCursor instance. The cursor tracks the current offset and length of the string, and whether to use extended grapheme cluster rules.

```APIDOC
## GraphemeCursor::new(offset: usize, len: usize, is_extended: bool) -> GraphemeCursor

### Description
Initializes a new cursor for grapheme segmentation. The `offset` must be on a valid codepoint boundary.

### Parameters
- **offset** (usize) - The starting byte offset.
- **len** (usize) - The total length of the string in bytes.
- **is_extended** (bool) - Whether to use extended grapheme cluster rules.

### Example
```rust
let s = "हिन्दी";
let mut extended = GraphemeCursor::new(0, s.len(), true);
```
```

--------------------------------

### View remaining string in GraphemeIndices

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=u32+-%3E+bool

Demonstrates how to inspect the remaining portion of the string during iteration of grapheme indices.

```rust
# use unicode_segmentation::UnicodeSegmentation;
let mut iter = "abc".grapheme_indices(true);
assert_eq!(iter.as_str(), "abc");
iter.next();
assert_eq!(iter.as_str(), "bc");
iter.next();
iter.next();
assert_eq!(iter.as_str(), "");
```

--------------------------------

### Find next grapheme boundary

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/grapheme.rs.html?search=

Demonstrates finding boundaries in a full string and handling partial string chunks.

```rust
# use unicode_segmentation::GraphemeCursor;
let flags = "\u{1F1F7}\u{1F1F8}\u{1F1EE}\u{1F1F4}";
let mut cursor = GraphemeCursor::new(4, flags.len(), false);
assert_eq!(cursor.next_boundary(flags, 0), Ok(Some(8)));
assert_eq!(cursor.next_boundary(flags, 0), Ok(Some(16)));
assert_eq!(cursor.next_boundary(flags, 0), Ok(None));
```

```rust
# use unicode_segmentation::{GraphemeCursor, GraphemeIncomplete};
let s = "abcd";
let mut cursor = GraphemeCursor::new(0, s.len(), false);
assert_eq!(cursor.next_boundary(&s[..2], 0), Ok(Some(1)));
assert_eq!(cursor.next_boundary(&s[..2], 0), Err(GraphemeIncomplete::NextChunk));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(Some(2)));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(Some(3)));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(Some(4)));
assert_eq!(cursor.next_boundary(&s[2..4], 2), Ok(None));
```

--------------------------------

### Test ASCII word boundary reverse iteration matching

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/word.rs.html

Verifies that the fast-path ASCII word boundary index iterator produces identical results to the general Unicode iterator when reversed.

```rust
        /// Fast path must equal general path for any ASCII input, forwards and backwards.
        #[test]
        fn proptest_ascii_matches_unicode_word_indices_rev(
            // Vec<char> → String, length 0‒99
            s in proptest::collection::vec(ascii_char(), 0..100)
                   .prop_map(|v| v.into_iter().collect::<String>())
        ) {
            let fast_rev: Vec<(usize, &str)> = new_ascii_word_bound_indices(&s).rev().collect();
            let uni_rev : Vec<(usize, &str)> = new_word_bound_indices(&s).rev().collect();
            prop_assert_eq!(fast_rev, uni_rev);
        }
```

--------------------------------

### Word Segmentation Tests

Source: https://docs.rs/unicode-segmentation/latest/src/unicode_segmentation/word.rs.html

Unit tests verifying word category classification and boundary index logic for various string inputs.

```rust
#[test]
fn test_syriac_abbr_mark() {
    use crate::tables::word as wd;
    let (_, _, cat) = wd::word_category('\u{70f}');
    assert_eq!(cat, wd::WC_ALetter);
}

#[test]
fn test_end_of_ayah_cat() {
    use crate::tables::word as wd;
    let (_, _, cat) = wd::word_category('\u{6dd}');
    assert_eq!(cat, wd::WC_Numeric);
}

#[test]
fn test_ascii_word_bound_indices_various_cases() {
    let s = "Hello, world!";
    let words: Vec<(usize, &str)> = new_ascii_word_bound_indices(s).collect();
    let expected = vec![
        (0, "Hello"), // simple letters
        (5, ","),
        (6, " "),     // space after comma
        (7, "world"), // skip comma+space, stop at '!'
        (12, "!"),    // punctuation at the end
    ];
    assert_eq!(words, expected);
}

#[test]
fn test_ascii_word_indices_various_cases() {
    let s = "Hello, world! can't e.g. var1 123,456 foo_bar example.com 127.0.0.1:9090";
    let words: Vec<&str> = new_unicode_words_ascii(s).collect();
    let expected = vec![
        ("Hello"), // simple letters
        ("world"), // skip comma+space, stop at '!'
        ("can't"), // apostrophe joins letters
        ("e.g"),
        ("var1"),
        ("123,456"), // digits+comma+digits
        ("foo_bar"),
        ("example.com"),
        ("127.0.0.1"),
        ("9090"), // port number
    ];
    assert_eq!(words, expected);
}
```
Tokens:
Raw