Charabia

https://docs.rs/charabia/0.9.9
Admin
Charabia is a Rust library that tokenizes text by detecting scripts and languages, while performing...
Tokens:138,869
Snippets:1,187
Trust Score:9.1
Update:5 months ago
Show doc for...
Tokens:
Raw
### Token Classification Examples

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/classify.rs.html

Illustrates the creation of various `Token` instances with different lemmas and kinds, including separators and words. These examples are used in tests to verify normalizer behavior.

```rust
Token {
                lemma: Cow::Borrowed("\""),
                kind: TokenKind::Separator(SeparatorKind::Soft),
                ..Default::default()
            },
```

```rust
Token {
                lemma: Cow::Borrowed("@"),
                kind: TokenKind::Separator(SeparatorKind::Soft),
                ..Default::default()
            },
```

```rust
Token {
                lemma: Cow::Borrowed("."),
                kind: TokenKind::Separator(SeparatorKind::Soft),
                ..Default::default()
            },
```

```rust
Token {
                lemma: Cow::Borrowed(". "),
                kind: TokenKind::Separator(SeparatorKind::Hard),
                ..Default::default()
            },
```

```rust
Token {
                lemma: Cow::Borrowed("。 সম্প"),
                kind: TokenKind::Separator(SeparatorKind::Hard),
                ..Default::default()
            },
```

```rust
Token { lemma: Cow::Borrowed("S.O.S"), kind: TokenKind::Word, ..Default::default() }
```

```rust
Token { lemma: Cow::Borrowed("ь"), kind: TokenKind::Word, ..Default::default() }
```

--------------------------------

### Persian Tokenization Example

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/persian.rs.html

Demonstrates tokenization of a Persian sentence, including character mapping for normalization.

```rust
vec![
            Token {
                lemma: Owned("علی".to_string()),
                char_end: 3,
                byte_end: 6, // 3 chars * 2 bytes
                script: Script::Arabic,
                language: Some(Language::Pes),
                char_map: Some(vec![(2, 2), (2, 2), (2, 2)]),
                ..Default::default()
            },
            Token {
                lemma: Owned("کتاب".to_string()),
                char_end: 4,
                byte_end: 8, // 4 chars * 2 bytes
                script: Script::Arabic,
                language: Some(Language::Pes),
                char_map: Some(vec![(2, 2), (2, 2), (2, 2), (2, 2)]),
                ..Default::default()
            },
            Token {
                lemma: Owned("1234567890".to_string()),
                char_end: 10,
                byte_end: 10, // Corrected from 20 to 10 (10 ASCII digits * 1 byte)
                script: Script::Arabic,
                language: Some(Language::Pes),
                char_map: Some(vec![(2, 1); 10]),
                ..Default::default()
            },
            Token {
                lemma: Owned("کیک 123 یک کتاب".to_string()),
                char_end: 13,
                byte_end: 24,
                script: Script::Arabic,
                language: Some(Language::Pes),
                char_map: Some(vec![
                    (2, 2),
                    (2, 2),
                    (2, 2), // کیک
                    (1, 1), // space
                    (2, 1),
                    (2, 1),
                    (2, 1), // ۱۲۳ (Persian digits, normalized to ASCII)
                    (1, 1), // space
                    (2, 2),
                    (2, 2), // یک
                    (1, 1), // space
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2), // کتاب
                ]),
                ..Default::default()
            },
            Token {
                lemma: Owned("سلام".to_string()),
                char_end: 4,
                byte_end: 8, // 4 chars * 2 bytes
                script: Script::Arabic,
                language: Some(Language::Pes),
                ..Default::default()
            },
            Token {
                lemma: Owned("خانه".to_string()),
                char_end: 4,
                byte_end: 8, // 4 chars * 2 bytes
                script: Script::Arabic,
                language: Some(Language::Pes),
                ..Default::default()
            },
            Token {
                lemma: Owned("گژ پژ".to_string()),
                char_end: 5,
                byte_end: 10, // 5 chars * 2 bytes
                script: Script::Arabic,
                language: Some(Language::Pes),
                ..Default::default()
            },
            Token {
                lemma: Owned("قنات قصبه شهر گناباد عمیقترین و قدیمیترین کاریز جهان است.".to_string()),
                char_end: 56,
                byte_end: 112,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                char_map: Some(vec![
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (1, 1),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (1, 1),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (1, 1),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (1, 1),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (3, 0),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (1, 1),
                    (2, 2),
                ]),
                ..Default::default()
            },
        ]
    }
```

--------------------------------

### Token Normalization Example

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/mod.rs.html

Demonstrates the application of multiple normalizers (CompatibilityDecomposition, Lowercase, Quote) to a token with specific options. The process involves initializing a token, applying each normalizer sequentially, and displaying the token's state after each step.

```rust
let options = crate::normalizer::NormalizerOption {
            create_char_map: true,
            lossy: true,
            ..Default::default()
        };

        let string = "0÷IÖꞪz";
        let mut normalized = Token { lemma: Cow::Borrowed(string), ..Default::default() };
        display_token::<()>(&normalized);
        normalized = CompatibilityDecompositionNormalizer.normalize(normalized, &options);
        display_token::<CompatibilityDecompositionNormalizer>(&normalized);
        normalized = LowercaseNormalizer.normalize(normalized, &options);
        display_token::<LowercaseNormalizer>(&normalized);
        normalized = QuoteNormalizer.normalize(normalized, &options);
        display_token::<QuoteNormalizer>(&normalized);
        let _ = normalized;
```

--------------------------------

### Arabic Normalizer Test Setup

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/arabic.rs.html

This macro sets up a test for the Arabic normalizer, comparing the raw tokens with the expected normalized tokens.

```rust
test_normalizer!(ArabicNormalizer, tokens(), normalizer_result(), normalized_tokens());
```

--------------------------------

### Create TokenizerBuilder with Default Settings

Source: https://docs.rs/charabia/0.9.9/charabia/struct.TokenizerBuilder.html

Initializes a TokenizerBuilder using default settings. This is a convenient starting point if stop words are not immediately required.

```rust
use charabia::TokenizerBuilder;

// create the builder.
let mut builder = TokenizerBuilder::default();
```

--------------------------------

### Tokenization Example with Control Characters

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/control_char.rs.html

Illustrates the tokenization process, showing how words are represented as tokens with associated character maps and script information. This is part of the control character normalization tests.

```rust
Token {
                lemma: Owned("生而自由oo".to_string()),
                char_end: 9,
                byte_end: 17,
                script: Script::Cj,
                char_map: Some(vec![
                    (1, 0),
                    (3, 3),
                    (3, 3),
                    (3, 3),
                    (3, 3),
                    (1, 0),
                    (1, 1),
                    (1, 1),
                    (1, 0),
                ]),
                kind: TokenKind::Word,
                ..Default::default()
            }
```

--------------------------------

### ChineseNormalizer Test Setup

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/chinese.rs.html

Provides base tokens for testing the ChineseNormalizer. These tokens represent Chinese text with different characteristics to verify normalization.

```rust
use std::borrow::Cow::Owned;

use crate::normalizer::test::test_normalizer;
use crate::normalizer::{Normalizer, NormalizerOption};
use crate::token::TokenKind;

// base tokens to normalize.
fn tokens() -> Vec<Token<'static>> {
    vec![
        Token {
            lemma: Owned("尊嚴".to_string()),
            char_end: 2,
            byte_end: 6,
            script: Script::Cj,
            language: Some(Language::Cmn),
            ..Default::default()
        },
        Token {
            lemma: Owned("生而自由".to_string()),
            char_end: 4,
            byte_end: 12,
            script: Script::Cj,
            language: Some(Language::Cmn),
            ..Default::default()
        },
        Token {
            lemma: Owned("澚䀾亚㮺刄杤".to_string()),
            char_end: 5,
            byte_end: 15,
            script: Script::Cj,
            language: Some(Language::Zho),
            ..Default::default()
        },
    ]
}
```

--------------------------------

### Default Implementation for NormalizerOption

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.NormalizerOption.html

Returns the default value for NormalizerOption, providing a sensible starting configuration.

```rust
fn default() -> NormalizerOption<'tb>
```

--------------------------------

### Turkish Normalizer Test Setup

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/turkish.rs.html

Provides a set of tokens for testing the TurkishNormalizer. This includes Turkish alphabet characters and texts containing or not containing 'ı', to verify normalization logic.

```rust
fn tokens() -> Vec<Token<'static>> {
    vec![
        // Turkish alphabet
        Token {
            lemma: Owned("ABCÇDEFGĞHIİJKLMNOÖPRSŞTUÜVYZ".to_string()),
            char_end: 29,
            byte_end: 35,
            script: Script::Latin,
            ..Default::default()
        },
        Token {
            lemma: Owned("abcçdefgğhıijklmnoöprsştuüvyz".to_string()),
            char_end: 29,
            byte_end: 35,
            script: Script::Latin,
            ..Default::default()
        },
        // Turkish texts containing 'ı'
        Token {
            lemma: Owned("çalışma".to_string()),
            char_end: 7,
            byte_end: 10,
            script: Script::Latin,
            ..Default::default()
        },
        Token {
            lemma: Owned("şarkı".to_string()),
            char_end: 5,
            byte_end: 7,
            script: Script::Latin,
            ..Default::default()
        },
        Token {
            lemma: Owned("ışık".to_string()),
            char_end: 4,
            byte_end: 7,
            script: Script::Latin,
            ..Default::default()
        },
        // Turkish texts without 'ı'
        // - verify that the complete pipeline normalizes turkish text as expected
        Token {
            lemma: Owned("günlük".to_string()),
            char_end: 6,
            byte_end: 8,
            script: Script::Latin,
            ..Default::default()
        },
        Token {
            lemma: Owned("İstanbul".to_string()),
            char_end: 8,
            byte_end: 9,
            script: Script::Latin,
            ..Default::default()
        },
        Token {
            lemma: Owned("İstasyon".to_string()),
            char_end: 8,
            byte_end: 9,
            script: Script::Latin,
            ..Default::default()
        },
        Token {
            lemma: Owned("ömür".to_string()),
            char_end: 4,
            byte_end: 6,
            script: Script::Latin,
            ..Default::default()
        },
        Token {
            lemma: Owned("ütü".to_string()),
            char_end: 3,
            byte_end: 5,
            script: Script::Latin,
            ..Default::default()
        },
    ]
}
```

--------------------------------

### Test Japanese Tokenization with IPADIC and Transliteration

Source: https://docs.rs/charabia/0.9.9/src/charabia/segmenter/japanese.rs.html

Tests the tokenization of Japanese text, specifically handling the word 'トートバッグ' with conditional transliteration. This example shows the expected output when the 'japanese-transliteration' feature is enabled versus when it is not.

```rust
const TOKENIZED: &[&str] = if cfg!(feature = "japanese-segmentation-ipadic") {
    &[
        "関西",
        "国際",
        "空港",
        "限定",
        // Use "とうとばっぐ" instead when feature "japanese-transliteration" is enabled or become default
        #[cfg(feature = "japanese-transliteration")]
        "とうとは\u{3099}っく\u{3099}",
        #[cfg(not(feature = "japanese-transliteration"))]
        "トートハ\u{3099}ック\u{3099}",
        " ",
        "すもも",
        "も",
        "もも",
        "も",
        "もも",
        "の",
        "うち",
        " ",
        "123",
        " ",
        "456",
    ]
} else if cfg!(feature = "japanese-segmentation-unidic") {
    &[
        "関西",
        "国際",
        "空港",
        "限定",
        // Use "とうとばっぐ" instead when feature "japanese-transliteration" is enabled or become default
        #[cfg(feature = "japanese-transliteration")]
        "とうと",
        #[cfg(not(feature = "japanese-transliteration"))]
        "トート",
        #[cfg(feature = "japanese-transliteration")]
        "は\u{3099}っく\u{3099}",
        #[cfg(not(feature = "japanese-transliteration"))]
        "ハ\u{3099}ック\u{3099}",
        " ",
        "すもも",
        "も",
        "もも",
        "も",
        "もも",
        "の",
        "うち",
        " ",
        "123",
        " ",
        "456",
    ]
} else {
    &[]
};
```

--------------------------------

### Get Length of Next Character

Source: https://docs.rs/charabia/0.9.9/src/charabia/segmenter/utils.rs.html

Calculates and returns the UTF-8 byte length of the next character in the string starting from the current offset.

```rust
fn next_character_length(&self) -> usize {
        self.to_segment[self.offset..].chars().next().unwrap().len_utf8()
    }
```

--------------------------------

### TokenizerBuilder::type_id

Source: https://docs.rs/charabia/0.9.9/charabia/struct.TokenizerBuilder.html

Gets the TypeId of the TokenizerBuilder instance.

```APIDOC
## TokenizerBuilder::type_id

### Description
Gets the `TypeId` of `self`.

### Signature
```rust
fn type_id(&self) -> TypeId
```
```

--------------------------------

### Configure Tokenizer Separators

Source: https://docs.rs/charabia/0.9.9/src/charabia/tokenizer.rs.html

Shows how to initialize a TokenizerBuilder with default settings, define custom separators, and then build and use the tokenizer to segment a string.

```rust
use charabia::TokenizerBuilder;

// create the builder.
let mut builder = TokenizerBuilder::default();

// create a custom list of separators.
let separators = [" ", ", ", ". ", "?", "!"];

// configurate separators.
builder.separators(&separators);

// build the tokenizer passing the text to tokenize.
let tokenizer = builder.build();

// text to tokenize.
let orig = "The quick (\"brown\") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

let output: Vec<_> = tokenizer.segment_str(orig).collect();
assert_eq!(
  &output,
  &["The", " ", "quick", " ", "(\"brown\")", " ", "fox", " ", "can't", " ", "jump", " ", "32.3", " ", "feet", ", ", "right", "?", " ", "Brr", ", ", "it's", " ", "29.3°F", "!"]
);

```

--------------------------------

### Build Tokenizer with Stop Words

Source: https://docs.rs/charabia/0.9.9/charabia/struct.TokenizerBuilder.html

Demonstrates how to create a TokenizerBuilder, configure it with a set of stop words, and then build the tokenizer. This is useful for excluding common words from the tokenized output.

```rust
use fst::Set;

use charabia::TokenizerBuilder;

// text to tokenize.
let orig = "The quick (\"brown\") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

// create the builder.
let mut builder = TokenizerBuilder::new();

// create a set of stop words.
let stop_words: Set<Vec<u8>> = Set::from_iter(["the"].iter()).unwrap();

// configurate stop words.
builder.stop_words(&stop_words);

// build the tokenizer passing the text to tokenize.
let tokenizer = builder.build();
```

--------------------------------

### impl<T> Any for T

Source: https://docs.rs/charabia/0.9.9/charabia/struct.ReconstructedTokenIter.html

Provides the `type_id` method to get the `TypeId` of `self`.

```APIDOC
## fn type_id(&self) -> TypeId

### Description
Gets the `TypeId` of `self`.

### Returns
A `TypeId` representing the dynamic type of `self`.
```

--------------------------------

### Get Type ID of Classifier

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.Classifier.html

Retrieves the TypeId of the Classifier instance. This is part of the Any trait implementation.

```rust
fn type_id(&self) -> TypeId
```

--------------------------------

### Build Tokenizer with Custom Stop Words

Source: https://docs.rs/charabia/0.9.9/src/charabia/tokenizer.rs.html

Demonstrates how to create a TokenizerBuilder, configure it with custom stop words using an FST Set, and then build the tokenizer.

```rust
use fst::Set;

use charabia::TokenizerBuilder;

// text to tokenize.
let orig = "The quick (\"brown\") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

// create the builder.
let mut builder = TokenizerBuilder::new();

// create a set of stop words.
let stop_words: Set<Vec<u8>> = Set::from_iter(["the"].iter()).unwrap();

// configurate stop words.
builder.stop_words(&stop_words);

// build the tokenizer passing the text to tokenize.
let tokenizer = builder.build();

```

--------------------------------

### Get Last Element from SegmentedTokenIter

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Consumes the iterator and returns the last element, if any. This is available since 1.0.0.

```rust
fn last(self) -> Option<Self::Item>
where Self: Sized,
```

--------------------------------

### TokenizerBuilder::build

Source: https://docs.rs/charabia/0.9.9/charabia/struct.TokenizerBuilder.html

Builds and returns the configured Tokenizer instance.

```APIDOC
## TokenizerBuilder::build

### Description
Build the configurated `Tokenizer`.

### Returns
The configured `Tokenizer` instance.
```

--------------------------------

### Get Normalized Lemma

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Returns a reference to the normalized lemma of the token. This is the processed string representation of the token.

```rust
pub fn lemma(&self) -> &str
```

--------------------------------

### fold_results

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Folds `Result` values from an iterator using a starting value and a folding function. This method is deprecated.

```APIDOC
## fn fold_results<A, E, B, F>(&mut self, start: B, f: F) -> Result<B, E>

### Description
Fold `Result` values from an iterator.

### Deprecated
Deprecated since 0.10.0: Use `.fold_ok()` instead.

### Method
`fold_results`

### Parameters
- `start` (B): The initial value for the accumulator.
- `f` (F): The folding function.

### Returns
`Result<B, E>` - The final accumulated value or the first error encountered.
```

--------------------------------

### Configure Tokenizer with Words Dictionary

Source: https://docs.rs/charabia/0.9.9/charabia/struct.TokenizerBuilder.html

Illustrates how to provide a custom dictionary of words that should be segmented before other rules. This is useful for handling specific multi-word phrases or abbreviations.

```rust
use charabia::TokenizerBuilder;

// create the builder.
let mut builder = TokenizerBuilder::default();

// create a custom list of words.
let words = ["J. R. R.", "Dr.", "J. K."];

// configurate words.
builder.words_dict(&words);

// build the tokenizer passing the text to tokenize.
let tokenizer = builder.build();

// text to tokenize.
let orig = "J. R. R. Tolkien. J. K. Rowling. Dr. Seuss";

let output: Vec<_> = tokenizer.segment_str(orig).collect();
assert_eq!(
  &output,
  &["J. R. R.", " ", "Tolkien", ". ", "J. K.", " ", "Rowling", ". ", "Dr.", " ", "Seuss"]
);
```

--------------------------------

### positions

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.NormalizedTokenIter.html

Returns an iterator adaptor that yields the indices of all elements satisfying a predicate, counted from the start of the iterator.

```APIDOC
## positions

### Description
Returns an iterator adaptor that yields the indices of all elements satisfying a predicate, counted from the start of the iterator.

### Signature
```rust
fn positions<P>(self, predicate: P) -> Positions<Self, P>
where
    Self: Sized,
    P: FnMut(Self::Item) -> bool,
```
```

--------------------------------

### Token Implementations

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Provides methods for interacting with and retrieving information from a Token instance.

```APIDOC
### impl Token<'_>

*   `lemma(&self) -> &str` - Returns a reference over the normalized lemma.
*   `byte_len(&self) -> usize` - Returns the length in bytes of the normalized lemma.
*   `original_byte_len(&self) -> usize` - Returns the length in bytes of the original lemma.
*   `char_count(&self) -> usize` - Returns the count of characters of the normalized lemma.
*   `original_char_count(&self) -> usize` - Returns the count of characters of the original lemma.
*   `kind(&self) -> TokenKind` - Returns the `TokenKind` of the current token.
*   `is_word(&self) -> bool` - Returns true if the current token is a word (not a separator or stop word).
*   `is_stopword(&self) -> bool` - Returns true if the current token is a stop word.
*   `is_separator(&self) -> bool` - Returns true if the current token is a separator.
*   `separator_kind(&self) -> Option<SeparatorKind>` - Returns Some(`SeparatorKind`) if the token is a separator, otherwise None.
*   `original_lengths(&self, num_bytes: usize) -> (usize, usize)` - Returns the number of characters and bytes in the original token using the number of bytes in the normalized token. Arguments: `num_bytes` (usize) - number of bytes in normalized token.
```

--------------------------------

### Get Next Token from SegmentedTokenIter

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Advances the iterator and returns the next Token. Returns None if the iterator is exhausted.

```rust
fn next(&mut self) -> Option<Self::Item>
```

--------------------------------

### Get Token Kind

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Returns the TokenKind of the current token. This indicates whether the token is a word, separator, etc.

```rust
pub fn kind(&self) -> TokenKind
```

--------------------------------

### compare

Source: https://docs.rs/charabia/0.9.9/charabia/enum.Script.html

Compares self to a given key and returns their ordering.

```APIDOC
## compare

### Description
Compare self to `key` and return their ordering.

### Method
`fn compare(&self, key: &K) -> Ordering`
```

--------------------------------

### Initialize StrDetection

Source: https://docs.rs/charabia/0.9.9/charabia/struct.StrDetection.html

Creates a new StrDetection instance. It takes the string to analyze and an optional allow list of languages.

```rust
pub fn new(inner: &'o str, allow_list: Option<&'al [Language]>) -> Self
```

--------------------------------

### Get Language from StrDetection

Source: https://docs.rs/charabia/0.9.9/charabia/struct.StrDetection.html

Retrieves the detected language from the StrDetection instance. This method requires a mutable reference to self.

```rust
pub fn language(&mut self) -> Option<Language>
```

--------------------------------

### Get Script from StrDetection

Source: https://docs.rs/charabia/0.9.9/charabia/struct.StrDetection.html

Retrieves the detected script from the StrDetection instance. This method requires a mutable reference to self.

```rust
pub fn script(&mut self) -> Script
```

--------------------------------

### pad_using

Source: https://docs.rs/charabia/0.9.9/charabia/struct.ReconstructedTokenIter.html

Pads the sequence to a minimum length using a closure to generate fill values. The closure is called with the index of the missing element.

```APIDOC
## fn pad_using<F>(self, min: usize, f: F) -> PadUsing<Self, F>

### Description
Return an iterator adaptor that pads the sequence to a minimum length of `min` by filling missing elements using a closure `f`.

### Parameters
* `min`: The minimum length of the resulting iterator.
* `f`: A closure that takes the index of the missing element (starting from the end of the original iterator) and returns the item to fill.

### Type Parameters
* `F`: The type of the closure, must be `FnMut(usize) -> Self::Item`.

### Constraints
* `Self`: Must implement `Sized`.

### Returns
A `PadUsing` iterator adaptor.
```

--------------------------------

### fold_ok

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Folds `Result` values from an iterator using a starting value and a folding function. This is the recommended alternative to `fold_results`.

```APIDOC
## fn fold_ok<A, E, B, F>(&mut self, start: B, f: F) -> Result<B, E>

### Description
Fold `Result` values from an iterator.

### Method
`fold_ok`

### Parameters
- `start` (B): The initial value for the accumulator.
- `f` (F): The folding function.

### Returns
`Result<B, E>` - The final accumulated value or the first error encountered.
```

--------------------------------

### Get Separator Kind

Source: https://docs.rs/charabia/0.9.9/src/charabia/token.rs.html

Returns `Some(SeparatorKind)` if the token is a separator, and `None` otherwise. This allows distinguishing between `Hard` and `Soft` separators.

```rust
/// Returns Some([`SeparatorKind`]) if the token is a separator and None if it's a word or a stop word.
pub fn separator_kind(&self) -> Option<SeparatorKind> {
    if let TokenKind::Separator(s) = self.kind {
        Some(s)
    } else {
        None
    }
}
```

--------------------------------

### pad_using

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedStrIter.html

Pads the sequence to a minimum length using a provided closure for filling missing elements.

```APIDOC
## fn pad_using<F>(self, min: usize, f: F) -> PadUsing<Self, F>

### Description
Return an iterator adaptor that pads the sequence to a minimum length of `min` by filling missing elements using a closure `f`.

### Type Parameters
- `Self`: The iterator type, must implement `Sized`.
- `F`: The type of the closure used for padding, must implement `FnMut(usize) -> Self::Item`.

### Parameters
- `min`: The minimum length the resulting iterator should have.
- `f`: A closure that takes the current index (starting from the end of the original iterator) and returns the item to use for padding.

### Returns
A `PadUsing` iterator adaptor.
```

--------------------------------

### StrDetection::new

Source: https://docs.rs/charabia/0.9.9/charabia/struct.StrDetection.html

Creates a new StrDetection instance.

```APIDOC
## StrDetection::new

### Description
Creates a new `StrDetection` instance.

### Signature
```rust
pub fn new(inner: &'o str, allow_list: Option<&'al [Language]>) -> Self
```

### Parameters
* `inner`: The string slice to analyze.
* `allow_list`: An optional slice of `Language` to restrict detection to.
```

--------------------------------

### Get Size Hint for SegmentedTokenIter

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Returns the lower and upper bounds of the remaining elements in the iterator. This is available since 1.0.0.

```rust
fn size_hint(&self) -> (usize, Option<usize>)
```

--------------------------------

### pad_using

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Pads the sequence to a minimum length of `min` by filling missing elements using a closure `f`.

```APIDOC
## fn pad_using<F>(self, min: usize, f: F) -> PadUsing<Self, F>

### Description
Returns an iterator adaptor that pads the sequence to a minimum length of `min` by filling missing elements using a closure `f`.

### Parameters
- **min** (usize) - The minimum length of the resulting iterator.
- **f** (F) - A closure that takes the current index and returns an item to fill the padding.

### Type Constraints
- `Self` must implement `Sized`.
- `F` must implement `FnMut(usize) -> Self::Item`.
```

--------------------------------

### Get Remaining String Segment

Source: https://docs.rs/charabia/0.9.9/src/charabia/segmenter/utils.rs.html

Returns the portion of the string that has not yet been segmented. Returns None if the offset is at the end of the string.

```rust
fn tail(&self) -> Option<&'o str> {
        self.to_segment.get(self.offset..).filter(|s| !s.is_empty())
    }
```

--------------------------------

### Get Separator Kind

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Returns Some(SeparatorKind) if the token is a separator, otherwise returns None. This provides more detail about the type of separator.

```rust
pub fn separator_kind(&self) -> Option<SeparatorKind>
```

--------------------------------

### Normalize Token with Classifier Options

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/classify.rs.html

Demonstrates how to create and use a Classifier with specific options, including stop words and separators, to normalize a token. Asserts the normalized token's properties.

```rust
        let options = NormalizerOption {
            create_char_map,
            classifier: ClassifierOption {
                stop_words: Some(stop_words),
                separators: Some(&separators),
            },
            lossy,
        };

        let token = Classifier
            .normalize(Token { lemma: Cow::Borrowed(&lemma), ..Default::default() }, &options);
        assert!(token.is_stopword());
        assert!(!token.is_separator());
```

--------------------------------

### TokenizerBuilder::new

Source: https://docs.rs/charabia/0.9.9/charabia/struct.TokenizerBuilder.html

Creates a new TokenizerBuilder with default settings. Prefer TokenizerBuilder::default if stop_words are not planned to be set.

```APIDOC
## TokenizerBuilder::new

### Description
Create a `TokenizerBuilder` with default settings.

### Returns
A new `TokenizerBuilder` instance.

### Example
```rust
use charabia::TokenizerBuilder;

let mut builder = TokenizerBuilder::new();
```
```

--------------------------------

### Get Character Count of Original Lemma

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Returns the count of characters in the original lemma. This provides the character count before normalization.

```rust
pub fn original_char_count(&self) -> usize
```

--------------------------------

### take

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.NormalizedTokenIter.html

Creates an iterator that yields the first `n` elements, or fewer if the underlying iterator ends sooner.

```APIDOC
## take

### Description
Creates an iterator that yields the first `n` elements, or fewer if the underlying iterator ends sooner.

### Signature
```rust
fn take(self, n: usize) -> Take<Self>
where
    Self: Sized,
```
```

--------------------------------

### Get Language Code

Source: https://docs.rs/charabia/0.9.9/charabia/enum.Language.html

Retrieves the static string slice representing the language code for a given Language enum variant.

```rust
pub fn code(&self) -> &'static str
```

--------------------------------

### fold_options

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Folds `Option` values from an iterator using a starting value and a folding function. If any element is `None`, the result is `None`.

```APIDOC
## fn fold_options<A, B, F>(&mut self, start: B, f: F) -> Option<B>

### Description
Fold `Option` values from an iterator.

### Method
`fold_options`

### Parameters
- `start` (B): The initial value for the accumulator.
- `f` (F): The folding function.

### Returns
`Option<B>` - The final accumulated value, or `None` if any element was `None`.
```

--------------------------------

### from

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Creates a new Token from a given value.

```APIDOC
## fn from(t: T) -> T

### Description
Returns the argument unchanged.

### Method
`from`

### Parameters
- `t` (*T*): The value to create the Token from.

### Returns
- `T`: The created Token.
```

--------------------------------

### Configure Tokenizer Separators

Source: https://docs.rs/charabia/0.9.9/charabia/struct.TokenizerBuilder.html

Shows how to customize the separators used by the tokenizer. This allows defining specific strings that should be treated as word boundaries.

```rust
use charabia::TokenizerBuilder;

// create the builder.
let mut builder = TokenizerBuilder::default();

// create a custom list of separators.
let separators = [" ", ", ", ". ", "?", "!"];

// configurate separators.
builder.separators(&separators);

// build the tokenizer passing the text to tokenize.
let tokenizer = builder.build();

// text to tokenize.
let orig = "The quick (\"brown\") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

let output: Vec<_> = tokenizer.segment_str(orig).collect();
assert_eq!(
  &output,
  &["The", " ", "quick", " ", "(\"brown\")", " ", "fox", " ", "can't", " ", "jump", " ", "32.3", " ", "feet", ", ", "right", "?", " ", "Brr", ", ", "it's", " ", "29.3°F", "!"]
);
```

--------------------------------

### Get Token Kind

Source: https://docs.rs/charabia/0.9.9/src/charabia/token.rs.html

Returns the `TokenKind` of the current token. This indicates whether the token is a word, stop word, separator, or unknown.

```rust
/// Returns the [`TokenKind`] of the current token.
pub fn kind(&self) -> TokenKind {
    self.kind
}
```

--------------------------------

### Get Nth Element from SegmentedTokenIter

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Returns the `n`th element of the iterator. This consumes elements up to `n`. This is available since 1.28.0.

```rust
fn nth(&mut self, n: usize) -> Option<Self::Item>
```

--------------------------------

### Normalized Persian Tokens

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/persian.rs.html

Illustrates the structure of normalized Persian tokens, including lemma, character and byte ends, script, language, token kind, and character map.

```rust
fn normalized_tokens() -> Vec<Token<'static>> {
        vec![
            Token {
                lemma: Owned("علی".to_string()),
                char_end: 3,
                byte_end: 6,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                char_map: Some(vec![(2, 2), (2, 2), (2, 2)]),
                ..Default::default()
            },
            Token {
                lemma: Owned("کتاب".to_string()),
                char_end: 4,
                byte_end: 8,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                char_map: Some(vec![(2, 2), (2, 2), (2, 2), (2, 2)]),
                ..Default::default()
            },
            Token {
                lemma: Owned("1234567890".to_string()),
                char_end: 10,
                byte_end: 10,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                char_map: Some(vec![(2, 1); 10]),
                ..Default::default()
            },
            Token {
                lemma: Owned("کیک 123 یک کتاب".to_string()),
                char_end: 13,
                byte_end: 24,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                char_map: Some(vec![
                    (2, 2),
                    (2, 2),
                    (2, 2), // کیک
                    (1, 1), // space
                    (2, 1),
                    (2, 1),
                    (2, 1), // ۱۲۳ (Persian digits, normalized to ASCII)
                    (1, 1), // space
                    (2, 2),
                    (2, 2), // یک
                    (1, 1), // space
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2), // کتاب
                ]),
                ..Default::default()
            },
            Token {
                lemma: Owned("سلام".to_string()),
                char_end: 4,
                byte_end: 8,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                ..Default::default()
            },
            Token {
                lemma: Owned("خانه".to_string()),
                char_end: 4,
                byte_end: 8,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                ..Default::default()
            },
            Token {
                lemma: Owned("گژ پژ".to_string()),
                char_end: 5,
                byte_end: 10,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                ..Default::default()
            },
            Token {
                lemma: Owned("قنات قصبه شهر گناباد عمیقترین و قدیمیترین کاریز جهان است.".to_string()),
                char_end: 56,
                byte_end: 112,
                script: Script::Arabic,
                language: Some(Language::Pes),
                kind: TokenKind::Word,
                char_map: Some(vec![
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (1, 1),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                    (1, 1),
                    (2, 2),
                    (2, 2),
                    (2, 2),
                ]),
                ..Default::default()
            },
        ]
    }
```

--------------------------------

### Get Next Chunk of Tokens

Source: https://docs.rs/charabia/0.9.9/charabia/segmenter/struct.SegmentedTokenIter.html

Advances the iterator and returns an array containing the next N values. This is a nightly-only experimental API.

```rust
fn next_chunk<const N: usize>( &mut self, ) -> Result<[Self::Item; N], IntoIter<Self::Item, N>>
where Self: Sized,
```

--------------------------------

### Build Tokenizer with Custom Words and Separators

Source: https://docs.rs/charabia/0.9.9/src/charabia/tokenizer.rs.html

The `build` method constructs the `Tokenizer`. It creates an Aho-Corasick automaton for pre-segmentation if custom words or separators are provided, merging them for efficient pattern matching.

```rust
pub fn build(&mut self) -> Tokenizer<'_> {
    // If a custom list of separators or/and a custom list of words have been given,
    // then an Aho-Corasick automaton is created to pre-segment the text during the tokenization process
    // TODO: avoid recreating the automaton if nothing changed
    match (self.normalizer_option.classifier.separators, self.words_dict) {
        (Some(separators), None) => {
            let pattern = separators.iter().filter(|s| !s.is_empty());
            let aho = AhoCorasick::builder()
                .match_kind(MatchKind::LeftmostLongest)
                .build(pattern)
                .unwrap();

            self.segmenter_option.aho = Some(aho).filter(|aho| aho.patterns_len() != 0);
        }
        (separators, Some(words)) => {
            // use the default separators' list if a custom words' list is given but no custom separators' list.
            let separators = separators.unwrap_or(DEFAULT_SEPARATORS);
            // merge both lists together and create the Aho-Corasick automaton.
            let pattern = words.iter().chain(separators).filter(|s| !s.is_empty());
            let aho = AhoCorasick::builder()
                .match_kind(MatchKind::LeftmostLongest)
                .build(pattern)
                .unwrap();

            self.segmenter_option.aho = Some(aho).filter(|aho| aho.patterns_len() != 0);
        }
        // reset the state in case the builder is reused.
        (None, None) => self.segmenter_option.aho = None,
    }

    Tokenizer {
        normalizer_option: Cow::Borrowed(&self.normalizer_option),
        segmenter_option: Cow::Borrowed(&self.segmenter_option),
    }
}
```

--------------------------------

### Get Byte Length of Original Lemma

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Returns the length in bytes of the original lemma before any normalization. This can be useful for comparing with the normalized length.

```rust
pub fn original_byte_len(&self) -> usize
```

--------------------------------

### Language Methods

Source: https://docs.rs/charabia/0.9.9/charabia/enum.Language.html

Provides methods to interact with the Language enum, such as getting the string code for a language or creating a Language from a code string.

```APIDOC
## Implementations

### impl Language

#### pub fn code(&self) -> &'static str

#### pub fn from_code<S: AsRef<str>>(code: S) -> Option<Language>
```

--------------------------------

### clone_to_uninit

Source: https://docs.rs/charabia/0.9.9/charabia/enum.Script.html

Performs copy-assignment from `self` to `dest`. This is a nightly-only experimental API.

```APIDOC
## clone_to_uninit

### Description
Performs copy-assignment from `self` to `dest`. This is a nightly-only experimental API.

### Method
`unsafe fn clone_to_uninit(&self, dest: *mut u8)`
```

--------------------------------

### Script Name Conversion

Source: https://docs.rs/charabia/0.9.9/charabia/enum.Script.html

Provides a method to get the static string name of a Script variant. Useful for display or logging purposes.

```rust
pub fn name(&self) -> &'static str
```

--------------------------------

### pad_using

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.NormalizedTokenIter.html

Returns an iterator adaptor that pads the sequence to a minimum length of `min` by filling missing elements using a closure `f`.

```APIDOC
## pad_using

### Description
Returns an iterator adaptor that pads the sequence to a minimum length of `min` by filling missing elements using a closure `f`.

### Signature
```rust
fn pad_using<F>(self, min: usize, f: F) -> PadUsing<Self, F>
where
    Self: Sized,
    F: FnMut(usize) -> Self::Item,
```
```

--------------------------------

### partition

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.NormalizedTokenIter.html

Consumes an iterator, creating two collections from it.

```APIDOC
## partition

### Description
Consumes an iterator, creating two collections from it.

### Signature
```rust
fn partition<B, F>(self, f: F) -> (B, B)
where
    Self: Sized,
    B: Default + Extend<Self::Item>,
    F: FnMut(&Self::Item) -> bool,
```
```

--------------------------------

### Jieba Initialization

Source: https://docs.rs/charabia/0.9.9/src/charabia/segmenter/chinese.rs.html

Initializes the Jieba segmenter using LazyLock for thread-safe, one-time initialization.

```rust
static JIEBA: LazyLock<Jieba> = LazyLock::new(Jieba::new);
```

--------------------------------

### Get Normalized Lemma Character Count

Source: https://docs.rs/charabia/0.9.9/src/charabia/token.rs.html

Returns the total number of characters in the normalized lemma. This count is based on Unicode scalar values.

```rust
/// Returns the count of characters of the normalized lemma.
pub fn char_count(&self) -> usize {
    self.lemma.chars().count()
}
```

--------------------------------

### Get Normalized Lemma Byte Length

Source: https://docs.rs/charabia/0.9.9/src/charabia/token.rs.html

Returns the length in bytes of the normalized lemma. This is useful for understanding the storage size of the processed token.

```rust
/// Returns the length in bytes of the normalized lemma.
pub fn byte_len(&self) -> usize {
    self.lemma.len()
}
```

--------------------------------

### Script Enum Usage and Name Conversion

Source: https://docs.rs/charabia/0.9.9/src/charabia/detection/script_language.rs.html

Demonstrates converting script names to their corresponding enum variants and vice versa. Ensure the script name is valid.

```rust
assert_eq!(Script::from_name("Latin"), Script::Latin);
        assert_eq!(Script::Cj.name(), "Mandarin");
        assert_eq!(Script::from_name("Mandarin"), Script::Cj);
```

--------------------------------

### Get Character Count of Normalized Lemma

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Returns the count of characters in the normalized lemma. This differs from byte length for multi-byte character encodings.

```rust
pub fn char_count(&self) -> usize
```

--------------------------------

### Clone Implementation for NormalizerOption

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.NormalizerOption.html

Provides methods to duplicate or copy-assign a NormalizerOption. Use `clone` for a new copy and `clone_from` to overwrite an existing instance.

```rust
fn clone(&self) -> NormalizerOption<'tb>
```

```rust
fn clone_from(&mut self, source: &Self)
```

--------------------------------

### Clone To Uninit Method (Nightly)

Source: https://docs.rs/charabia/0.9.9/charabia/normalizer/struct.ClassifierOption.html

Performs copy-assignment from self to an uninitialized memory location. This is a nightly-only experimental API.

```rust
unsafe fn clone_to_uninit(&self, dest: *mut u8)
```

--------------------------------

### Get Byte Length of Normalized Lemma

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Returns the length in bytes of the normalized lemma. This is useful for understanding the memory footprint of the processed token.

```rust
pub fn byte_len(&self) -> usize
```

--------------------------------

### into

Source: https://docs.rs/charabia/0.9.9/charabia/enum.Script.html

Calls `U::from(self)` for type conversion.

```APIDOC
## into

### Description
Calls `U::from(self)`.
That is, this conversion is whatever the implementation of `From<T> for U` chooses to do.

### Method
`fn into(self) -> U`
```

--------------------------------

### into

Source: https://docs.rs/charabia/0.9.9/charabia/struct.Token.html

Converts the Token into another type.

```APIDOC
## fn into(self) -> U

### Description
Calls `U::from(self)`. That is, this conversion is whatever the implementation of `From<T> for U` chooses to do.

### Method
`into`

### Parameters
- `self` (*self*): The Token instance to convert.

### Returns
- `U`: The converted value.
```

--------------------------------

### Script PartialEq Implementation

Source: https://docs.rs/charabia/0.9.9/charabia/enum.Script.html

Enables equality comparison between Script variants. This is fundamental for checking if two script values are the same.

```rust
fn eq(&self, other: &Script) -> bool
```

```rust
fn ne(&self, other: &Rhs) -> bool
```

--------------------------------

### Get Original Lemma Character Count

Source: https://docs.rs/charabia/0.9.9/src/charabia/token.rs.html

Returns the total number of characters in the original, unnormalized lemma. This count is based on Unicode scalar values.

```rust
/// Returns the count of characters of the original lemma.
pub fn original_char_count(&self) -> usize {
    self.char_end - self.char_start
}
```

--------------------------------

### Context Separator Set Initialization

Source: https://docs.rs/charabia/0.9.9/src/charabia/normalizer/classify.rs.html

Initializes the `CONTEXT_SEPARATOR_SET` using `LazyLock` for efficient, one-time collection of context-specific separators from `crate::separators::CONTEXT_SEPARATORS`.

```rust
pub static CONTEXT_SEPARATOR_SET: LazyLock<HashSet<&str>> = 
    LazyLock::new(|| crate::separators::CONTEXT_SEPARATORS.iter().copied().collect());
```
Tokens:
Raw