Wordchipper

https://docs.rs/wordchipper/latest/
Admin
Wordchipper is a high-performance LLM tokenizer suite for Rust that supports multiple token types...
Tokens:355,643
Snippets:3,623
Trust Score:9.1
Update:5 months ago
Show doc for...
Tokens:
Raw
### Get HashSet Capacity

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashSet.html

This example demonstrates how to check the capacity of a HashSet, ensuring it meets a minimum requirement.

```rust
use std::collections::HashSet;
let set: HashSet<i32> = HashSet::with_capacity(100);
assert!(set.capacity() >= 100);
```

--------------------------------

### Token Decoders Example

Source: https://docs.rs/wordchipper/latest/wordchipper/decoders/index.html

An example demonstrating how to use TokenDecoderOptions to build a decoder and decode a batch of tokens to strings.

```APIDOC
## §Token Decoders
### §Example
```rust
use std::sync::Arc;

use wordchipper::{
    TokenDecoder,
    TokenDecoderOptions,
    TokenType,
    UnifiedTokenVocab,
};

fn example<T: TokenType>(
    vocab: Arc<UnifiedTokenVocab<T>>,
    batch: &[Vec<T>],
) -> Vec<String> {
    let decoder = TokenDecoderOptions::default().build(vocab);

    let slices: Vec<&[T]> = batch.iter().map(|v| v.as_ref()).collect();

    decoder
        .try_decode_batch_to_strings(&slices)
        .unwrap()
        .unwrap()
}
```
```

--------------------------------

### Token Encoder Example

Source: https://docs.rs/wordchipper/latest/src/wordchipper/encoders/mod.rs.html

Demonstrates how to build and use a token encoder with a vocabulary. This example requires the `TokenType` trait and `Arc` for shared ownership of the vocabulary.

```rust
use std::sync::Arc;

use wordchipper::{
    TokenEncoder,
    TokenEncoderOptions,
    TokenType,
    UnifiedTokenVocab,
};

fn example<T: TokenType>(
    vocab: Arc<UnifiedTokenVocab<T>>,
    batch: &[&str],
) -> Vec<Vec<T>> {
    let vocab1 = vocab.clone();
    let encoder = TokenEncoderOptions::default().build(vocab1);
    encoder.try_encode_batch(batch, None).unwrap()
}
```

--------------------------------

### Get HashSet Length

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashSet.html

This example shows how to get the number of elements currently in the HashSet.

```rust
use std::collections::HashSet;

let mut v = HashSet::new();
assert_eq!(v.len(), 0);
v.insert(1);
assert_eq!(v.len(), 1);
```

--------------------------------

### Handling File Metadata with `and_then`

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCResult.html

Shows how to use `and_then` to chain operations that might fail, such as getting file metadata and then its modified time. This example highlights error handling for file system operations.

```rust
use std::{io::ErrorKind, path::Path};

// Note: on Windows "/" maps to "C:\"
let root_modified_time = Path::new("/").metadata().and_then(|md| md.modified());
assert!(root_modified_time.is_ok());

let should_fail = Path::new("/bad/path").metadata().and_then(|md| md.modified());
assert!(should_fail.is_err());
assert_eq!(should_fail.unwrap_err().kind(), ErrorKind::NotFound);
```

--------------------------------

### Example usage of join_patterns! macro

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/utility/pattern_tools.rs.html

Illustrates the use of the `join_patterns!` macro to combine expressions with a '|' separator.

```rust
assert_eq!(join_patterns!("a", "b", "c"), "a|b|c");
```

--------------------------------

### new

Source: https://docs.rs/wordchipper/latest/wordchipper/struct.UnifiedTokenVocab.html

Initializes a UnifiedTokenVocab with TextSpanningConfig, SpanMapVocab, and PairMapVocab.

```APIDOC
## pub fn new( span_config: TextSpanningConfig<T>, span_vocab: SpanMapVocab<T>, pair_vocab: PairMapVocab<T>, ) -> WCResult<Self>

### Description
Initialize a `UnifiedTokenVocab`.

### Arguments
  * `span_config` - The spanners configuration.
  * `word_vocab` - The span map vocabulary.
  * `pair_vocab` - The pair map vocabulary.

### Returns
A `Result<UnifiedTokenVocab>`, with errors on vocab conflict.
```

--------------------------------

### Initialize VocabQuery with Path

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/factory/vocab_query.rs.html

Illustrates creating a VocabQuery and setting its path using the `with_path` method. Confirms that the path and name are correctly assigned.

```rust
let query = VocabQuery::new(None, None, "vocab_name").with_path(Some("path_name"));
assert_eq!(query.path(), Some("path_name"));
assert_eq!(query.name(), "vocab_name");
```

--------------------------------

### Initialize VocabQuery with Schema

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/factory/vocab_query.rs.html

Shows how to create a VocabQuery and then explicitly set its schema using the `with_schema` method. Verifies that the schema and name are correctly set.

```rust
let query = VocabQuery::new(None, None, "vocab_name").with_schema(Some("provider"));
assert_eq!(query.schema(), Some("provider"));
assert_eq!(query.name(), "vocab_name");
```

--------------------------------

### Getting HashMap Length

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashMap.html

Demonstrates how to get the number of elements in a HashMap.

```rust
use std::collections::HashMap;

let mut a = HashMap::new();
assert_eq!(a.len(), 0);
a.insert(1, "a");
assert_eq!(a.len(), 1);
```

--------------------------------

### init

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.SpanMapVocab.html

Initializes a with the given initializer.

```APIDOC
## init

### Description
Initializes a with the given initializer.

### Method
`unsafe fn init(init: <T as Pointable>::Init) -> usize`
```

--------------------------------

### Get Type ID

Source: https://docs.rs/wordchipper/latest/wordchipper/decoders/struct.TokenDecoderOptions.html

Gets the TypeId of the TokenDecoderOptions. This is part of the Any trait implementation.

```rust
fn type_id(&self) -> TypeId
```

--------------------------------

### UnifiedTokenVocab::new

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.UnifiedTokenVocab.html

Initialize a UnifiedTokenVocab.

```APIDOC
## UnifiedTokenVocab::new

### Description
Initialize a `UnifiedTokenVocab`.

### Arguments
- `span_config` (TextSpanningConfig<T>) - The spanners configuration.
- `word_vocab` (SpanMapVocab<T>) - The span map vocabulary.
- `pair_vocab` (PairMapVocab<T>) - The pair map vocabulary.

### Returns
A `Result<UnifiedTokenVocab>`, with errors on vocab conflict.
```

--------------------------------

### Retrieving a Value by Key

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashMap.html

Demonstrates how to get a reference to a value associated with a key using the `get` method. Returns `None` if the key is not found.

```rust
use std::collections::HashMap;

let mut map = HashMap::new();
map.insert(1, "a");
assert_eq!(map.get(&1), Some(&"a"));
assert_eq!(map.get(&2), None);
```

--------------------------------

### Get OpenAI Tokenizer Factory

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/openai/factories.rs.html

Provides a method to get the corresponding `ConstVocabularyFactory` for each `OATokenizer` variant. This allows accessing the correct vocabulary loader.

```rust
impl OATokenizer {
    /// Get the tokenizer vocabulary factory.
    pub fn factory(&self) -> &ConstVocabularyFactory {
        use OATokenizer::*;
        match self {
            R50kBase => &OA_R50K_BASE_VOCAB_FACTORY,
            P50kBase => &OA_P50K_BASE_VOCAB_FACTORY,
            P50kEdit => &OA_P50K_EDIT_VOCAB_FACTORY,

```

--------------------------------

### Init

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.SpanMapVocab.html

The type for initializers.

```APIDOC
## Init

### Description
The type for initializers.

### Associated Type
`type Init = T`
```

--------------------------------

### VocabListing::new

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/factory/struct.VocabListing.html

Constructs a new VocabListing instance.

```APIDOC
## pub fn new( source: &str, description: &str, vocabs: Vec<VocabDescription>, ) -> Self

### Description
Build a new vocabulary listing.

### Parameters
* `source`: &str - The source identifier for the vocabulary listing.
* `description`: &str - A textual description of the vocabulary listing.
* `vocabs`: Vec<VocabDescription> - A vector of `VocabDescription` structs.

### Returns
- Self: A new `VocabListing` instance.
```

--------------------------------

### Example usage of join_strs! macro

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/utility/pattern_tools.rs.html

Demonstrates concatenating string literals with different separators and handling single literals.

```rust
use wordchipper::join_strs;

// Concatenating string literals with a comma as the separator
let result = join_strs!( ",", ("Hello", "World", "Rust"));
assert_eq!(result, "Hello,World,Rust");

// Concatenating string literals with a dash as the separator
let result = join_strs!("-", ("A", "B", "C"));
assert_eq!(result, "A-B-C");

// Concatenating a single string literal without a separator
let result = join_strs!(";", ("OnlyOne"));
assert_eq!(result, "OnlyOne");
```

--------------------------------

### Load Unified Vocab from Path

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/io/base64_vocab.rs.html

Builds a `UnifiedTokenVocab` from a Base64 encoded vocabulary file. Requires a path to the file and text spanning configuration.

```rust
pub fn load_base64_unified_vocab_path<T: TokenType>(
    path: impl AsRef<Path>,
    spanning: TextSpanningConfig<T>,
) -> WCResult<UnifiedTokenVocab<T>> {
    let mut reader = BufReader::new(File::open(path)?);
    read_base64_unified_vocab(&mut reader, spanning)
}
```

--------------------------------

### RegexPattern Methods

Source: https://docs.rs/wordchipper/latest/src/wordchipper/support/regex/regex_pattern.rs.html

Provides methods to access the underlying regex string and compile the pattern. Use `as_str` to get the pattern string and `compile` to get a `RegexWrapper`.

```rust
impl RegexPattern {
    /// Get the underlying regex pattern.
    ///
    /// ## Returns
    /// The regex pattern string slice.
    pub fn as_str(&self) -> &str {
        match self {
            Self::Basic(pattern) => pattern,
            Self::Fancy(pattern) => pattern,
            Self::Adaptive(pattern) => pattern,
        }
    }

    /// Compile the regex pattern into a `RegexWrapper`.
    ///
    /// ## Returns
    /// A `Result` containing the compiled `RegexWrapper` or an `ErrorWrapper`.
    pub fn compile(&self) -> Result<RegexWrapper, ErrorWrapper> {
        match self {
            Self::Basic(pattern) => regex::Regex::new(pattern)
                .map(RegexWrapper::from)
                .map_err(ErrorWrapper::from),
            Self::Fancy(pattern) => fancy_regex::Regex::new(pattern)
                .map(RegexWrapper::from)
                .map_err(ErrorWrapper::from),
            Self::Adaptive(pattern) => {
                regex::Regex::new(pattern)
                    .map(RegexWrapper::from)
                    .or_else(|_| {
                        fancy_regex::Regex::new(pattern)
                            .map(RegexWrapper::from)
                            .map_err(ErrorWrapper::from)
                    })
            }
        }
    }
}
```

--------------------------------

### token_pairs

Source: https://docs.rs/wordchipper/latest/src/wordchipper/decoders/utility/pair_decoder.rs.html

Get the [`TokenPairMap`].

```APIDOC
## fn token_pairs(&self) -> &TokenPairMap<T>

### Description
Get the [`TokenPairMap`].

### Returns
A reference to the `TokenPairMap`.
```

--------------------------------

### new

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashSet.html

Creates an empty `HashSet`.

```APIDOC
## fn new() -> HashSet<T, FixedState>

### Description
Creates an empty `HashSet`.

### Response
- **HashSet<T, FixedState>** - An empty HashSet.
```

--------------------------------

### byte_vocab

Source: https://docs.rs/wordchipper/latest/src/wordchipper/decoders/utility/pair_decoder.rs.html

Get the [`ByteMapVocab`].

```APIDOC
## fn byte_vocab(&self) -> &ByteMapVocab<T>

### Description
Get the [`ByteMapVocab`].

### Returns
A reference to the `ByteMapVocab`.
```

--------------------------------

### HashMap Capacity Management

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashMap.html

Demonstrates how to initialize a HashMap with a specific capacity, insert elements, and then shrink the capacity.

```rust
use std::collections::HashMap;

let mut map: HashMap<i32, i32> = HashMap::with_capacity(100);
map.insert(1, 2);
map.insert(3, 4);
assert!(map.capacity() >= 100);
map.shrink_to(10);
assert!(map.capacity() >= 10);
map.shrink_to(0);
assert!(map.capacity() >= 2);
```

--------------------------------

### UnifiedTokenVocab::pair_vocab

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.UnifiedTokenVocab.html

Get the {(T, T) -> T} PairMapVocab.

```APIDOC
## UnifiedTokenVocab::pair_vocab

### Description
Get the `{ (T, T) -> T }` `PairMapVocab`.
```

--------------------------------

### From

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.ByteMapVocab.html

Creates a new instance from a given value.

```APIDOC
## from

### Description
Returns the argument unchanged.

### Method
`fn from(t: T) -> T`
```

--------------------------------

### UnifiedTokenVocab::spanning

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.UnifiedTokenVocab.html

Get the TextSpanningConfig.

```APIDOC
## UnifiedTokenVocab::spanning

### Description
Get the `TextSpanningConfig`.
```

--------------------------------

### Create and Verify UnifiedTokenVocab

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/unified_vocab.rs.html

Demonstrates creating a UnifiedTokenVocab from a span configuration and a span vocabulary, then verifies its byte and pair vocabularies against expected values. This is useful for setting up and testing vocabulary structures.

```rust
let vocab = UnifiedTokenVocab::from_span_vocab(seg_config, span_vocab.clone()).unwrap();
assert_eq!(vocab.len(), 256 + 2);

let byte_vocab = vocab.byte_vocab();

{
    let mut expected: PairTokenMap<T> = Default::default();
    expected.insert(
        (
            T::from_u8('a' as u8).unwrap(),
            T::from_u8('t' as u8).unwrap(),
        ),
        300,
    );
    expected.insert((300, T::from_u8('e' as u8).unwrap()), 301);
    let expected: PairMapVocab<T> = 
        PairMapVocab::new(byte_vocab.clone(), expected).unwrap();

    assert_eq!(vocab.pair_vocab(), &expected);
}

{
    let mut expected: SpanTokenMap<T> = byte_vocab.span_pairs().collect();
    expected.extend(span_vocab.span_pairs());
    let expected: SpanMapVocab<T> = expected.into();

    assert_eq!(vocab.span_vocab(), &expected);
}

assert_eq!(vocab.span_pairs().collect::<Vec<_>>(), vocab.span_vocab.span_pairs().collect::<Vec<_>>());

assert_eq!(vocab.lookup_token("at".as_bytes()), Some(300));
assert_eq!(vocab.lookup_token("ate".as_bytes()), Some(301));
assert_eq!(vocab.lookup_token("a".as_bytes()), Some(byte_vocab.get_token(b'a')));

assert_eq!(vocab.lookup_pair(&(byte_vocab.get_token(b'a'), byte_vocab.get_token(b't'))), Some(300));
```

--------------------------------

### Load and Configure Tokenizer with UnifiedTokenVocab

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/unified_vocab.rs.html

Demonstrates loading a pre-trained vocabulary and creating a TokenEncoderBuilder, either with default settings or custom configurations like parallel processing.

```rust
use wordchipper::vocab::UnifiedTokenVocab;

// Load a pre-trained vocabulary
let vocab: UnifiedTokenVocab<u32> = load_pretrained_vocab()?;

// Create a default encoder
let encoder = TokenEncoderBuilder::default(vocab.clone());

// Or configure with custom options
let encoder = TokenEncoderBuilder::new(vocab.clone())
    .parallel(true)
    .build();
```

--------------------------------

### type_id

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/openai/factories/struct.OATokenizerIter.html

Gets the TypeId of self.

```APIDOC
## fn type_id(&self) -> TypeId

### Description
Gets the `TypeId` of `self`.
```

--------------------------------

### Creating and Iterating Over HashMap Keys

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashMap.html

Demonstrates creating a HashMap and collecting its keys into a sorted vector. Note that keys are produced in arbitrary order.

```rust
use std::collections::HashMap;

let map = HashMap::from([
    ("a", 1),
    ("b", 2),
    ("c", 3),
]);

let mut vec: Vec<&str> = map.into_keys().collect();
// The `IntoKeys` iterator produces keys in arbitrary order, so the
// keys must be sorted to test them against a sorted array.
vec.sort_unstable();
assert_eq!(vec, ["a", "b", "c"]);
```

--------------------------------

### span_encoder

Source: https://docs.rs/wordchipper/latest/src/wordchipper/encoders/encoder_options.rs.html

Gets the configured SpanEncoderSelector.

```APIDOC
## fn span_encoder(&self) -> Option<SpanEncoderSelector>

### Description
Get the configured [`SpanEncoderSelector`].

### Returns
- `Option<SpanEncoderSelector>` - The configured span encoder selector, if any.
```

--------------------------------

### ConstRegexPattern Methods

Source: https://docs.rs/wordchipper/latest/src/wordchipper/support/regex/regex_pattern.rs.html

Provides methods to access the underlying regex string, convert to a `RegexPattern`, and compile the pattern. Use `as_str` to get the pattern string, `to_pattern` to convert to a dynamic `RegexPattern`, and `compile` to get a `RegexWrapper`.

```rust
impl ConstRegexPattern {
    /// Get the underlying regex pattern.
    ///
    /// ## Returns
    /// The regex pattern string slice.
    pub const fn as_str(&self) -> &str {
        match self {
            Self::Basic(pattern) => pattern,
            Self::Fancy(pattern) => pattern,
        }
    }

    /// Convert to [`RegexPattern`]
    ///
    /// ## Returns
    /// A new `RegexWrapperPattern` instance.
    pub fn to_pattern(&self) -> RegexPattern {
        (*self).into()
    }

    /// Compile the regex pattern into a `RegexWrapper`.
    ///
    /// ## Returns
    /// A `Result` containing the compiled `RegexWrapper` or an `ErrorWrapper`.
    pub fn compile(&self) -> Result<RegexWrapper, ErrorWrapper> {
        RegexPattern::from(*self).compile()
    }
}
```

--------------------------------

### Using Entry API for In-place Manipulation

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashMap.html

Shows how to use the `entry` API to efficiently insert or modify values based on whether a key already exists. This is useful for counting occurrences.

```rust
use std::collections::HashMap;

let mut letters = HashMap::new();

for ch in "a short treatise on fungi".chars() {
    letters.entry(ch).and_modify(|counter| *counter += 1).or_insert(1);
}

assert_eq!(letters[&'s'], 2);
assert_eq!(letters[&'t'], 3);
assert_eq!(letters[&'u'], 1);
assert_eq!(letters.get(&'y'), None);
```

--------------------------------

### SpanMapVocab::span_map

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.SpanMapVocab.html

Get the SpanTokenMap from SpanMapVocab.

```APIDOC
## SpanMapVocab::span_map

### Description
Get the `SpanTokenMap`.

### Returns
A reference to the `SpanTokenMap`.
```

--------------------------------

### Loading Pretrained Models

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/mod.rs.html

Demonstrates how to load a pretrained model by reading its vocabulary and configuring spanners. It uses a simplified constructor to download, cache, and load the vocabulary.

```APIDOC
## Loading Pretrained Models

Loading a pre-trained model requires reading the vocabulary, as well as configuring the spanners (regex and special words) configuration.

For a number of pretrained models, simplified constructors are available to download, cache, and load the vocabulary.

### Example Usage

```rust,no_run
use std::sync::Arc;

use wordchipper::{
    Tokenizer,
    TokenizerOptions,
    UnifiedTokenVocab,
    disk_cache::WordchipperDiskCache,
    load_vocab,
};

fn example() -> wordchipper::WCResult<Arc<Tokenizer<u32>>> {
    let mut disk_cache = WordchipperDiskCache::default();
    let loaded = load_vocab("openai:o200k_harmony", &mut disk_cache)?;

    let tokenizer =
        TokenizerOptions::default().build(loaded.vocab().clone());

    Ok(tokenizer)
}
```

### Functions

- **`load_vocab(model_name: &str, cache: &mut impl WordchipperDiskCache)`**: Loads a pretrained vocabulary model. Returns a `WCResult` containing the loaded vocabulary information.
```

--------------------------------

### SpanMapVocab::byte_vocab

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.SpanMapVocab.html

Get the ByteMapVocab from SpanMapVocab.

```APIDOC
## SpanMapVocab::byte_vocab

### Description
Get the `ByteMapVocab`.

### Returns
A reference to the `ByteMapVocab`.
```

--------------------------------

### Load Vocabulary from Disk Path

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/utility/factories/struct.ConstVocabularyFactory.html

Loads the pretrained vocabulary directly from a file path on disk. Ensure the path points to a valid vocabulary file.

```rust
pub fn load_vocab_path<T: TokenType>( &self, path: impl AsRef<Path>, ) -> WCResult<UnifiedTokenVocab<T>>
```

--------------------------------

### Policy Combination: Or

Source: https://docs.rs/wordchipper/latest/wordchipper/support/resources/struct.ConstUrlResource.html

Creates a new Policy that returns Action::Follow if either self or another policy returns Action::Follow.

```APIDOC
## fn or<P, B, E>(self, other: P) -> Or<T, P>

### Description
Create a new `Policy` that returns `Action::Follow` if either `self` or `other` returns `Action::Follow`.

### Parameters
- **other** (P) - Description: The other policy to combine with.
```

--------------------------------

### UnifiedTokenVocab::byte_vocab

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.UnifiedTokenVocab.html

Get the {u8 -> T} ByteMapVocab.

```APIDOC
## UnifiedTokenVocab::byte_vocab

### Description
Get the `{ u8 -> T }` `ByteMapVocab`.
```

--------------------------------

### parallel

Source: https://docs.rs/wordchipper/latest/src/wordchipper/encoders/encoder_options.rs.html

Gets the configured parallelism value.

```APIDOC
## fn parallel(&self) -> bool

### Description
Gets the configured parallelism value. Enabling parallelism will request threaded implementations. See: [`is_concurrent`](Self::is_concurrent).

### Returns
- `bool` - `true` if parallelism is enabled, `false` otherwise.
```

--------------------------------

### get

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashMap.html

Returns a reference to the value corresponding to the key.

```APIDOC
## get<Q>(&self, k: &Q) -> Option<&V>

### Description
Returns a reference to the value corresponding to the key. The key may be any borrowed form of the map’s key type, but `Hash` and `Eq` on the borrowed form _must_ match those for the key type.

### Method
`get`

### Parameters
#### Path Parameters
- **k** (&Q) - Required - The key to look up in the map.

### Request Example
```rust
use std::collections::HashMap;

let mut map = HashMap::new();
map.insert(1, "a");
assert_eq!(map.get(&1), Some(&"a"));
assert_eq!(map.get(&2), None);
```
```

--------------------------------

### VocabQuery::new

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/factory/vocab_query.rs.html

Constructs a new VocabQuery instance from schema, path, and name components.

```APIDOC
## VocabQuery::new

### Description
Build a new query from structure.

### Signature
```rust
pub fn new(schema: Option<&str>, path: Option<&str>, name: &str) -> Self
```

### Parameters
* `schema` - An optional string slice representing the schema.
* `path` - An optional string slice representing the path.
* `name` - A string slice representing the name of the vocabulary.
```

--------------------------------

### UnifiedTokenVocab::special_vocab_mut

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.UnifiedTokenVocab.html

Get a mutable view of the SpecialVocab.

```APIDOC
## UnifiedTokenVocab::special_vocab_mut

### Description
Get a mutable view of the `SpecialVocab`
```

--------------------------------

### Create TextSpannerBuilder from Vocabulary

Source: https://docs.rs/wordchipper/latest/wordchipper/spanners/struct.TextSpannerBuilder.html

Initializes a `TextSpannerBuilder` by cloning the spanner configuration from a provided vocabulary. This is a convenient way to start building a `TextSpanner` with existing settings.

```rust
pub fn from_vocab(vocab: &UnifiedTokenVocab<T>) -> Self
```

--------------------------------

### PairMapVocab::new

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.PairMapVocab.html

Initializes a new PairMapVocab instance.

```APIDOC
## PairMapVocab::new

### Description
Initialize a `PairMapVocab`.

### Arguments
* `byte_vocab` - The byte vocabulary mapping.
* `pairs` - The pair token map.

### Returns
A `Result` containing the new `PairMapVocab` instance or an error.
```

--------------------------------

### UnifiedTokenVocab::span_vocab

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.UnifiedTokenVocab.html

Get the {Vec<u8> -> T} SpanMapVocab.

```APIDOC
## UnifiedTokenVocab::span_vocab

### Description
Get the `{ Vec<u8> -> T }` `SpanMapVocab`.
```

--------------------------------

### Get TokenPairMap

Source: https://docs.rs/wordchipper/latest/wordchipper/decoders/utility/struct.PairExpansionDecoder.html

Retrieves the TokenPairMap associated with the PairExpansionDecoder.

```rust
pub fn token_pairs(&self) -> &TokenPairMap<T>
```

--------------------------------

### take

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/openai/struct.OATokenizerIter.html

Creates an iterator that yields the first `n` elements.

```APIDOC
## take(self, n: usize) -> Take<Self>

### Description
Creates an iterator that yields the first `n` elements, or fewer if the underlying iterator ends sooner.

### Parameters
- **n**: usize - The maximum number of elements to yield.
```

--------------------------------

### parallel()

Source: https://docs.rs/wordchipper/latest/src/wordchipper/decoders/decoder_options.rs.html

Gets the configured parallelism value for the TokenDecoder.

```APIDOC
## fn parallel(&self) -> bool

### Description
Gets the configured parallelism value.

Enabling parallelism will request a threaded implementation.
```

--------------------------------

### CloneToUninit Trait Implementation (Nightly Only)

Source: https://docs.rs/wordchipper/latest/wordchipper/struct.TokenEncoderOptions.html

Documentation for the experimental `CloneToUninit` trait implementation, allowing cloning to uninitialized memory.

```APIDOC
## unsafe fn clone_to_uninit(&self, dest: *mut u8)

### Description
Performs copy-assignment from `self` to `dest`.
This is a nightly-only experimental API.

### Method
`clone_to_uninit`

### Parameters
- `dest` (*mut u8): A raw pointer to the destination memory location.

### Safety
This function is unsafe as it operates on raw pointers and assumes the destination is valid and sufficiently large.
```

--------------------------------

### unstable_current_thread_id_hash

Source: https://docs.rs/wordchipper/latest/wordchipper/support/concurrency/threads/index.html

Gets a hash of the current thread's ID.

```APIDOC
## Function: unstable_current_thread_id_hash

### Description
Current Thread -> u64 Pool.

### Signature
`fn unstable_current_thread_id_hash() -> u64`
```

--------------------------------

### Basic HashSet Operations

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashSet.html

Demonstrates the creation of a new HashSet, checking if it's empty, inserting an element, and checking its non-empty state.

```rust
use std::collections::HashSet;

let mut v = HashSet::new();
assert!(v.is_empty());
v.insert(1);
assert!(!v.is_empty());
```

--------------------------------

### Get VocabQuery Name

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/struct.VocabQuery.html

Retrieves the name of the vocabulary from the VocabQuery.

```rust
pub fn name(&self) -> &str
```

--------------------------------

### Load Base64 Unified Vocab Path

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/io/fn.load_base64_unified_vocab_path.html

Builds a `UnifiedTokenVocab` from a pretrained base64 vocab file. Requires a path to the file and text spanning configuration.

```rust
pub fn load_base64_unified_vocab_path<T: TokenType>(
    path: impl AsRef<Path>,
    spanning: TextSpanningConfig<T>,
) -> WCResult<UnifiedTokenVocab<T>>
```

--------------------------------

### Get VocabQuery Path

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/struct.VocabQuery.html

Retrieves the path associated with the VocabQuery.

```rust
pub fn path(&self) -> Option<&str>
```

--------------------------------

### Loading a Pretrained Model

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/index.html

Demonstrates how to load a pretrained model using its name and a disk cache. This involves initializing a disk cache, loading the vocabulary using `load_vocab`, and then building the tokenizer with the loaded vocabulary.

```APIDOC
## Loading a Pretrained Model

This example shows how to load a pretrained model by its identifier, such as `"openai:o200k_harmony"`.

### Function
`load_vocab`

### Parameters
- `model_name`: (string) The identifier of the pretrained model to load.
- `disk_cache`: (&mut WordchipperDiskCache) A mutable reference to the disk cache to use for storing and retrieving model data.

### Returns
A `WCResult` containing an `Arc<Tokenizer<u32>>` on success, or an error.

### Example
```rust
use std::sync::Arc;

use wordchipper::{
    Tokenizer,
    TokenizerOptions,
    UnifiedTokenVocab,
    disk_cache::WordchipperDiskCache,
    load_vocab,
};

fn example() -> wordchipper::WCResult<Arc<Tokenizer<u32>>> {
    let mut disk_cache = WordchipperDiskCache::default();
    let loaded = load_vocab("openai:o200k_harmony", &mut disk_cache)?;

    let tokenizer =
        TokenizerOptions::default().build(loaded.vocab().clone());

    Ok(tokenizer)
}
```
```

--------------------------------

### Get VocabQuery Schema

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/struct.VocabQuery.html

Retrieves the schema associated with the VocabQuery.

```rust
pub fn schema(&self) -> Option<&str>
```

--------------------------------

### Get VocabDescription Description

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/factory/struct.VocabDescription.html

Retrieves the textual description of the vocabulary.

```rust
pub fn description(&self) -> &str
```

--------------------------------

### Create ByteMapVocab from Token-to-Byte Map

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/byte_vocab.rs.html

Builds a ByteMapVocab from a hash map where keys are tokens and values are their corresponding byte values. Panics if the input is not a 1:1 bijection.

```rust
pub fn from_token_byte_map(token_byte_map: &TokenByteMap<T>) -> Self {
        let token_bytes = token_byte_map.clone();

        let ord_map: ByteTokenMap<T> = token_bytes.iter().map(|(&t, &b)| (b, t)).collect();
        assert_eq!(ord_map.len(), 256);

        let mut ord_items = ord_map.into_iter().collect::<Vec<_>>();
        ord_items.sort_by_key(|(b, _)| *b);

        let byte_tokens: [T; 256] = ord_items
            .into_iter()
            .map(|(_, t)| t)
            .collect::<Vec<_>>()
            .try_into()
            .unwrap();

        Self {
            byte_tokens,
            token_bytes,
        }
    }
```

--------------------------------

### Get LabeledVocab Description

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/factory/struct.LabeledVocab.html

Retrieves the description associated with the vocabulary.

```rust
pub fn description(&self) -> &VocabDescription
```

--------------------------------

### TextSpannerBuilder Configuration and Build

Source: https://docs.rs/wordchipper/latest/src/wordchipper/spanners/spanner_builder.rs.html

This section covers the methods available on the TextSpannerBuilder for configuring its behavior and for building the final TextSpanner instance.

```APIDOC
## TextSpannerBuilder Methods

### `max_pool()`

Retrieves the current maximum pool size configuration.

#### Returns
- `Option<NonZeroUsize>`: The current max pool size, if set.

### `set_max_pool(max_pool: NonZeroUsize)`

Sets the maximum pool size for the `TextSpanner`.

#### Parameters
- **max_pool** (`NonZeroUsize`): The desired maximum pool size.

### `with_max_pool(max_pool: NonZeroUsize)`

Sets the maximum pool size and returns the builder instance.

#### Parameters
- **max_pool** (`NonZeroUsize`): The desired maximum pool size.

#### Returns
- `Self`: The builder instance with the max pool size updated.

### `build()`

Builds a `TextSpanner` with the current configuration.

This method automatically selects the fastest available word lexer based on the configured pattern and enabled features. It falls back to a compiled regex if accelerated lexers are not available or suitable. The special lexer, if configured, is always built from its regex pattern.

#### Returns
- `Arc<dyn TextSpanner>`: An `Arc`-wrapped `TextSpanner` instance.
```

--------------------------------

### Initialize PairMapVocab

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.PairMapVocab.html

Creates a new PairMapVocab instance. Requires a ByteMapVocab for byte-to-token mapping and a PairTokenMap for token pair mappings. Returns a WCResult.

```rust
pub fn new( byte_vocab: ByteMapVocab<T>, pairs: PairTokenMap<T>, ) -> WCResult<Self>
```

--------------------------------

### get_disjoint_mut

Source: https://docs.rs/wordchipper/latest/wordchipper/type.WCHashMap.html

Attempts to get mutable references to N values in the map at once.

```APIDOC
## get_disjoint_mut<Q, const N: usize>( &mut self, ks: [&Q; N], ) -> [Option<&mut V>; N]

### Description
Attempts to get mutable references to `N` values in the map at once. Returns an array of length `N` with the results of each query. For soundness, at most one mutable reference will be returned to any value. `None` will be used if the key is missing. This method performs a check to ensure there are no duplicate keys, which currently has a time-complexity of O(n^2), so be careful when passing many keys.

### Method
`get_disjoint_mut`

### Parameters
#### Path Parameters
- **ks** ([&Q; N]) - Required - An array of keys to look up in the map.

### Panics
Panics if any keys are overlapping.

### Request Example
```rust
use std::collections::HashMap;

let mut libraries = HashMap::new();
libraries.insert("Bodleian Library".to_string(), 1602);
libraries.insert("Athenæum".to_string(), 1807);
libraries.insert("Herzogin-Anna-Amalia-Bibliothek".to_string(), 1691);
libraries.insert("Library of Congress".to_string(), 1800);

// Get Athenæum and Bodleian Library
let [Some(a), Some(b)] = libraries.get_disjoint_mut([
    "Athenæum",
    "Bodleian Library",
]) else { panic!() };

// Assert values of Athenæum and Library of Congress
let got = libraries.get_disjoint_mut([
    "Athenæum",
    "Library of Congress",
]);
assert_eq!(
    got,
    [
        Some(&mut 1807),
        Some(&mut 1800),
    ],
);

// Missing keys result in None
let got = libraries.get_disjoint_mut([
    "Athenæum",
    "New York Public Library",
]);
assert_eq!(
    got,
    [
        Some(&mut 1807),
        None
    ]
);
```

ⓘ```rust
use std::collections::HashMap;

let mut libraries = HashMap::new();
libraries.insert("Athenæum".to_string(), 1807);

// Duplicate keys panic!
let got = libraries.get_disjoint_mut([
    "Athenæum",
    "Athenæum",
]);
```
```

--------------------------------

### Get SpanTokenMap from SpanMapVocab

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.SpanMapVocab.html

Retrieves a reference to the underlying SpanTokenMap from the SpanMapVocab.

```rust
pub fn span_map(&self) -> &SpanTokenMap<T>
```

--------------------------------

### Create SpanMapVocab from ByteMapVocab

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/span_vocab.rs.html

Initializes a SpanMapVocab using an existing ByteMapVocab. The resulting vocabulary will contain 255 span entries, each one byte long. Panics if initialization fails.

```rust
pub fn from_byte_vocab(byte_vocab: ByteMapVocab<T>) -> Self {
        let span_map: SpanTokenMap<T> = byte_vocab.span_pairs().collect();

        Self::new(byte_vocab, span_map).unwrap()
    }
```

--------------------------------

### UnifiedTokenVocab::special_vocab

Source: https://docs.rs/wordchipper/latest/wordchipper/vocab/struct.UnifiedTokenVocab.html

Get the {Vec<u8> -> T} special token SpanMapVocab.

```APIDOC
## UnifiedTokenVocab::special_vocab

### Description
Get the `{ Vec<u8> -> T }` special token `SpanMapVocab`.
```

--------------------------------

### Create VocabQuery from String

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/factory/vocab_query.rs.html

Demonstrates creating a VocabQuery instance by parsing a string representation. Asserts the string representation and the constructed query match expected values.

```rust
let q = VocabQuery::from_str("xyz:foo/bar/vocab_name").unwrap();
assert_eq!(q.to_string(), "xyz:foo/bar/vocab_name");
assert_eq!(
    q,
    VocabQuery::new(Some("xyz"), Some("foo/bar"), "vocab_name")
);
```

--------------------------------

### Get SpecialVocab

Source: https://docs.rs/wordchipper/latest/wordchipper/struct.UnifiedTokenVocab.html

Provides access to the SpecialVocab, which is a SpanMapVocab for special tokens.

```rust
pub fn special_vocab(&self) -> &SpecialVocab<T>
```

--------------------------------

### Contraction Split Examples

Source: https://docs.rs/wordchipper/latest/src/wordchipper/spanners/span_lexers/logos/gpt2_family.rs.html

Demonstrates the usage of the `contraction_split` function with various inputs, including valid splits and non-split cases.

```rust
use wordchipper::spanners::span_lexers::logos::gpt2_family::contraction_split;

assert_eq!(contraction_split(b"'There"), Some(2));
assert_eq!(contraction_split(b"'llama"), Some(3));
assert_eq!(contraction_split(b"'t"), None);
assert_eq!(contraction_split(b"hello"), None);
```

--------------------------------

### Get VocabDescription ID

Source: https://docs.rs/wordchipper/latest/wordchipper/pretrained/factory/struct.VocabDescription.html

Retrieves the vocabulary ID from a VocabDescription instance.

```rust
pub fn id(&self) -> &VocabQuery
```

--------------------------------

### Load Pretrained Vocabulary Model

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/mod.rs.html

Demonstrates how to load a pretrained vocabulary model using a disk cache. This is useful for initializing a Tokenizer with a specific pretrained vocabulary.

```rust
use std::sync::Arc;

use wordchipper::{
    Tokenizer,
    TokenizerOptions,
    UnifiedTokenVocab,
    disk_cache::WordchipperDiskCache,
    load_vocab,
};

fn example() -> wordchipper::WCResult<Arc<Tokenizer<u32>>> {
    let mut disk_cache = WordchipperDiskCache::default();
    let loaded = load_vocab("openai:o200k_harmony", &mut disk_cache)?;

    let tokenizer =
        TokenizerOptions::default().build(loaded.vocab().clone());

    Ok(tokenizer)
}

```

--------------------------------

### Create VocabQuery with New Name Components

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/factory/vocab_query.rs.html

Illustrates creating a new VocabQuery instance with updated schema, path, and name components using the `with_name` method. Asserts that the new components are correctly applied.

```rust
let query = VocabQuery::new(Some("old"), Some("old_path"), "old_name")
    .with_name("new_provider:new_path/new_name");
assert_eq!(query.schema(), Some("new_provider"));
assert_eq!(query.path(), Some("new_path"));
assert_eq!(query.name(), "new_name");
```

--------------------------------

### Get TextSpanningConfig

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/unified_vocab.rs.html

Retrieves an immutable reference to the TextSpanningConfig associated with the vocabulary.

```rust
pub fn spanning(&self) -> &TextSpanningConfig<T> {
        &self.spanning
    }
```

--------------------------------

### oa_o200k_harmony_spanning_config

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/openai/spanning.rs.html

Get the TextSpanningConfig for the "o200k_harmony" pretrained vocabulary.

```APIDOC
## oa_o200k_harmony_spanning_config<T: TokenType>() -> TextSpanningConfig<T>

### Description
Retrieves the `TextSpanningConfig` for the `o200k_harmony` OpenAI tokenizer.

### Parameters
None

### Returns
A `TextSpanningConfig` instance configured for the `o200k_harmony` vocabulary.
```

--------------------------------

### Initialize UnifiedTokenVocab

Source: https://docs.rs/wordchipper/latest/wordchipper/struct.UnifiedTokenVocab.html

Creates a new UnifiedTokenVocab by combining TextSpanningConfig, SpanMapVocab, and PairMapVocab. This is the most comprehensive initialization method.

```rust
pub fn new(
  span_config: TextSpanningConfig<T>,
  span_vocab: SpanMapVocab<T>,
  pair_vocab: PairMapVocab<T>,
) -> WCResult<Self>
```

--------------------------------

### oa_o200k_base_spanning_config

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/openai/spanning.rs.html

Get the TextSpanningConfig for the "o200k_base" pretrained vocabulary.

```APIDOC
## oa_o200k_base_spanning_config<T: TokenType>() -> TextSpanningConfig<T>

### Description
Retrieves the `TextSpanningConfig` for the `o200k_base` OpenAI tokenizer.

### Parameters
None

### Returns
A `TextSpanningConfig` instance configured for the `o200k_base` vocabulary.
```

--------------------------------

### Load Base64 Unified Vocab Path

Source: https://docs.rs/wordchipper/latest/src/wordchipper/vocab/io/mod.rs.html

Demonstrates how to load a vocabulary from a base64 encoded file path. This is useful for initializing the tokenizer with a predefined vocabulary. Ensure the vocabulary file exists at the specified path.

```rust
use std::sync::Arc;

use wordchipper::{
    Tokenizer,
    TokenizerOptions,
    UnifiedTokenVocab,
    pretrained::openai::OA_O200K_BASE_PATTERN,
    spanners::TextSpanningConfig,
    vocab::io::load_base64_unified_vocab_path,
};

fn example() -> wordchipper::WCResult<Arc<Tokenizer<u32>>> {
    let vocab: Arc<UnifiedTokenVocab<u32>> =
        load_base64_unified_vocab_path(
            "vocab.tiktoken",
            TextSpanningConfig::from_pattern(OA_O200K_BASE_PATTERN),
        )
        .expect("failed to load vocab")
        .into();

    let tokenizer: Arc<Tokenizer<u32>> =
        TokenizerOptions::default().with_parallel(true).build(vocab);

    Ok(tokenizer)
}
```

--------------------------------

### oa_cl100k_base_spanning_config

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/openai/spanning.rs.html

Get the TextSpanningConfig for the "cl100k_base" pretrained vocabulary.

```APIDOC
## oa_cl100k_base_spanning_config<T: TokenType>() -> TextSpanningConfig<T>

### Description
Retrieves the `TextSpanningConfig` for the `cl100k_base` OpenAI tokenizer.

### Parameters
None

### Returns
A `TextSpanningConfig` instance configured for the `cl100k_base` vocabulary.
```

--------------------------------

### Load Pre-trained Vocab and Create Tokenizer

Source: https://docs.rs/wordchipper/latest/wordchipper/struct.UnifiedTokenVocab.html

Demonstrates loading a pre-trained vocabulary and creating a TokenEncoder, either with default settings or custom options like parallel processing.

```rust
let vocab: UnifiedTokenVocab<u32> = load_pretrained_vocab()?;

let encoder = TokenEncoderBuilder::default(vocab.clone());

let encoder = TokenEncoderBuilder::new(vocab.clone())
    .parallel(true)
    .build();
```

--------------------------------

### oa_p50k_edit_spanning_config

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/openai/spanning.rs.html

Get the TextSpanningConfig for the "p50k_edit" pretrained vocabulary.

```APIDOC
## oa_p50k_edit_spanning_config<T: TokenType>() -> TextSpanningConfig<T>

### Description
Retrieves the `TextSpanningConfig` for the `p50k_edit` OpenAI tokenizer.

### Parameters
None

### Returns
A `TextSpanningConfig` instance configured for the `p50k_edit` vocabulary.
```

--------------------------------

### Pointable Methods

Source: https://docs.rs/wordchipper/latest/wordchipper/struct.Tokenizer.html

Methods for pointer manipulation and initialization.

```APIDOC
## Pointable Methods

### const ALIGN

#### Description
The alignment of the pointer.

### unsafe fn init

#### Description
Initializes a with the given initializer.

#### Method Signature
`unsafe fn init(init: <T as Pointable>::Init) -> usize`

### unsafe fn deref

#### Description
Dereferences the given pointer.

#### Method Signature
`unsafe fn deref<'a>(ptr: usize) -> &'a T`

### unsafe fn deref_mut

#### Description
Mutably dereferences the given pointer.

#### Method Signature
`unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T`

### unsafe fn drop

#### Description
Drops the object pointed to by the given pointer.

#### Method Signature
`unsafe fn drop(ptr: usize)`
```

--------------------------------

### oa_p50k_base_spanning_config

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/openai/spanning.rs.html

Get the TextSpanningConfig for the "p50k_base" pretrained vocabulary.

```APIDOC
## oa_p50k_base_spanning_config<T: TokenType>() -> TextSpanningConfig<T>

### Description
Retrieves the `TextSpanningConfig` for the `p50k_base` OpenAI tokenizer.

### Parameters
None

### Returns
A `TextSpanningConfig` instance configured for the `p50k_base` vocabulary.
```

--------------------------------

### Create LexerTextSpanner from Configuration

Source: https://docs.rs/wordchipper/latest/src/wordchipper/spanners/span_lexers/lexer_spanner.rs.html

Instantiates a LexerTextSpanner using a TextSpanningConfig. It compiles the main pattern and optionally a special pattern.

```rust
fn from_config<T: TokenType>(config: &TextSpanningConfig<T>) -> LexerTextSpanner {
        LexerTextSpanner::new(
            Arc::new(config.pattern().clone().compile().unwrap()),
            config
                .special_pattern()
                .map(|p| Arc::new(p.compile().unwrap()) as Arc<dyn SpanLexer>),
        )
    }
```

--------------------------------

### oa_r50k_base_spanning_config

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/openai/spanning.rs.html

Get the TextSpanningConfig for the "r50k_base" pretrained vocabulary.

```APIDOC
## oa_r50k_base_spanning_config<T: TokenType>() -> TextSpanningConfig<T>

### Description
Retrieves the `TextSpanningConfig` for the `r50k_base` OpenAI tokenizer.

### Parameters
None

### Returns
A `TextSpanningConfig` instance configured for the `r50k_base` vocabulary.
```

--------------------------------

### CloneToUninit

Source: https://docs.rs/wordchipper/latest/wordchipper/encoders/token_span_encoder/span_encoders/struct.SpanEncoderSelectorIter.html

Experimental API for copying data to uninitialized memory.

```APIDOC
## unsafe fn clone_to_uninit(&self, dest: *mut u8)

### Description
Performs copy-assignment from `self` to `dest`.

### Method
`clone_to_uninit`

### Note
This is a nightly-only experimental API.
```

--------------------------------

### effective_span_encoder

Source: https://docs.rs/wordchipper/latest/src/wordchipper/encoders/encoder_options.rs.html

Gets the effective span encoder selector based on configuration.

```APIDOC
## fn effective_span_encoder(&self) -> SpanEncoderSelector

### Description
Gets the effective span encoder selector. Will return any explicit setting, otherwise will select based upon parallel and concurrency settings.

### Returns
- `SpanEncoderSelector` - The determined span encoder selector.
```

--------------------------------

### TryFrom Methods

Source: https://docs.rs/wordchipper/latest/wordchipper/struct.Tokenizer.html

Methods for attempting conversions.

```APIDOC
## TryFrom Methods

### type Error

#### Description
The type returned in the event of a conversion error.

### fn try_from

#### Description
Performs the conversion.

#### Method Signature
`fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>`
```

--------------------------------

### VocabQuery Constructor and Getters

Source: https://docs.rs/wordchipper/latest/src/wordchipper/pretrained/factory/vocab_query.rs.html

Provides methods to create a new VocabQuery instance and retrieve its schema, path, and name components. Schema and path are returned as Option<&str>.

```rust
/// Build a new query from structure.
pub fn new(
    schema: Option<&str>,
    path: Option<&str>,
    name: &str,
) -> Self {
    Self {
        schema: schema.map(|s| s.to_string()),
        path: path.map(|s| s.to_string()),
        name: name.to_string(),
    }
}

/// Get the schema.
pub fn schema(&self) -> Option<&str> {
    self.schema.as_deref()
}

/// Get the path.
pub fn path(&self) -> Option<&str> {
    self.path.as_deref()
}

/// Get the name.
pub fn name(&self) -> &str {
    &self.name
}
```

--------------------------------

### offset_range

Source: https://docs.rs/wordchipper/latest/src/wordchipper/support/ranges.rs.html

Adds a specified offset to both the start and end of a given range.

```APIDOC
## offset_range

### Description
Adds an offset to the start and end of a [`Range<usize>`].

### Signature
```rust
pub fn offset_range<Idx>(range: Range<usize>, offset: usize) -> Range<usize>
where
    Idx: PrimInt,
```

### Parameters

* `range` (Range<usize>) - The original range to offset.
* `offset` (usize) - The value to add to the start and end of the range.

### Returns

A new `Range<usize>` with the offset applied.

### Example

```rust
use wordchipper::support::ranges::offset_range;

let original_range = 0..10;
let offset = 5;
let new_range = offset_range(original_range, offset);

assert_eq!(new_range, 5..15);
```
```

--------------------------------

### Build and Use a Token Decoder

Source: https://docs.rs/wordchipper/latest/src/wordchipper/decoders/mod.rs.html

Demonstrates how to construct a `TokenDecoder` using `TokenDecoderOptions` and then use it to decode batches of tokens into strings. Ensure the `TokenType` generic parameter matches your vocabulary type.

```rust
use std::sync::Arc;

use wordchipper::{
    TokenDecoder,
    TokenDecoderOptions,
    TokenType,
    UnifiedTokenVocab,
};

fn example<T: TokenType>(
    vocab: Arc<UnifiedTokenVocab<T>>,
    batch: &[Vec<T>],
) -> Vec<String> {
    let decoder = TokenDecoderOptions::default().build(vocab);

    let slices: Vec<&[T]> = batch.iter().map(|v| v.as_ref()).collect();

    decoder
        .try_decode_batch_to_strings(&slices)
        .unwrap()
        .unwrap()
}
```
Tokens:
Raw