Toktrie

https://docs.rs/toktrie
Admin
Toktrie is a highly optimized Rust utility library providing efficient token trie implementations...
Tokens:95,305
Snippets:840
Trust Score:9.1
Update:5 months ago
Show doc for...
Tokens:
Raw
### byte_allowed Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example description for `byte_allowed`, explaining its role in checking stack transitions.

```rust
check if stack.top() transitions via byte to a viable state
```

--------------------------------

### collapse Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example implementation of `collapse` which reduces the stack to its top element.

```rust
“Collapse” the stack so that it consists only of its former top element. X = stack.top(); stack.empty(); stack.push(X)
```

--------------------------------

### trie_started Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example description for `trie_started`, indicating it's called at the beginning of trie iteration.

```rust
Called when iteration over the trie is started
```

--------------------------------

### try_push_byte Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example description for `try_push_byte`, highlighting its performance optimization by combining push and allow operations.

```rust
This combines `push_byte` and `byte_allowed` into one function for performance.
```

--------------------------------

### save_stats Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example description for `save_stats`, used for recording statistics like nodes walked.

```rust
save_stats(&mut self, _nodes_walked: usize)
```

--------------------------------

### get_error Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example description for `get_error`, used to retrieve any reported errors.

```rust
Check if there are any errors to be reported to the user.
```

--------------------------------

### Create a new SimpleVob

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Initializes a new, empty SimpleVob. This is useful when starting with a fresh set of token permissions.

```rust
pub fn new() -> Self
```

--------------------------------

### pop_bytes Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example implementation of `pop_bytes` which removes a specified number of bytes from the stack.

```rust
for _ in 0..num { stack.pop() }
```

--------------------------------

### trie_finished Implementation Example

Source: https://docs.rs/toktrie/latest/toktrie/trait.Recognizer.html

Example implementation of `trie_finished` which handles stack cleanup when trie iteration completes.

```rust
Called when iteration over the trie is finished Stack has exactly one element then, except when iteration started from non-root node. In that case, the stack may have more than one element, and trie_finished() needs to pop the excessive elements.
```

--------------------------------

### Handle Trie Started Event

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/struct.StackRecognizer.html

Called when iteration over the trie is started.

```rust
fn trie_started(&mut self, _dbg_lbl: &str)
```

--------------------------------

### Implementing `MyTrait` for `Arc<UdpSocket>`

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

This example shows how to implement a custom trait `MyTrait` that requires `AsRawFd` on an `Arc<UdpSocket>`. This demonstrates enabling traits that depend on `AsRawFd` for types wrapped in `Arc`.

```rust
use std::net::UdpSocket;
use std::sync::Arc;
trait MyTrait: AsRawFd {
}
impl MyTrait for Arc<UdpSocket> {}
impl MyTrait for Box<UdpSocket> {}
```

--------------------------------

### Zeroable Implementation for U32Pair

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Details the `Zeroable` trait implementation for `U32Pair`, providing a way to get a zero-initialized instance.

```APIDOC
### impl Zeroable for U32Pair
Source

#### fn zeroed() -> Self
Calls `zeroed`. Read more
Source
```

--------------------------------

### Get Stream Position

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Returns the current seek position from the start of the stream.

```APIDOC
## fn stream_position(&mut self) -> Result<u64, Error>

### Description
Returns the current seek position from the start of the stream.

### Returns
- `Result<u64, Error>`: The current stream position if successful, or an Error if it fails.
```

--------------------------------

### Get All Subtokens

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Retrieves all TokenIds that can be formed by traversing the trie starting from any position within the given byte slice. This is useful for finding all possible tokenizations of substrings.

```rust
pub fn all_subtokens(&self, bytes: &[u8]) -> Vec<TokenId> {
        let mut r = Vec::new();
        for i in 0..bytes.len() {
            let mut n = self.root();
            for &b in &bytes[i..] {
                n = match self.child_at_byte(n, b) {
                    Some(n) => n,
                    None => break,
                };
                if let Some(tok) = n.token_id() {
                    r.push(tok);
                }
            }
        }
        r
    }
```

--------------------------------

### Get all special tokens

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Collects all special tokens present in the TokTrie. It traverses the trie starting from the special token prefix node and collects any nodes that represent a token.

```rust
pub fn get_special_tokens(&self) -> Vec<TokenId> {
        let mut res = Vec::new();
        let pref_node = self
            .child_at_byte(self.root(), TokTrie::SPECIAL_TOKEN_MARKER)
            .expect("missing special token prefix");
        let mut stack = vec![pref_node];
        while let Some(n) = stack.pop() {
            for c in self.node_children(n) {
                if let Some(tok) = c.token_id() {
                    res.push(tok);
                    if res.len() > Self::MAX_DBG_TOKENS + 1 {
                        break;
                    }
                }
                stack.push(c);
            }
        }
        res.remove(0);
        res
    }
```

--------------------------------

### Define FunctionalRecognizer Trait

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/trait.FunctionalRecognizer.html

Defines the core methods for a functional recognizer: `initial` to get the starting state and `try_append` to transition between states based on input bytes. The `get_error` method is provided for diagnostic messages.

```rust
pub trait FunctionalRecognizer<S: Copy> {
    // Required methods
    fn initial(&self) -> S;
    fn try_append(&self, state: S, byte: u8) -> Option<S>;

    // Provided method
    fn get_error(&self, _state: S) -> Option<String> { ... }
}
```

--------------------------------

### SimpleVob::new

Source: https://docs.rs/toktrie/latest/src/toktrie/svob.rs.html

Creates a new, empty SimpleVob.

```APIDOC
## SimpleVob::new

### Description
Creates a new, empty `SimpleVob`.

### Method
`SimpleVob::new()`

### Returns
A new `SimpleVob` instance with zero size and an empty data vector.
```

--------------------------------

### Blanket Implementations

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Highlights blanket implementations available for U32Pair, such as Any, Borrow, BorrowMut, CheckedBitPattern, CloneToUninit, From, Into, ToOwned, TryFrom, TryInto, AnyBitPattern, and NoUninit.

```APIDOC
## Blanket Implementations
### impl<T> Any for T
#### fn type_id(&self) -> TypeId
### impl<T> Borrow<T> for T
#### fn borrow(&self) -> &T
### impl<T> BorrowMut<T> for T
#### fn borrow_mut(&mut self) -> &mut T
### impl<T> CheckedBitPattern for T
#### type Bits = T
#### fn is_valid_bit_pattern(_bits: &T) -> bool
### impl<T> CloneToUninit for T
#### unsafe fn clone_to_uninit(&self, dest: *mut u8)
### impl<T> From<T> for T
#### fn from(t: T) -> T
### impl<T, U> Into<U> for T
#### fn into(self) -> U
### impl<T> ToOwned for T
#### type Owned = T
#### fn to_owned(&self) -> T
#### fn clone_into(&self, target: &mut T)
### impl<T, U> TryFrom<U> for T
#### type Error = Infallible
#### fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
### impl<T, U> TryInto<U> for T
#### type Error = <U as TryFrom<T>>::Error
#### fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>
### impl<T> AnyBitPattern for T
### impl<T> NoUninit for T
```

--------------------------------

### TokEnvWithTrie::new

Source: https://docs.rs/toktrie/latest/toktrie/struct.TokEnvWithTrie.html

Constructs a new TokEnvWithTrie instance.

```APIDOC
## TokEnvWithTrie::new

### Description
Constructs a new `TokEnvWithTrie` instance.

### Signature
```rust
pub fn new(base_env: TokEnv, tok_trie: TokTrie) -> Self
```

### Parameters
* `base_env`: The base `TokEnv` to use.
* `tok_trie`: The `TokTrie` to associate with this environment.
```

--------------------------------

### Create By Reference Adapter for Write

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Creates a "by reference" adapter for a Write instance.

```APIDOC
## fn by_ref(&mut self) -> &mut Self

### Description
Creates a "by reference" adapter for this instance of `Write`.

### Returns
- `&mut Self`: A mutable reference to the adapter.
```

--------------------------------

### SimpleVob Initialization and Allocation

Source: https://docs.rs/toktrie/latest/src/toktrie/svob.rs.html

Provides methods for creating new SimpleVob instances, allocating them with a specific size, and initializing them from slices or with all bits set.

```rust
const BITS: usize = 32;

// Compile-time assertion: BITS must be 32 because the implementation uses Vec<u32>
// and hardcoded bit shift operations (>> 5 and & 31) that only work for 32-bit words
const _: () = assert!(
    BITS == 32,
    "BITS must be 32 to match Vec<u32> storage and bit shift operations"
);

impl SimpleVob {
    pub fn new() -> Self {
        Self {
            data: Vec::new(),
            size: 0,
        }
    }

    pub fn from_slice(bits: &[bool]) -> Self {
        let mut r = Self::alloc(bits.len());
        for (idx, b) in bits.iter().enumerate() {
            r.set(idx, *b);
        }
        r
    }

    pub fn alloc(size: usize) -> Self {
        let mut r = Self::new();
        r.resize(size);
        r
    }

    pub fn alloc_ones(size: usize) -> Self {
        let mut r = Self::alloc(size);
        r.set_all(true);
        r
    }

    pub fn alloc_with_capacity(size: usize, capacity: usize) -> Self {
        let mut r = Self::new();
        assert!(size <= capacity);
        r.resize(capacity);
        r.size = size;
        r
    }
```

--------------------------------

### Create a sampling Branch

Source: https://docs.rs/toktrie/latest/src/toktrie/lib.rs.html

Initializes a Branch configured for sampling, specifying the set of allowed tokens and an optional temperature.

```rust
pub fn sample(set: S, temperature: Option<f32>) -> Self {
        Branch {
            sample_mask: Some(set),
            temperature,
            splices: vec![],
        }
    }
```

--------------------------------

### fn initial()

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/trait.FunctionalRecognizer.html

Returns the initial state of the recognizer, before any bytes have been processed.

```APIDOC
## fn initial()

### Description
Returns the initial state of the recognizer, before any bytes have been processed.

### Signature
```rust
fn initial(&self) -> S
```
```

--------------------------------

### Get Error from StackRecognizer

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/struct.StackRecognizer.html

Checks if there are any errors to be reported to the user.

```rust
fn get_error(&mut self) -> Option<String>
```

--------------------------------

### Try Creating Arc<T> with Allocator API

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Shows how to attempt creating an Arc<T> using the allocator API, returning a Result that indicates success or allocation failure. Requires the `allocator_api` feature.

```rust
#![feature(allocator_api)]
use std::sync::Arc;

let five = Arc::try_new(5)?;

```

--------------------------------

### Get Root Node

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Returns a reference to the root node of the trie.

```rust
pub fn root(&self) -> &TrieNode {
        &self.nodes[0]
    }
```

--------------------------------

### Branch::sample Constructor

Source: https://docs.rs/toktrie/latest/toktrie/struct.Branch.html

Creates a Branch configured for sampling with a specified token set and optional temperature.

```rust
pub fn sample(set: S, temperature: Option<f32>) -> Self
```

--------------------------------

### type_id

Source: https://docs.rs/toktrie/latest/toktrie/struct.TokTrie.html

Gets the `TypeId` of `self`. This method is part of the `Any` trait implementation.

```APIDOC
## fn type_id(&self) -> TypeId

### Description
Gets the `TypeId` of `self`.

### Method
`type_id`

### Parameters
None

### Returns
- `TypeId`: The type identifier of the object.
```

--------------------------------

### Configuration Methods

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Methods for configuring the TokTrie, such as setting EOS tokens or updating information.

```APIDOC
## `with_eos_token`

### Description
Creates a new TokTrie with a single specified End-Of-Sequence (EOS) token.

### Signature
`pub fn with_eos_token(&self, eos_token: TokenId) -> Self`

## `with_eos_tokens`

### Description
Creates a new TokTrie with a slice of specified End-Of-Sequence (EOS) tokens. Asserts that the provided tokens are not empty and are within the vocabulary range.

### Signature
`pub fn with_eos_tokens(&self, eos_tokens: &[TokenId]) -> Self`

## `with_info`

### Description
Creates a new TokTrie by updating the TokRxInfo and setting the primary EOS token from the new info.

### Signature
`pub fn with_info(&self, info: TokRxInfo) -> Self`

## `build_chat_mode_trie`

### Description
Builds a TokTrie configured for chat mode, using the `tok_end_of_turn` from `info` if available, otherwise falling back to the primary EOS token.

### Signature
`pub fn build_chat_mode_trie(&self) -> Self`
```

--------------------------------

### SimpleVob Size and Utility Methods

Source: https://docs.rs/toktrie/latest/src/toktrie/svob.rs.html

Includes methods for checking the size of the SimpleVob, its emptiness, and counting the number of set bits.

```rust
    pub fn len(&self) -> usize {
        self.size
    }

    pub fn is_empty(&self) -> bool {
        self.size == 0
    }

    pub fn num_set(&self) -> usize {
        self.data.iter().map(|x| x.count_ones() as usize).sum()
    }
```

--------------------------------

### token_id_at_bytes

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Finds the TokenId associated with a sequence of bytes starting from the root of the trie.

```APIDOC
## token_id_at_bytes

### Description
Finds the TokenId associated with a sequence of bytes starting from the root of the trie.

### Signature
`pub fn token_id_at_bytes(&self, bytes: &[u8]) -> Option<TokenId>`

### Parameters
- `bytes`: A slice of bytes to search for.

### Returns
An `Option` containing the `TokenId` if found, otherwise `None`.
```

--------------------------------

### Get Underlying Recognizer

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/struct.StackRecognizer.html

Returns a shared reference to the `FunctionalRecognizer` wrapped by `StackRecognizer`.

```rust
pub fn recognizer(&self) -> &R
```

--------------------------------

### Converting `Cow<'_, str>` to `Arc<str>`

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Shows how to create an `Arc<str>` from a `Cow<'_, str>` by copying its content. This is useful when you have data that might be borrowed or owned and you want to share it atomically.

```rust
let cow: Cow<'_, str> = Cow::Borrowed("eggplant");
let shared: Arc<str> = Arc::from(cow);
assert_eq!("eggplant", &shared[..]);
```

--------------------------------

### Get Stream Length

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Returns the length of the stream in bytes. This is a nightly-only experimental API.

```APIDOC
## fn stream_len(&mut self) -> Result<u64, Error>

### Description
Returns the length of this stream (in bytes). This is a nightly-only experimental API.

### Returns
- `Result<u64, Error>`: The length of the stream if successful, or an Error if it fails.
```

--------------------------------

### TokRxInfo Implementations

Source: https://docs.rs/toktrie/latest/toktrie/struct.TokRxInfo.html

Provides methods for creating and manipulating TokRxInfo instances.

```APIDOC
### impl TokRxInfo

#### pub fn new(vocab_size: u32, tok_eos: TokenId) -> Self
Creates a new `TokRxInfo` instance.

#### pub fn from_bin(info: &BinTokRxInfo) -> Self
Creates a `TokRxInfo` instance from binary data.

#### pub fn to_bin(&self) -> BinTokRxInfo
Converts the `TokRxInfo` instance to binary data.
```

--------------------------------

### Get Maximum Token Length

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Returns the maximum length of any token stored in the trie.

```rust
pub fn max_token_len(&self) -> usize {
        self.max_token_len
    }
```

--------------------------------

### Create and Convert Arc with Allocator

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Demonstrates creating an `Arc` with a specific allocator and converting it to and from a raw pointer. Ensure the pointer is converted back to an `Arc` to prevent memory leaks.

```rust
#![feature(allocator_api)]

use std::sync::Arc;
use std::alloc::System;

let x = Arc::new_in("hello".to_owned(), System);
let (x_ptr, alloc) = Arc::into_raw_with_allocator(x);

unsafe {
    // Convert back to an `Arc` to prevent leak.
    let x = Arc::from_raw_in(x_ptr, System);
    assert_eq!(&*x, "hello");

    // Further calls to `Arc::from_raw(x_ptr)` would be memory-unsafe.
}

// The memory was freed when `x` went out of scope above, so `x_ptr` is now dangling!
```

--------------------------------

### Check if Token is Allowed

Source: https://docs.rs/toktrie/latest/src/toktrie/svob.rs.html

Checks if a token ID is allowed by calling the `get` method.

```rust
pub fn is_allowed(&self, tok: TokenId) -> bool {
        self.get(tok as usize)
    }
```

--------------------------------

### Get an iterator for SimpleVob

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Returns an iterator over the SimpleVob, yielding pairs of (allowed_status, token_id).

```rust
pub fn iter(&self) -> SimpleVobIter<'_>
```

--------------------------------

### Basic Arc<T> Creation

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Demonstrates the fundamental creation of an Arc<T> with a simple value.

```rust
use std::sync::Arc;

let five = Arc::new(5);

```

--------------------------------

### Get Mutable Underlying Recognizer

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/struct.StackRecognizer.html

Returns a mutable reference to the `FunctionalRecognizer` wrapped by `StackRecognizer`.

```rust
pub fn recognizer_mut(&mut self) -> &mut R
```

--------------------------------

### Clone Implementation for U32Pair

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Demonstrates the `Clone` trait implementation for `U32Pair`, allowing instances to be duplicated.

```APIDOC
### impl Clone for U32Pair
Source

#### fn clone(&self) -> U32Pair
Returns a duplicate of the value. Read more
1.0.0 · Source

#### fn clone_from(&mut self, source: &Self)
Performs copy-assignment from `source`. Read more
Source
```

--------------------------------

### AsRawFd Implementation

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Provides a way to get the raw file descriptor from an Arc<T> where T implements AsRawFd.

```APIDOC
## fn as_fd(&self) -> BorrowedFd<'_>

Borrows the file descriptor.

Source: 1.63.0
```

```APIDOC
## fn as_raw_fd(&self) -> i32

Extracts the raw file descriptor.

Source: 1.5.0
```

--------------------------------

### Creating an Arc with a Custom Allocator

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Shows how to construct a new `Arc` using a specified allocator, utilizing the `allocator_api` feature.

```rust
#![feature(allocator_api)]

use std::sync::Arc;
use std::alloc::System;

let five = Arc::new_in(5, System);
```

--------------------------------

### Create By Reference Adapter for Read

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Creates a "by reference" adapter for a Read instance.

```APIDOC
## fn by_ref(&mut self) -> &mut Self

### Description
Creates a "by reference" adapter for this instance of `Read`.

### Returns
- `&mut Self`: A mutable reference to the adapter.
```

--------------------------------

### Get the length of the SimpleVob

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Returns the total number of bits (token IDs) represented by the SimpleVob.

```rust
pub fn len(&self) -> usize
```

--------------------------------

### Get unconditional splice

Source: https://docs.rs/toktrie/latest/src/toktrie/lib.rs.html

Returns the single Splice if the Branch has no sampling and exactly one Splice.

```rust
pub fn unconditional_splice(&self) -> Option<&Splice> {
        if self.splices.len() == 1 && self.splices[0].when_sampled.is_empty() {
            Some(&self.splices[0])
        } else {
            None
        }
    }
```

--------------------------------

### PartialEq Implementation for U32Pair

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Allows comparison of U32Pair instances for equality. The `eq` method tests for equality, and `ne` tests for inequality.

```rust
fn eq(&self, other: &U32Pair) -> bool
```

```rust
fn ne(&self, other: &Rhs) -> bool
```

--------------------------------

### from

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Creates a SimpleVob from a value of the same type.

```APIDOC
## fn from(t: T) -> T

### Description
Returns the argument unchanged.

### Method
`from`

### Parameters
- `t` (T): The value to be converted.

### Returns
- T: The original value.
```

--------------------------------

### Arc::weak_count

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Gets the number of Weak pointers pointing to the allocation. Useful for understanding the lifecycle of the allocation.

```APIDOC
## weak_count

### Description
Gets the number of `Weak` pointers to this allocation.

### Method
`Arc::weak_count`

### Parameters
- **this** (`&Arc<T, A>`) - The `Arc` to query the weak count for.

### Response
- `usize` - The number of `Weak` pointers.

### Safety
This method by itself is safe, but using it correctly requires extra care. Another thread can change the weak count at any time, including potentially between calling this method and acting on the result.

### Example
```rust
use std::sync::Arc;

let five = Arc::new(5);
let _weak_five = Arc::downgrade(&five);

// This assertion is deterministic because we haven't shared
// the `Arc` or `Weak` between threads.
assert_eq!(1, Arc::weak_count(&five));
```
```

--------------------------------

### PartialEq Implementation for U32Pair

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Illustrates the `PartialEq` trait implementation for `U32Pair`, allowing for equality comparisons.

```APIDOC
### impl PartialEq for U32Pair
Source

#### fn eq(&self, other: &U32Pair) -> bool
Tests for `self` and `other` values to be equal, and is used by `==`.
1.0.0 · Source

#### fn ne(&self, other: &Rhs) -> bool
Tests for `!=`. The default implementation is almost always sufficient, and should not be overridden without very good reason.
Source
```

--------------------------------

### Get Underlying Allocator

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Returns a reference to the allocator used by the Arc. This is an experimental API and an associated function.

```rust
use std::sync::Arc;
use std::alloc::System;

// Example usage would require an Arc created with a specific allocator.
// let my_alloc = System;
// let arc_with_alloc = Arc::new_in(5, my_alloc);
// let allocator_ref = Arc::allocator(&arc_with_alloc);
```

--------------------------------

### set_from

Source: https://docs.rs/toktrie/latest/src/toktrie/svob.rs.html

Copies the contents of another SimpleVob into this one.

```APIDOC
## set_from

### Description
Copies the bit data from the `other` `SimpleVob` into the current `SimpleVob`. Asserts that the sizes are equal.

### Signature
`pub fn set_from(&mut self, other: &SimpleVob)`

### Parameters
* `other`: The `SimpleVob` to copy data from.
```

--------------------------------

### Get All Tokens from TokTrie

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Collects all tokens stored in the TokTrie. It iterates through the vocabulary size and retrieves each token.

```rust
    pub fn all_tokens(&self) -> Vec<Vec<u8>> {
        (0..self.vocab_size())
            .map(|idx| self.token(idx as u32).to_vec())
            .collect()
    }
```

--------------------------------

### Get Token Allowed Status

Source: https://docs.rs/toktrie/latest/src/toktrie/svob.rs.html

Retrieves the allowed status of a token ID by checking its corresponding bit.

```rust
pub fn get(&self, idx: usize) -> bool {
        let byte_idx = idx / 32;
        let bit_idx = idx % 32;
        (self.data[byte_idx] & (1 << bit_idx)) != 0
    }
```

--------------------------------

### Try Creating Zeroed Arc<T> with Allocator API

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Illustrates attempting to create an Arc<T> with zeroed memory using the allocator API, returning a Result. Requires `unsafe` for `assume_init` and the `allocator_api` feature.

```rust
#![feature( allocator_api)]

use std::sync::Arc;

let zero = Arc::<u32>::try_new_zeroed()?;
let zero = unsafe { zero.assume_init() };

assert_eq!(*zero, 0);

```

--------------------------------

### Raw Pointer Conversion with Allocator

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Demonstrates converting an Arc to a raw pointer and back, managing allocation explicitly. Requires careful handling of reference counts and allocator.

```rust
#![feature(allocator_api)]

use std::sync::Arc;
use std::alloc::System;

let five = Arc::new_in(5, System);

unsafe {
    let (ptr, _alloc) = Arc::into_raw_with_allocator(five);
    Arc::increment_strong_count_in(ptr, System);

    // Those assertions are deterministic because we haven't shared
    // the `Arc` between threads.
    let five = Arc::from_raw_in(ptr, System);
    assert_eq!(2, Arc::strong_count(&five));
    Arc::decrement_strong_count_in(ptr, System);
    assert_eq!(1, Arc::strong_count(&five));
}

```

--------------------------------

### From<Cow<'a, B>> Implementation

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Allows conversion from Cow<'a, B> to Arc<B>.

```APIDOC
## fn from(cow: Cow<'a, B>) -> Arc<B>

Creates an atomically reference-counted pointer from a clone-on-write pointer by copying its content.

### Example
```rust
let cow: Cow<'_, str> = Cow::Borrowed("eggplant");
let shared: Arc<str> = Arc::from(cow);
assert_eq!("eggplant", &shared[..]);
```

Source: 1.6.0
```

--------------------------------

### Implement FunctionalRecognizer for AnythingGoes

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/trait.FunctionalRecognizer.html

An example implementation of the FunctionalRecognizer trait for the `AnythingGoes` type, indicating that any byte sequence is accepted.

```rust
impl FunctionalRecognizer<()> for AnythingGoes
```

--------------------------------

### Arc::new_uninit_in

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Constructs a new Arc with uninitialized contents using the provided allocator. This is a nightly-only experimental API.

```APIDOC
## pub fn new_uninit_in(alloc: A) -> Arc<MaybeUninit<T>, A>

### Description
Constructs a new `Arc` with uninitialized contents in the provided allocator.

### Method
`Arc::new_uninit_in`

### Parameters
- **alloc** (`A`) - The allocator to use for the Arc's allocation.

### Returns
- `Arc<MaybeUninit<T>, A>` - A new Arc instance with uninitialized contents.

### Examples
```rust
#![feature(get_mut_unchecked)]
#![feature(allocator_api)]

use std::sync::Arc;
use std::alloc::System;

let mut five = Arc::<u32, _>::new_uninit_in(System);

let five = unsafe {
    // Deferred initialization:
    Arc::get_mut_unchecked(&mut five).as_mut_ptr().write(5);

    five.assume_init()
};

assert_eq!(*five, 5)
```
```

--------------------------------

### Checking if a token is allowed

Source: https://docs.rs/toktrie/latest/src/toktrie/svob.rs.html

The `is_allowed` method is a convenience wrapper around `get` for checking if a specific `TokenId` is allowed.

```APIDOC
pub fn is_allowed(&self, tok: TokenId) -> bool
```

--------------------------------

### get_special_tokens

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Retrieves a list of all special tokens present in the TokTrie. This method traverses the trie starting from the special token prefix.

```APIDOC
## get_special_tokens

### Description
Retrieves a vector of all `TokenId`s that are designated as special tokens within the trie.

### Method
`pub fn get_special_tokens(&self) -> Vec<TokenId>`

### Returns
- `Vec<TokenId>` - A vector containing the `TokenId`s of all special tokens.
```

--------------------------------

### Recognizer::trie_finished Implementation

Source: https://docs.rs/toktrie/latest/toktrie/struct.AnythingGoes.html

Called when trie iteration finishes. Handles stack cleanup, especially when starting from a non-root node.

```rust
fn trie_finished(&mut self)
Called when iteration over the trie is finished Stack has exactly one element then, except when iteration started from non-root node. In that case, the stack may have more than one element, and trie_finished() needs to pop the excessive elements.
```

--------------------------------

### Arc::new_uninit

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Constructs a new Arc with uninitialized contents.

```APIDOC
## pub fn new_uninit() -> Arc<MaybeUninit<T>>

### Description
Constructs a new `Arc` with uninitialized contents.

### Method
`new_uninit`

### Request Example
```rust
use std::sync::Arc;

let mut five = Arc::<u32>::new_uninit();

// Deferred initialization:
Arc::get_mut(&mut five).unwrap().write(5);

let five = unsafe { five.assume_init() };

assert_eq!(*five, 5)
```
```

--------------------------------

### Arc::strong_count

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Gets the number of strong (Arc) pointers pointing to the allocation. Indicates how many strong references exist.

```APIDOC
## strong_count

### Description
Gets the number of strong (`Arc`) pointers to this allocation.

### Method
`Arc::strong_count`

### Parameters
- **this** (`&Arc<T, A>`) - The `Arc` to query the strong count for.

### Response
- `usize` - The number of strong (`Arc`) pointers.

### Safety
This method by itself is safe, but using it correctly requires extra care. Another thread can change the strong count at any time, including potentially between calling this method and acting on the result.

### Example
```rust
use std::sync::Arc;

let five = Arc::new(5);
let _also_five = Arc::clone(&five);

// This assertion is deterministic because we haven't shared
// the `Arc` between threads.
assert_eq!(2, Arc::strong_count(&five));
```
```

--------------------------------

### CloneToUninit Implementation

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Provides an unsafe method to clone data into an uninitialized memory location. This is an experimental nightly-only API.

```rust
unsafe fn clone_to_uninit(&self, dest: *mut u8)
```

--------------------------------

### Get the value of a specific bit

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Returns whether the token ID at the given index is allowed (true) or disallowed (false).

```rust
pub fn get(&self, idx: usize) -> bool
```

--------------------------------

### greedy_tokenize

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Tokenizes a byte slice using a greedy approach. It repeatedly finds the longest matching token starting from the current position.

```APIDOC
## greedy_tokenize

### Description
Tokenizes a byte slice into a sequence of `TokenId`s using a greedy longest-match strategy. It iterates through the byte slice, finding the longest possible token at each step.

### Method
`pub fn greedy_tokenize(&self, bytes: &[u8]) -> Vec<TokenId>`

### Parameters
- **bytes** (`&[u8]`) - The byte slice to tokenize.

### Returns
- `Vec<TokenId>` - A vector of `TokenId`s representing the tokenized input.
```

--------------------------------

### From Implementation

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Allows conversion from a type to itself.

```rust
fn from(t: T) -> T
```

--------------------------------

### Get Node Children Iterator

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Provides an iterator over the children of a given trie node. The iterator yields references to child TrieNodes.

```rust
pub fn node_children(&self, n: &TrieNode) -> NodeChildren<'_> {
        let off = self.node_offset(n);
        NodeChildren {
            trie: self,
            current_offset: off + 1,
            end_offset: off + n.subtree_size(),
        }
    }
```

--------------------------------

### try_from

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Attempts to convert a value into a SimpleVob, returning a Result.

```APIDOC
## fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

### Description
Performs the conversion from type U to type T.

### Method
`try_from`

### Parameters
- `value` (U): The value to be converted.

### Returns
- Result<T, <T as TryFrom<U>>::Error>: Ok(T) if the conversion is successful, or an error if it fails.
```

--------------------------------

### Check if a token is allowed

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Returns true if the specified token ID is allowed, false otherwise. This is a convenient wrapper around `get`.

```rust
pub fn is_allowed(&self, tok: u32) -> bool
```

--------------------------------

### ApproximateTokEnv::new

Source: https://docs.rs/toktrie/latest/toktrie/struct.ApproximateTokEnv.html

Creates a new ApproximateTokEnv with the provided TokTrie.

```APIDOC
## ApproximateTokEnv::new

### Description
Creates a new ApproximateTokEnv with the provided TokTrie.

### Signature
```rust
pub fn new(trie: TokTrie) -> Self
```

### Parameters
* **trie** (TokTrie) - The TokTrie to use for tokenization.
```

--------------------------------

### Traverse Trie by Byte Sequence

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Finds the TrieNode corresponding to a sequence of bytes starting from a given node. Returns None if the path does not exist.

```rust
pub fn child_at_bytes<'a>(&'a self, mut n: &'a TrieNode, bytes: &[u8]) -> Option<&'a TrieNode> {
        for &byte in bytes {
            n = self.child_at_byte(n, byte)?
        }
        Some(n)
    }
```

--------------------------------

### Clone Implementation for U32Pair

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Provides methods for cloning U32Pair instances. `clone` creates a duplicate, while `clone_from` performs assignment.

```rust
fn clone(&self) -> U32Pair
```

```rust
fn clone_from(&mut self, source: &Self)
```

--------------------------------

### Get Child Node by Byte

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Finds the child node of a given node that corresponds to a specific byte. Returns None if no such child exists.

```rust
pub fn child_at_byte<'a>(&'a self, n: &'a TrieNode, byte: u8) -> Option<&'a TrieNode> {
        self.node_children(n).find(|&child| child.byte() == byte)
    }
```

--------------------------------

### SimpleVob Methods

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Provides methods for creating, manipulating, and querying the SimpleVob bit vector.

```APIDOC
## SimpleVob

A compact bit vector representing a set of allowed `crate::TokenId`s. Used as the output of constrained-decoding mask computation: bit _i_ is set when token _i_ is permitted by the current grammar state.

### Methods

* `fn new() -> Self`
* `fn from_slice(bits: &[bool]) -> Self`
* `fn alloc(size: usize) -> Self`
* `fn alloc_ones(size: usize) -> Self`
* `fn alloc_with_capacity(size: usize, capacity: usize) -> Self`
* `fn len(&self) -> usize`
* `fn is_empty(&self) -> bool`
* `fn num_set(&self) -> usize`
* `fn to_bin_string(&self) -> String`
* `fn negated(&self) -> Self`
* `fn as_ptr(&self) -> *const u32`
* `fn as_slice(&self) -> &[u32]`
* `fn iter_set_entries(&self, f: impl FnMut(usize))`
* `fn iter_unset_entries(&self, f: impl FnMut(usize))`
* `fn iter_entries(&self, f: impl FnMut(bool, usize))`
* `fn write_to(&self, buf: &mut [u8])`
* `fn allow_token(&mut self, tok: u32)`
* `unsafe fn allow_token_unchecked(&mut self, tok: u32)`
* `fn disallow_token(&mut self, tok: u32)`
* `fn set(&mut self, idx: usize, val: bool)`
* `fn allow_range(&mut self, range: RangeInclusive<u32>)`
* `fn resize(&mut self, size: usize)`
* `fn get(&self, idx: usize) -> bool`
* `fn is_allowed(&self, tok: u32) -> bool`
* `fn set_all(&mut self, val: bool)`
* `fn apply_to(&self, logits: &mut [f32])`
* `fn iter(&self) -> SimpleVobIter<'_>`
* `fn set_from(&mut self, other: &SimpleVob)`
* `fn or(&mut self, other: &SimpleVob)`
* `fn trim_trailing_zeros(&mut self)`
* `fn or_minus(&mut self, other: &SimpleVob, minus: &SimpleVob)`
* `fn and(&mut self, other: &SimpleVob)`
* `fn is_zero(&self) -> bool`
* `fn and_is_zero(&self, other: &SimpleVob) -> bool`
* `fn sub(&mut self, other: &SimpleVob)`
* `fn first_bit_set_here_and_in(&self, other: &SimpleVob) -> Option<usize>`
* `fn first_bit_set(&self) -> Option<usize>`
* `fn to_list(&self) -> Vec<u32>`
```

--------------------------------

### Recognizer::trie_started Implementation

Source: https://docs.rs/toktrie/latest/toktrie/struct.AnythingGoes.html

Called when iteration over the trie begins. Use for initialization tasks before traversal.

```rust
fn trie_started(&mut self, _dbg_lbl: &str)
Called when iteration over the trie is started
```

--------------------------------

### Get SimpleVob as a slice of u32

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Returns a slice of `u32` representing the internal storage of the SimpleVob. This provides low-level access to the bit representation.

```rust
pub fn as_slice(&self) -> &[u32]
```

--------------------------------

### fn get_error(state: S)

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/trait.FunctionalRecognizer.html

Get a diagnostic message for the current state. Returns a descriptive message about the given state (or None in this default implementation).

```APIDOC
## fn get_error(state: S)

### Description
Get a diagnostic message for the current state
Returns a descriptive message about the given state (or `None` in this default implementation). Users will most likely call this in response to a rejection (i.e., when `try_append` returns `None`), and if used via `StackRecognizer`, the state passed to `get_error` will be the state which rejected the byte. As such, the most useful messages will describe what bytes the state would accept.

### Signature
```rust
fn get_error(&self, _state: S) -> Option<String>
```
```

--------------------------------

### impl Clone for AnythingGoes

Source: https://docs.rs/toktrie/latest/toktrie/recognizer/struct.AnythingGoes.html

Provides cloning capabilities for the AnythingGoes struct, allowing for the creation of duplicate recognizer instances.

```APIDOC
## impl Clone for AnythingGoes

### fn clone(&self) -> AnythingGoes
Returns a duplicate of the value.

### fn clone_from(&mut self, source: &Self)
Performs copy-assignment from `source`.
```

--------------------------------

### Provided Method: tokenize_bytes_marker

Source: https://docs.rs/toktrie/latest/toktrie/trait.TokenizerEnv.html

Tokenizes a byte sequence, interpreting text starting with SPECIAL_TOKEN_MARKER as special tokens. It returns the tokens and the count of tokens that should not be re-tokenized.

```rust
fn tokenize_bytes_marker(&self, s: &[u8]) -> (Vec<TokenId>, usize) { ... }
```

--------------------------------

### TrieBuilder Initialization

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Initializes a new TrieBuilder with a pre-allocated capacity and sets up the root node. The root node is always at index 0.

```rust
fn new(root_byte: u8, vocab_size: u32) -> TrieBuilder {
    let estimated_nodes = (vocab_size as usize).saturating_mul(3).max(1024);
    let mut builder = TrieBuilder {
        nodes: Vec::with_capacity(estimated_nodes),
        root_children: [NO_NODE; 256],
    };
    // The root node is always at index 0.
    builder.nodes.push(BuilderNode {
        token_id: NO_TOKEN,
        byte: root_byte,
        first_child: NO_NODE,
        next_sibling: NO_NODE,
        last_child: NO_NODE,
    });
    builder
}
```

--------------------------------

### Get Sorted Tokens from TokTrie

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Retrieves all tokens from the TokTrie along with their IDs, sorted by token ID. It traverses the trie and collects token-byte pairs.

```rust
    pub fn sorted_tokens(&self) -> Vec<(u32, Vec<u8>)> {
        let mut res = vec![];
        let n = self.root();
        let off = self.node_offset(n);
        let mut p = off + 1;
        let endp = off + n.subtree_size();
        let mut next_pop = 0;
        let mut bytes = vec![];
        while p < endp {
            bytes.drain(bytes.len() - next_pop..);
            let n = &self.nodes[p];
            let b = n.byte();
            bytes.push(b);
            if let Some(t) = n.token_id() {
                res.push((t, bytes.clone()));
            }
            next_pop = if n.subtree_size() == 1 {
                n.num_parents()
            } else {
                0
            };
            p += 1;
        }
        res
    }
```

--------------------------------

### Try Creating Uninitialized Arc<T> with Allocator API

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Demonstrates attempting to create an Arc<T> with uninitialized contents using the allocator API, returning a Result. Requires `unsafe` for `assume_init` and the `allocator_api` feature.

```rust
#![feature(allocator_api)]

use std::sync::Arc;

let mut five = Arc::<u32>::try_new_uninit()?;

// Deferred initialization:
Arc::get_mut(&mut five).unwrap().write(5);

let five = unsafe { five.assume_init() };

assert_eq!(*five, 5);

```

--------------------------------

### Other Trait Implementations

Source: https://docs.rs/toktrie/latest/toktrie/bytes/struct.U32Pair.html

Lists other standard trait implementations for U32Pair, including Copy, Eq, Pod, StructuralPartialEq, Freeze, RefUnwindSafe, Send, Sync, Unpin, UnsafeUnpin, and UnwindSafe.

```APIDOC
### impl Copy for U32Pair
### impl Eq for U32Pair
### impl Pod for U32Pair
### impl StructuralPartialEq for U32Pair
## Auto Trait Implementations
### impl Freeze for U32Pair
### impl RefUnwindSafe for U32Pair
### impl Send for U32Pair
### impl Sync for U32Pair
### impl Unpin for U32Pair
### impl UnsafeUnpin for U32Pair
### impl UnwindSafe for U32Pair
```

--------------------------------

### Get Token String Representation

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Retrieves the string representation of a token ID. This is a convenience method that calls `token` and converts the resulting bytes to a String.

```rust
pub fn token_str(&self, idx: u32) -> String {
        String::from_utf8_lossy(self.token(idx)).to_string()
    }
```

--------------------------------

### Creating an Uninitialized Arc with a Custom Allocator

Source: https://docs.rs/toktrie/latest/toktrie/type.TokEnv.html

Demonstrates creating an `Arc` with uninitialized contents using a custom allocator. This requires `allocator_api` and `get_mut_unchecked` features for deferred initialization.

```rust
#![feature(get_mut_unchecked)]
#![feature(allocator_api)]

use std::sync::Arc;
use std::alloc::System;

let mut five = Arc::<u32, _>::new_uninit_in(System);

let five = unsafe {
    // Deferred initialization:
    Arc::get_mut_unchecked(&mut five).as_mut_ptr().write(5);

    five.assume_init()
};

assert_eq!(*five, 5)
```

--------------------------------

### Get a raw pointer to the SimpleVob's data

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Provides a raw pointer to the underlying `u32` array representing the bit vector. Use with caution.

```rust
pub fn as_ptr(&self) -> *const u32
```

--------------------------------

### Compare SimpleVobs for inequality

Source: https://docs.rs/toktrie/latest/toktrie/struct.SimpleVob.html

Implements the `ne` method for `PartialEq`, allowing comparison for inequality using the `!=` operator.

```rust
fn ne(&self, other: &Rhs) -> bool
```

--------------------------------

### Get a Splice, creating a default if none found

Source: https://docs.rs/toktrie/latest/src/toktrie/lib.rs.html

Retrieves a matching Splice or creates a new one with the sampled token if no suitable Splice exists.

```rust
pub fn spliced(&self, sampled: TokenId) -> Splice {
        self.find_splice(sampled)
            .cloned()
            .unwrap_or_else(|| Splice {
                when_sampled: vec![],
                backtrack: 0,
                ff_tokens: vec![sampled],
            })
    }
```

--------------------------------

### TokRxInfo Trait Implementations

Source: https://docs.rs/toktrie/latest/toktrie/struct.TokRxInfo.html

Details the trait implementations for TokRxInfo, including Clone, Debug, PartialEq, Copy, Eq, and StructuralPartialEq.

```APIDOC
### impl Clone for TokRxInfo

#### fn clone(&self) -> TokRxInfo
Returns a duplicate of the value.

#### fn clone_from(&mut self, source: &Self)
Performs copy-assignment from `source`.

### impl Debug for TokRxInfo

#### fn fmt(&self, f: &mut Formatter<'_>) -> Result
Formats the value using the given formatter.

### impl PartialEq for TokRxInfo

#### fn eq(&self, other: &TokRxInfo) -> bool
Tests for `self` and `other` values to be equal.

#### fn ne(&self, other: &Rhs) -> bool
Tests for `!=`.
```

--------------------------------

### Greedy tokenization of byte slice

Source: https://docs.rs/toktrie/latest/src/toktrie/toktree.rs.html

Tokenizes a byte slice using a greedy approach. It repeatedly finds the longest matching token starting from the current position.

```rust
pub fn greedy_tokenize(&self, bytes: &[u8]) -> Vec<TokenId> {
        let mut tokens = Vec::new();
        let mut i = 0;
        while i < bytes.len() {
            let mut node = self.root();
            let mut last_tok = None;
            let mut last_idx = i;
            #[allow(clippy::needless_range_loop)]
            for j in i..bytes.len() {
                if let Some(child) = self.child_at_byte(node, bytes[j]) {
                    node = child;
                    if let Some(tok) = node.token_id() {
                        last_tok = Some(tok);
                        last_idx = j;
                    }
                } else {
                    break;
                }
            }
            if let Some(t) = last_tok {
                tokens.push(t);
            } else {
                // whoops, there is a byte missing from the tokenizer
                // just carry on...
                // https://github.com/guidance-ai/llguidance/issues/138
            }
            i = last_idx + 1;
        }
        tokens
    }
```
Tokens:
Raw