Candelabra

https://docs.rs/candelabra/0.2.0/candelabra/
Admin
Candelabra is a desktop-friendly Rust wrapper around Candle for quantized GGUF models, providing...
Tokens:31,825
Snippets:287
Trust Score:9.1
Update:4 months ago
Show doc for...
Tokens:
Raw
### Getting File Metadata with and_then

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Shows how to use `and_then` to chain operations that might fail, such as getting file metadata and then its modification time. This example highlights error handling for file system operations.

```rust
use std::{io::ErrorKind, path::Path};

// Note: on Windows "/" maps to "C:\"
let root_modified_time = Path::new("/").metadata().and_then(|md| md.modified());
assert!(root_modified_time.is_ok());

let should_fail = Path::new("/bad/path").metadata().and_then(|md| md.modified());
assert!(should_fail.is_err());
assert_eq!(should_fail.unwrap_err().kind(), ErrorKind::NotFound);
```

--------------------------------

### Product of Results Example

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Multiplies numbers from a vector of strings, returning an Err if any string cannot be parsed. This demonstrates the `product` method on Result.

```rust
let nums = vec!["5", "10", "1", "2"];
let total: Result<usize, _> = nums.iter().map(|w| w.parse::<usize>()).product();
assert_eq!(total, Ok(100));
```

```rust
let nums = vec!["5", "10", "one", "2"];
let total: Result<usize, _> = nums.iter().map(|w| w.parse::<usize>()).product();
assert!(total.is_err());
```

--------------------------------

### Basic Model and Tokenizer Loading with Inference

Source: https://docs.rs/candelabra

This example demonstrates how to download a model and tokenizer, load them, and run a basic inference. It uses the `download_model`, `load_tokenizer_from_repo`, `Model::load`, and `run_inference` functions.

```rust
use candelabra::{download_model, load_tokenizer_from_repo, Model, InferenceConfig, run_inference};
use std::sync::{Arc, atomic::AtomicBool};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let model_path = download_model(
        "bartowski/SmolLM2-360M-Instruct-GGUF",
        "SmolLM2-360M-Instruct-Q4_K_M.gguf",
    )?;
    let tokenizer = load_tokenizer_from_repo("HuggingFaceTB/SmolLM2-360M-Instruct")?;
    let mut model = Model::load(&model_path)?;
    let cancel_token = Arc::new(AtomicBool::new(false));
    let config = InferenceConfig::default();

    let _result = run_inference(
        &mut model,
        &tokenizer,
        &config,
        cancel_token,
        |_| Ok(()),
    )?;

    Ok(())
}
```

--------------------------------

### Summing Elements in a Vector with Error Handling

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

This example demonstrates how to sum elements in a vector using `iter().map().sum()`. It shows how to handle potential errors, such as encountering a negative element, by returning an `Err`.

```Rust
let f = |&x: &i32| if x < 0 { Err("Negative element found") } else { Ok(x) };
let v = vec![1, 2];
let res: Result<i32, _> = v.iter().map(f).sum();
assert_eq!(res, Ok(3));
let v = vec![1, -2];
let res: Result<i32, _> = v.iter().map(f).sum();
assert_eq!(res, Err("Negative element found"));
```

--------------------------------

### Get Best Available Compute Device

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.get_device.html

Retrieves the best available compute device. Returns an error if a specifically preferred device type is requested but not found.

```rust
pub fn get_device(
    preferred: Option<DeviceType>,
) -> Result<(Device, DeviceType), CandelabraError>
```

--------------------------------

### Get Compute Device Type

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Returns the type of the compute device selected for this model.

```rust
pub fn device_type(&self) -> DeviceType
```

--------------------------------

### Providing a Default Value with unwrap_or

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Shows how to use `unwrap_or` to get the `Ok` value or a provided default if the `Result` is `Err`. The default value is eagerly evaluated.

```rust
let default = 2;
let x: Result<u32, &str> = Ok(9);
assert_eq!(x.unwrap_or(default), 9);

let x: Result<u32, &str> = Err("error");
assert_eq!(x.unwrap_or(default), default);
```

--------------------------------

### Get Model Architecture

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Retrieves the GGUF architecture name from the model's metadata.

```rust
pub fn architecture(&self) -> &str
```

--------------------------------

### Get Compute Device Name

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Retrieves a human-readable name of the compute device currently in use by the model.

```rust
pub fn device_name(&self) -> String
```

--------------------------------

### Safely get Ok value (never panics)

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the contained `Ok` value. This experimental API is guaranteed not to panic, serving as a compile-time safeguard against future changes that might introduce panics.

```rust
fn only_good_news() -> Result<String, !> {
    Ok("this is fine".into())
}

let s: String = only_good_news().into_ok();
println!("{s}");
```

--------------------------------

### Collect Iterator of Results (Success Case)

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Collects an iterator of `Result` items into a single `Result`. If all items are `Ok`, it returns a container with the unwrapped values. This example shows a successful addition with overflow checking.

```rust
let v = vec![1, 2];
let res: Result<Vec<u32>, &'static str> = v.iter().map(|x: &u32|
    x.checked_add(1).ok_or("Overflow!")
).collect();
assert_eq!(res, Ok(vec![2, 3]));
```

--------------------------------

### Get references to Result values

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Use `as_ref()` to obtain references to the contained values of a Result without consuming it. This is useful for inspecting values or passing them to functions that require references.

```rust
let x: Result<u32, &str> = Ok(2);
assert_eq!(x.as_ref(), Ok(&2));

let x: Result<u32, &str> = Err("Error");
assert_eq!(x.as_ref(), Err(&"Error"));
```

--------------------------------

### load_tokenizer_from_repo

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.load_tokenizer_from_repo.html

Downloads and loads a tokenizer from the Hugging Face cache. It takes a repository ID as input and returns a Tokenizer or a CandelabraError.

```APIDOC
## load_tokenizer_from_repo

### Description
Downloads and loads a tokenizer from the Hugging Face cache.

### Signature
```rust
pub fn load_tokenizer_from_repo(repo_id: &str) -> Result<Tokenizer, CandelabraError>
```

### Parameters
#### Path Parameters
- **repo_id** (string) - Required - The repository ID of the tokenizer on Hugging Face.

### Returns
- **Result<Tokenizer, CandelabraError>** - Returns a `Tokenizer` on success or a `CandelabraError` on failure.
```

--------------------------------

### Default InferenceConfig

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.InferenceConfig.html

Provides a default configuration for quick benchmarking, utilizing a small quantized model.

```rust
fn default() -> Self
```

--------------------------------

### Model::reset_state

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Rebuilds the model weights to clear the backend-owned KV cache, ensuring the next inference starts from an empty sequence state.

```APIDOC
## Model::reset_state

### Description
Rebuilds the model weights to clear backend-owned KV cache. Some quantized Candle backends keep cache inside private weight fields without exposing a clear method. Reloading preserves the selected device while guaranteeing the next inference starts from an empty sequence state.

### Method

```
pub fn reset_state(&mut self) -> Result<(), CandelabraError>
```

### Returns

- **Result<(), CandelabraError>** - Ok if the state was reset successfully, or an error if it failed.
```

--------------------------------

### load_tokenizer

Source: https://docs.rs/candelabra

Loads a tokenizer from disk.

```APIDOC
## Function: load_tokenizer

### Description
Loads a tokenizer from a local file path.

### Parameters
* **path** (string) - Required - The local file path to the tokenizer.

### Returns
* **Result<Tokenizer, CandelabraError>** - A Result containing the loaded tokenizer or a CandelabraError.
```

--------------------------------

### download_tokenizer_with_progress

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_tokenizer_with_progress.html

Downloads a tokenizer file with progress reporting via callback.

```APIDOC
## download_tokenizer_with_progress

### Description
Downloads a tokenizer file with progress reporting via callback.

### Function Signature
```rust
pub async fn download_tokenizer_with_progress<F>(
    repo_id: &str,
    on_progress: F,
) -> Result<PathBuf, CandelabraError>
where
    F: FnMut(DownloadProgress) + Send,
```

### Parameters
#### Path Parameters
- **repo_id** (str) - Required - The repository ID from which to download the tokenizer.
- **on_progress** (F) - Required - A mutable closure that accepts `DownloadProgress` and is `Send`. This callback is invoked to report download progress.

### Returns
- `Result<PathBuf, CandelabraError>` - Returns the path to the downloaded tokenizer file on success, or a `CandelabraError` on failure.
```

--------------------------------

### Get mutable references to Result values

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Use `as_mut()` to obtain mutable references to the contained values of a Result. This allows in-place modification of the Ok or Err values.

```rust
fn mutate(r: &mut Result<i32, i32>) {
    match r.as_mut() {
        Ok(v) => *v = 42,
        Err(e) => *e = 0,
    }
}

let mut x: Result<i32, i32> = Ok(2);
mutate(&mut x);
assert_eq!(x.unwrap(), 42);

let mut x: Result<i32, i32> = Err(13);
mutate(&mut x);
assert_eq!(x.unwrap_err(), 0);
```

--------------------------------

### get_device

Source: https://docs.rs/candelabra

Returns the best available compute device with detailed error information.

```APIDOC
## Function: get_device

### Description
Determines and returns the best available compute device for inference, providing detailed error information if no suitable device is found.

### Returns
* **Result<DeviceType, CandelabraError>** - A Result containing the best available compute device or a CandelabraError if no suitable device can be determined.
```

--------------------------------

### load_tokenizer

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.load_tokenizer.html

Loads a tokenizer from disk. It takes a path as input and returns a Result containing either a Tokenizer or a CandelabraError.

```APIDOC
## load_tokenizer

### Description
Loads a tokenizer from disk.

### Function Signature
```rust
pub fn load_tokenizer<P: AsRef<Path>>(
    path: P,
) -> Result<Tokenizer, CandelabraError>
```

### Parameters
* `path` (P: AsRef<Path>): The path to the tokenizer file on disk. This can be a string slice, a `PathBuf`, or any type that implements `AsRef<Path>`.
```

--------------------------------

### Rust Function Signature for download_tokenizer_with_channel

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_tokenizer_with_channel.html

This snippet shows the function signature for downloading a tokenizer with a progress channel. It takes a repository ID and a sender for download progress, returning a result with the path to the tokenizer or an error.

```rust
pub async fn download_tokenizer_with_channel(
    repo_id: &str,
    progress_tx: Sender<DownloadProgress>,
) -> Result<PathBuf, CandelabraError>
```

--------------------------------

### Collect Iterator of Results (Underflow Error Case)

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Collects an iterator of `Result` items. If any item is `Err`, the iteration stops, and that error is returned. This example demonstrates underflow checking.

```rust
let v = vec![1, 2, 0];
let res: Result<Vec<u32>, &'static str> = v.iter().map(|x: &u32|
    x.checked_sub(1).ok_or("Underflow!")
).collect();
assert_eq!(res, Err("Underflow!"));
```

--------------------------------

### download_tokenizer

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_tokenizer.html

Downloads a tokenizer file via `hf-hub` and returns the local cached path.

```APIDOC
## download_tokenizer

### Description
Downloads a tokenizer file via `hf-hub` and returns the local cached path.

### Function Signature
```
pub fn download_tokenizer(repo_id: &str) -> Result<PathBuf, CandelabraError>
```

### Parameters
#### Path Parameters
- **repo_id** (str) - Required - The repository ID of the tokenizer to download from Hugging Face Hub.
```

--------------------------------

### Serialize Implementation for DownloadProgress

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.DownloadProgress.html

Enables serializing DownloadProgress into various data formats using Serde. Use this to transmit or store progress data.

```rust
fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>
where __S: Serializer,
```

--------------------------------

### Safely get Err value (never panics)

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the contained `Err` value. This experimental API is guaranteed not to panic, serving as a compile-time safeguard against future changes that might introduce panics.

```rust
fn only_bad_news() -> Result<!, String> {
    Err("Oops, it failed".into())
}

let error: String = only_bad_news().into_err();
println!("{error}");
```

--------------------------------

### Rust Function Signature for download_tokenizer_with_progress

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_tokenizer_with_progress.html

This snippet shows the function signature for `download_tokenizer_with_progress`. It takes a repository ID and a mutable closure for progress reporting, returning a `PathBuf` or a `CandelabraError`.

```rust
pub async fn download_tokenizer_with_progress<
    F
>(
    repo_id: &str,
    on_progress: F,
) -> Result<PathBuf, CandelabraError>
where
    F: FnMut(DownloadProgress) + Send,

```

--------------------------------

### Clone Implementation for DownloadProgress

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.DownloadProgress.html

Allows creating a copy of a DownloadProgress instance. Useful for retaining progress snapshots.

```rust
fn clone(&self) -> DownloadProgress
```

```rust
fn clone_from(&mut self, source: &Self)
```

--------------------------------

### download_tokenizer_with_channel

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_tokenizer_with_channel.html

Downloads a tokenizer file from a repository, sending progress updates through a provided Tokio channel. It returns the path to the downloaded file or an error if the download fails.

```APIDOC
## Function download_tokenizer_with_channel

### Description
Downloads a tokenizer file with progress reporting via Tokio channel.

### Signature
```rust
pub async fn download_tokenizer_with_channel(
    repo_id: &str,
    progress_tx: Sender<DownloadProgress>,
) -> Result<PathBuf, CandelabraError>
```

### Parameters
* `repo_id` (*str*) - The repository ID from which to download the tokenizer.
* `progress_tx` (*Sender<DownloadProgress>*) - A Tokio channel sender to report download progress.

### Returns
* `Result<PathBuf, CandelabraError>` - Returns `Ok(PathBuf)` with the path to the downloaded tokenizer file on success, or `Err(CandelabraError)` if an error occurs during the download.
```

--------------------------------

### Debug Implementation for DownloadProgress

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.DownloadProgress.html

Enables formatted output for debugging DownloadProgress instances. Use this for logging or inspecting progress data.

```rust
fn fmt(&self, f: &mut Formatter<'_>) -> Result
```

--------------------------------

### Rebuild Model Weights to Clear KV Cache

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Rebuilds the model weights to clear the backend-owned KV cache, ensuring the next inference starts from an empty sequence state while preserving the selected device.

```rust
pub fn reset_state(&mut self) -> Result<(), CandelabraError>
```

--------------------------------

### into_ok

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the contained `Ok` value without panicking. This is an experimental API that guarantees never to panic, serving as a compile-time safeguard against potential panics.

```APIDOC
## pub const fn into_ok(self) -> T

### Description
Returns the contained `Ok` value, but never panics. This is a nightly-only experimental API.

### Method
`self.into_ok()`

### Parameters
None

### Request Example
```rust
fn only_good_news() -> Result<String, !> {
    Ok("this is fine".into())
}

let s: String = only_good_news().into_ok();
println!("{s}");
```

### Response
#### Success Response
- **T**: The contained `Ok` value.
```

--------------------------------

### Model::load_with_device

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Loads a GGUF model specifying the exact compute device and device type to be used.

```APIDOC
## Model::load_with_device

### Description
Loads a GGUF model with a specific device. Use this when you want to control which device is used, such as for testing or when the user has selected a specific device.

### Method

```
pub fn load_with_device<P: AsRef<Path>>(path: P, device: Device, device_type: DeviceType) -> Result<Self, CandelabraError>
```

### Parameters

#### Path Parameters

- **path** (P: AsRef<Path>) - Required - Path to the GGUF model file
- **device** (Device) - Required - The compute device to use
- **device_type** (DeviceType) - Required - The type of device (for reporting purposes)

### Returns

- **Result<Self, CandelabraError>** - A loaded `Model` ready for inference.
```

--------------------------------

### download_model_with_channel

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_model_with_channel.html

Downloads a model file with progress reporting via Tokio channel. It takes the repository ID, filename, and a sender for download progress as input, and returns the path to the downloaded file or an error.

```APIDOC
## Function download_model_with_channel

### Description
Downloads a model file with progress reporting via Tokio channel.

### Signature
```rust
pub async fn download_model_with_channel(
    repo_id: &str,
    filename: &str,
    progress_tx: Sender<DownloadProgress>,
) -> Result<PathBuf, CandelabraError>
```

### Parameters
* `repo_id` (&str): The repository ID from which to download the model.
* `filename` (&str): The name of the file to download.
* `progress_tx` (Sender<DownloadProgress>): A Tokio channel sender to report download progress.

### Returns
* `Result<PathBuf, CandelabraError>`: Returns the `PathBuf` of the downloaded file on success, or a `CandelabraError` on failure.
```

--------------------------------

### Rust load_tokenizer_from_repo Function Signature

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.load_tokenizer_from_repo.html

This snippet shows the function signature for `load_tokenizer_from_repo`. It takes a repository ID as a string slice and returns a Result containing either a Tokenizer or a CandelabraError.

```rust
pub fn load_tokenizer_from_repo(
    repo_id: &str,
) -> Result<Tokenizer, CandelabraError>

```

--------------------------------

### Load GGUF Model with Specific Device

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Loads a GGUF model specifying the compute device and device type. Use this for explicit device control.

```rust
pub fn load_with_device<P: AsRef<Path>>( path: P, device: Device, device_type: DeviceType, ) -> Result<Self, CandelabraError>
```

--------------------------------

### get_best_device

Source: https://docs.rs/candelabra

Returns the best available compute device with automatic fallback.

```APIDOC
## Function: get_best_device

### Description
Determines and returns the best available compute device for inference, with automatic fallback to CPU if necessary.

### Returns
* **DeviceType** - The best available compute device (e.g., CUDA, Metal, CPU).
```

--------------------------------

### Download Tokenizer Function Signature

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_tokenizer.html

This snippet shows the function signature for `download_tokenizer`. It takes a repository ID as a string slice and returns a Result containing either a PathBuf to the cached file or a CandelabraError.

```rust
pub fn download_tokenizer(repo_id: &str) -> Result<PathBuf, CandelabraError>
```

--------------------------------

### get_best_device

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.get_best_device.html

Returns the best available compute device with automatic fallback. The priority order is Metal (if enabled and successful), then CUDA (if enabled and successful), and finally CPU as a fallback.

```APIDOC
## Function get_best_device

### Description
Returns the best available compute device with automatic fallback. The priority order is Metal (if enabled and successful), then CUDA (if enabled and successful), and finally CPU as a fallback.

### Signature
```rust
pub fn get_best_device() -> (Device, DeviceType)
```

### Returns
A tuple containing the best available `Device` and its `DeviceType`.
```

--------------------------------

### Load GGUF Model Automatically

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Loads a GGUF model from a given path, automatically selecting the best available device (Metal > CUDA > CPU).

```rust
pub fn load<P: AsRef<Path>>(path: P) -> Result<Self, CandelabraError>
```

--------------------------------

### Rust Candelabra download_model Function Signature

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_model.html

This is the function signature for `download_model`. It takes a repository ID and a filename as string slices and returns a Result containing a PathBuf on success or a CandelabraError on failure.

```rust
pub fn download_model(
    repo_id: &str,
    filename: &str,
) -> Result<PathBuf, CandelabraError>
```

--------------------------------

### download_model_with_progress

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_model_with_progress.html

Downloads a model file from a repository, providing progress updates through a callback function. This is useful for large files where users need to track download status.

```APIDOC
## Function download_model_with_progress

### Description
Downloads a model file with progress reporting via callback.

### Signature
```rust
pub async fn download_model_with_progress<F>(
    repo_id: &str,
    filename: &str,
    on_progress: F,
) -> Result<PathBuf, CandelabraError>
where
    F: FnMut(DownloadProgress) + Send,
```

### Parameters
* `repo_id` (str): The repository ID from which to download the model.
* `filename` (str): The name of the file to download.
* `on_progress` (FnMut(DownloadProgress) + Send): A mutable closure that will be called with `DownloadProgress` updates during the download.

### Returns
* `Result<PathBuf, CandelabraError>`: Returns the `PathBuf` to the downloaded file on success, or a `CandelabraError` on failure.
```

--------------------------------

### Deserialize Implementation for DownloadProgress

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.DownloadProgress.html

Enables deserializing DownloadProgress from various data formats using Serde. Use this to reconstruct progress data from external sources.

```rust
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>
where __D: Deserializer<'de>,
```

--------------------------------

### download_model

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_model.html

Downloads a model file via `hf-hub` and returns the local cached path.

```APIDOC
## download_model

### Description
Downloads a model file via `hf-hub` and returns the local cached path.

### Function Signature
```rust
pub fn download_model(
    repo_id: &str,
    filename: &str,
) -> Result<PathBuf, CandelabraError>
```

### Parameters
#### Path Parameters
- **repo_id** (string) - Required - The repository ID from which to download the model.
- **filename** (string) - Required - The name of the file to download from the repository.

### Returns
- **Result<PathBuf, CandelabraError>** - Returns the `PathBuf` of the downloaded file on success, or a `CandelabraError` on failure.
```

--------------------------------

### Experimental Error::provide method

Source: https://docs.rs/candelabra/0.2.0/candelabra/enum.CandelabraError.html

This is a nightly-only experimental API for providing type-based access to error context, intended for error reports.

```rust
fn provide<'a>(&'a self, request: &mut Request<'a>)
```

--------------------------------

### Rust Function Signature: download_model_with_progress

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_model_with_progress.html

Signature for the `download_model_with_progress` function. It takes a repository ID, filename, and a mutable closure for progress reporting. The callback receives `DownloadProgress` and must be `Send`.

```rust
pub async fn download_model_with_progress<
    F
>(
    repo_id: &str,
    filename: &str,
    on_progress: F,
) -> Result<PathBuf, CandelabraError>
where
    F: FnMut(DownloadProgress) + Send,

```

--------------------------------

### Model::load

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Loads a GGUF model from a specified path, automatically selecting the best available compute device (Metal, CUDA, or CPU).

```APIDOC
## Model::load

### Description
Loads a GGUF model from the given path, automatically selecting the best available device (Metal > CUDA > CPU) and loading the model weights.

### Method

```
pub fn load<P: AsRef<Path>>(path: P) -> Result<Self, CandelabraError>
```

### Parameters

#### Path Parameters

- **path** (P: AsRef<Path>) - Required - Path to the GGUF model file

### Returns

- **Result<Self, CandelabraError>** - A loaded `Model` ready for inference.
```

--------------------------------

### DownloadProgress Struct Definition

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.DownloadProgress.html

Defines the structure for tracking download progress. Use this struct to hold and access information about ongoing downloads.

```rust
pub struct DownloadProgress {
    pub downloaded_bytes: u64,
    pub total_bytes: u64,
    pub percentage: f32,
    pub speed_bytes_per_sec: f64,
    pub filename: String,
}
```

--------------------------------

### impl<L, R> From<Either<L, R>> for Result<R, L>

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Converts an `Either<L, R>` into a `Result<R, L>`, mapping `Right` to `Ok` and `Left` to `Err`.

```APIDOC
## fn from(val: Either<L, R>) -> Result<R, L>

### Description
Converts to this type from the input type.

### Method
from

### Parameters
- **val** (*Either<L, R>*) - The `Either` value to convert.

### Response
#### Success Response
- **Result<R, L>** - The converted `Result` value.
```

--------------------------------

### Chain Results with `and`

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the second `Result` if the first is `Ok`, otherwise returns the first `Err`. Arguments are eagerly evaluated.

```rust
let x: Result<u32, &str> = Ok(2);
let y: Result<&str, &str> = Err("late error");
assert_eq!(x.and(y), Err("late error"));

let x: Result<u32, &str> = Err("early error");
let y: Result<&str, &str> = Ok("foo");
assert_eq!(x.and(y), Err("early error"));

let x: Result<u32, &str> = Err("not a 2");
let y: Result<&str, &str> = Err("late error");
assert_eq!(x.and(y), Err("not a 2"));

let x: Result<u32, &str> = Ok(2);
let y: Result<&str, &str> = Ok("different result type");
assert_eq!(x.and(y), Ok("different result type"));
```

--------------------------------

### download_model_with_channel Function Signature

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.download_model_with_channel.html

This is the function signature for downloading a model file with progress reporting via a Tokio channel. It requires the repository ID, filename, and a sender for download progress updates.

```rust
pub async fn download_model_with_channel(
    repo_id: &str,
    filename: &str,
    progress_tx: Sender<DownloadProgress>,
) -> Result<PathBuf, CandelabraError>
```

--------------------------------

### Using or to Provide a Default on Err

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Illustrates the `or` method, which returns the `Ok` value if the first Result is `Ok`, otherwise returns the second Result. This is useful for providing alternative error values or fallback `Ok` values.

```rust
let x: Result<u32, &str> = Ok(2);
let y: Result<u32, &str> = Err("late error");
assert_eq!(x.or(y), Ok(2));

let x: Result<u32, &str> = Err("early error");
let y: Result<u32, &str> = Ok(2);
assert_eq!(x.or(y), Ok(2));

let x: Result<u32, &str> = Err("not a 2");
let y: Result<u32, &str> = Err("late error");
assert_eq!(x.or(y), Err("late error"));

let x: Result<u32, &str> = Ok(2);
let y: Result<u32, &str> = Ok(100);
assert_eq!(x.or(y), Ok(2));
```

--------------------------------

### run_inference_with_channel

Source: https://docs.rs/candelabra

Runs inference and streams tokens over a Tokio channel.

```APIDOC
## Function: run_inference_with_channel

### Description
Executes an inference run and streams the generated tokens over a Tokio channel, allowing for real-time processing of output.

### Parameters
* **model** (*mut* Model) - Required - A mutable reference to the loaded model.
* **tokenizer** (Tokenizer) - Required - The tokenizer to use for processing input and output.
* **config** (InferenceConfig) - Required - The configuration for the inference run.
* **cancel_token** (Arc<AtomicBool>) - Required - An atomic boolean flag to signal cancellation of the inference.

### Returns
* **Result<(tokio::sync::mpsc::Receiver<String>, InferenceResult), CandelabraError>** - A Result containing a receiver for streamed tokens and the final inference result, or a CandelabraError.
```

--------------------------------

### Result::is_ok_and

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Checks if the Result is Ok and its value satisfies a given predicate.

```APIDOC
## `impl<T, E> Result<T, E>`

### `pub fn is_ok_and<F>(self, f: F) -> bool` where F: FnOnce(T) -> bool

**Description**: Returns `true` if the result is `Ok` and the value inside of it matches a predicate. This method consumes the `Result`.

**Parameters**:
- `self`: The `Result` instance to check.
- `f`: A closure that takes the success value and returns `true` if it matches the predicate.

**Returns**:
- `true` if the `Result` is `Ok` and the predicate returns `true`, `false` otherwise.

**Examples**:
```rust
let x: Result<u32, &str> = Ok(2);
assert_eq!(x.is_ok_and(|x| x > 1), true);

let x: Result<u32, &str> = Ok(0);
assert_eq!(x.is_ok_and(|x| x > 1), false);

let x: Result<u32, &str> = Err("hey");
assert_eq!(x.is_ok_and(|x| x > 1), false);

let x: Result<String, &str> = Ok("ownership".to_string());
assert_eq!(x.as_ref().is_ok_and(|x| x.len() > 1), true);
println!("still alive {:?}", x);
```
```

--------------------------------

### get_device

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.get_device.html

Returns the best available compute device with detailed error information. Unlike `get_best_device`, this function returns an error if a preferred device type is explicitly requested but unavailable.

```APIDOC
## get_device

### Description
Returns the best available compute device with detailed error information. Unlike `get_best_device`, this function returns an error if a preferred device type is explicitly requested but unavailable.

### Signature
```rust
pub fn get_device(
    preferred: Option<DeviceType>,
) -> Result<(Device, DeviceType), CandelabraError>
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Response
#### Success Response
Returns a tuple containing the `Device` and `DeviceType` if successful.

#### Error Response
Returns a `CandelabraError` if the preferred device type is unavailable or another error occurs.
```

--------------------------------

### Convert Result to Option with Ok value

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Use `ok()` to convert a Result into an Option, preserving the success value if it's Ok, and discarding the error by converting it to None.

```rust
let x: Result<u32, &str> = Ok(2);
assert_eq!(x.ok(), Some(2));

let x: Result<u32, &str> = Err("Nothing here");
assert_eq!(x.ok(), None);
```

--------------------------------

### and

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the second `Result` if the first is `Ok`, otherwise returns the `Err` value of the first `Result`. Arguments are eagerly evaluated.

```APIDOC
## pub fn and<U>(self, res: Result<U, E>) -> Result<U, E>

### Description
Returns `res` if the result is `Ok`, otherwise returns the `Err` value of `self`. Arguments passed to `and` are eagerly evaluated.

### Method
`self.and(other_result)`

### Parameters
- **res** (Result<U, E>) - Required - The `Result` to return if `self` is `Ok`.

### Request Example
```rust
let x: Result<u32, &str> = Ok(2);
let y: Result<&str, &str> = Err("late error");
assert_eq!(x.and(y), Err("late error"));

let x: Result<u32, &str> = Ok(2);
let y: Result<&str, &str> = Ok("different result type");
assert_eq!(x.and(y), Ok("different result type"));
```

### Response
#### Success Response
- **Result<U, E>** - Either the `res` value or the `Err` value from `self`.
```

--------------------------------

### run_inference_with_channel

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.run_inference_with_channel.html

Executes inference using a provided model, tokenizer, and configuration, streaming results through a channel and respecting a cancellation token.

```APIDOC
## Function run_inference_with_channel

### Description
Runs inference and streams tokens over a Tokio channel.

### Signature
```rust
pub fn run_inference_with_channel(
    model: &mut Model,
    tokenizer: &Tokenizer,
    config: &InferenceConfig,
    cancel_token: Arc<AtomicBool>,
    token_tx: Sender<String>,
) -> Result<InferenceResult, CandelabraError>
```

### Parameters
* **model** (*mut Model*) - A mutable reference to the inference model.
* **tokenizer** (*Tokenizer*) - The tokenizer to use for processing input and output.
* **config** (*InferenceConfig*) - Configuration settings for the inference process.
* **cancel_token** (*Arc<AtomicBool>*) - An atomic boolean flag to signal cancellation.
* **token_tx** (*Sender<String>*) - A Tokio channel sender to stream generated tokens.

### Returns
* **Result<InferenceResult, CandelabraError>** - Ok containing the `InferenceResult` on success, or `Err` with a `CandelabraError` on failure.
```

--------------------------------

### Product for Result<U, E> for Result<T, E>

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Implements the `Product` trait for `Result<T, E>` when iterating over `Result<U, E>`, allowing the computation of a product from an iterator of `Result` values.

```APIDOC
## impl<T, U, E> Product<Result<U, E>> for Result<T, E>
where T: Product<U>,

#### fn product<I>(iter: I) -> Result<T, E>
where I: Iterator<Item = Result<U, E>>,
Takes each element in the `Iterator`: if it is an `Err`, no further elements are taken, and the `Err` is returned. Should no `Err` occur, the product of all elements is returned.
##### §Examples
This multiplies each number in a vector of strings, if a string could not be parsed the operation returns `Err`:
```
let nums = vec!["5", "10", "1", "2"];
let total: Result<usize, _> = nums.iter().map(|w| w.parse::<usize>()).product();
assert_eq!(total, Ok(100));
let nums = vec!["5", "10", "one", "2"];
let total: Result<usize, _> = nums.iter().map(|w| w.parse::<usize>()).product();
assert!(total.is_err());
```
```

--------------------------------

### Chaining Fallible Operations with and_then

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Demonstrates chaining operations that return Result using `and_then`. This is useful for sequential fallible operations where each step depends on the success of the previous one.

```rust
assert_eq!(Ok(2).and_then(sq_then_to_string), Ok(4.to_string()));
assert_eq!(Ok(1_000_000).and_then(sq_then_to_string), Err("overflowed"));
assert_eq!(Err("not a number").and_then(sq_then_to_string), Err("not a number"));
```

--------------------------------

### Rust get_best_device Function Signature

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.get_best_device.html

This is the function signature for get_best_device. It returns a tuple containing the selected Device and its DeviceType. Ensure the 'metal' or 'cuda' features are enabled if you intend to use those specific hardware accelerators.

```rust
pub fn get_best_device() -> (Device, DeviceType)
```

--------------------------------

### run_inference_with_channel Function Signature

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.run_inference_with_channel.html

This is the function signature for `run_inference_with_channel`. It takes a mutable Model, a Tokenizer, an InferenceConfig, an AtomicBool cancel token, and a Tokio Sender for streaming String tokens. It returns a Result containing either an InferenceResult or a CandelabraError.

```rust
pub fn run_inference_with_channel(
    model: &mut Model,
    tokenizer: &Tokenizer,
    config: &InferenceConfig,
    cancel_token: Arc<AtomicBool>,
    token_tx: Sender<String>,
) -> Result<InferenceResult, CandelabraError>
```

--------------------------------

### map_or_default

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Maps a Result to a U by applying a function to the Ok value or returning the default value for U if Err.

```APIDOC
## pub const fn map_or_default<U, F>(self, f: F) -> U

### Description
Maps a `Result<T, E>` to a `U` by applying function `f` to the contained value if the result is `Ok`. If the result is `Err`, it returns the default value for the type `U`.

### Method
`map_or_default`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```rust
#![feature(result_option_map_or_default)]

let x: Result<_, &str> = Ok("foo");
let y: Result<&str, _> = Err("bar");

assert_eq!(x.map_or_default(|x| x.len()), 3);
assert_eq!(y.map_or_default(|y| y.len()), 0);
```

### Response
#### Success Response (U)
- The result of applying the function `f` to the `Ok` value, or the default value of `U` if the Result was `Err`.

#### Response Example
```json
3
```
```

--------------------------------

### unwrap_or

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the contained `Ok` value or a provided default. Arguments passed to `unwrap_or` are eagerly evaluated.

```APIDOC
## unwrap_or

### Description
Returns the contained `Ok` value or a provided default. Arguments passed to `unwrap_or` are eagerly evaluated; if you are passing the result of a function call, it is recommended to use `unwrap_or_else`, which is lazily evaluated.

### Method
`unwrap_or`

### Parameters
- `default` (T) - The default value to return if `self` is `Err`.

### Response
- `T` - The `Ok` value if `self` is `Ok`, otherwise the `default` value.

### Request Example
```rust
let default = 2;
let x: Result<u32, &str> = Ok(9);
assert_eq!(x.unwrap_or(default), 9);

let x: Result<u32, &str> = Err("error");
assert_eq!(x.unwrap_or(default), default);
```
```

--------------------------------

### load_tokenizer Function Signature

Source: https://docs.rs/candelabra/0.2.0/candelabra/fn.load_tokenizer.html

This is the function signature for load_tokenizer. It takes a path as input and returns a Result containing a Tokenizer or a CandelabraError.

```rust
pub fn load_tokenizer<P: AsRef<Path>>(
    path: P,
) -> Result<Tokenizer, CandelabraError>
```

--------------------------------

### Unwrap Ok Value with Panic on Err

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the contained Ok value, consuming the Result. Panics if the value is Err, with a message including the provided string and the error content. Use with caution; prefer pattern matching or other unwrap methods.

```rust
let x: Result<u32, &str> = Err("emergency failure");
x.expect("Testing expect"); // panics with `Testing expect: emergency failure`
```

```rust
let path = std::env::var("IMPORTANT_PATH")
    .expect("env variable `IMPORTANT_PATH` should be set by `wrapper_script.sh`");
```

--------------------------------

### Deprecated Error::description method

Source: https://docs.rs/candelabra/0.2.0/candelabra/enum.CandelabraError.html

This method is deprecated and users should prefer using the Display implementation or to_string() for error messages.

```rust
fn description(&self) -> &str
```

--------------------------------

### Rust Result is_ok() Method

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Checks if the Result is the Ok variant. Returns true if it is Ok, false otherwise.

```rust
let x: Result<i32, &str> = Ok(-3);
assert_eq!(x.is_ok(), true);

let x: Result<i32, &str> = Err("Some error message");
assert_eq!(x.is_ok(), false);
```

--------------------------------

### unwrap

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Retrieves the contained `Ok` value, consuming the `Result`. This method will panic if the `Result` is an `Err`. It is generally recommended to use alternative methods like the `?` operator or pattern matching for safer error handling.

```APIDOC
## pub fn unwrap(self) -> T

### Description
Returns the contained `Ok` value, consuming the `self` value. Panics if the value is an `Err`.

### Method
`self.unwrap()`

### Parameters
None

### Request Example
```rust
let x: Result<u32, &str> = Ok(2);
assert_eq!(x.unwrap(), 2);
```

### Response
#### Success Response
- **T**: The contained `Ok` value.

### Error Handling
Panics if the value is an `Err`, with a panic message provided by the `Err`’s value.
```

--------------------------------

### InferenceConfig Struct Definition

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.InferenceConfig.html

Defines the structure for inference configuration, including model ID, filename, prompt, token limits, temperature, and optional duration.

```rust
pub struct InferenceConfig {
    pub model_id: String,
    pub filename: String,
    pub prompt: String,
    pub max_tokens: usize,
    pub temperature: f64,
    pub max_duration_secs: Option<u64>,
    pub stop_on_eos: bool,
}
```

--------------------------------

### Implement Debug for CandelabraError

Source: https://docs.rs/candelabra/0.2.0/candelabra/enum.CandelabraError.html

Provides a way to format the CandelabraError enum for debugging purposes.

```rust
impl Debug for CandelabraError {
    fn fmt(&self, f: &mut Formatter<'_>) -> Result
}
```

--------------------------------

### Clone From Implementation for InferenceConfig

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.InferenceConfig.html

Enables performing copy-assignment from a source InferenceConfig to a mutable one.

```rust
fn clone_from(&mut self, source: &Self)
```

--------------------------------

### inspect

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Calls a function with a reference to the contained value if `Ok`. Returns the original result.

```APIDOC
## pub fn inspect<F>(self, f: F) -> Result<T, E>

### Description
Calls a function with a reference to the contained value if `Ok`.
Returns the original result.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response
None

#### Response Example
None
```

--------------------------------

### Unwrap with panic on Err

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Demonstrates that `unwrap()` panics when called on an `Err` variant.

```rust
let x: Result<u32, &str> = Err("emergency failure");
x.unwrap(); // panics with `emergency failure`
```

--------------------------------

### impl<T, E> Clone for Result<T, E>

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Provides the ability to create a duplicate of a Result value. This requires that both the Ok and Err types implement Clone.

```APIDOC
## fn clone(&self) -> Result<T, E>

### Description
Returns a duplicate of the value.

### Method
clone

### Parameters
None

### Response
#### Success Response
- **Result<T, E>** - A new Result instance that is a duplicate of the original.
```

```APIDOC
## fn clone_from(&mut self, source: &Result<T, E>)

### Description
Performs copy-assignment from `source`.

### Method
clone_from

### Parameters
- **source** (*&Result<T, E>*) - The Result to copy from.

### Response
None
```

--------------------------------

### Implement Display for CandelabraError

Source: https://docs.rs/candelabra/0.2.0/candelabra/enum.CandelabraError.html

Provides a way to format the CandelabraError enum for user-facing messages.

```rust
impl Display for CandelabraError {
    fn fmt(&self, __formatter: &mut Formatter<'_>) -> Result
}
```

--------------------------------

### Unwrap Ok value

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns the contained `Ok` value. Panics if the value is an `Err`. Use with caution.

```rust
let x: Result<u32, &str> = Ok(2);
assert_eq!(x.unwrap(), 2);
```

--------------------------------

### map_or

Source: https://docs.rs/candelabra/0.2.0/candelabra/type.Result.html

Returns a default value if the Result is Err, or applies a function to the contained Ok value.

```APIDOC
## pub fn map_or<U, F>(self, default: U, f: F) -> U

### Description
Returns the provided `default` value if the Result is `Err`, or applies the function `f` to the contained value if the Result is `Ok`.

### Method
`map_or`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```rust
let x: Result<_, &str> = Ok("foo");
assert_eq!(x.map_or(42, |v| v.len()), 3);

let x: Result<&str, _> = Err("bar");
assert_eq!(x.map_or(42, |v| v.len()), 42);
```

### Response
#### Success Response (U)
- The result of applying the function `f` to the `Ok` value, or the `default` value if the Result was `Err`.

#### Response Example
```json
3
```
```

--------------------------------

### Clone Implementation for InferenceConfig

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.InferenceConfig.html

Allows creating a duplicate of an existing InferenceConfig value.

```rust
fn clone(&self) -> InferenceConfig
```

--------------------------------

### Model::device_type

Source: https://docs.rs/candelabra/0.2.0/candelabra/struct.Model.html

Retrieves the type of compute device selected for this model.

```APIDOC
## Model::device_type

### Description
Returns the device type selected for this model.

### Method

```
pub fn device_type(&self) -> DeviceType
```

### Returns

- **DeviceType** - The type of device selected for the model.
```
Tokens:
Raw