Embellama

https://docs.rs/embellama/0.10.1/embellama/
Admin
Embellama is a high-performance Rust library for generating text embeddings using llama-cpp with...
Tokens:97,349
Snippets:934
Trust Score:9.1
Update:6 months ago
Show doc for...
Tokens:
Raw
### Run Example Cargo Project

Source: https://docs.rs/embellama

Command to run a specific example from the `examples/` directory. Replace `simple` with the desired example name.

```bash
cargo run --example simple
```

--------------------------------

### Build and Use Embedding Engine

Source: https://docs.rs/embellama

Quick start guide to build model and engine configurations, create an embedding engine, and generate single or batch embeddings.

```rust
use embellama::{ModelConfig, EngineConfig, EmbeddingEngine, NormalizationMode};

// Build model configuration
let model_config = ModelConfig::builder()
    .with_model_path("/path/to/model.gguf")
    .with_model_name("my-model")
    .with_normalization_mode(NormalizationMode::L2)
    .build()?;

// Build engine configuration
let engine_config = EngineConfig::builder()
    .with_model_config(model_config)
    .build()?;

// Create engine
let engine = EmbeddingEngine::new(engine_config)?;

// Generate single embedding
let text = "Hello, world!";
let embedding = engine.embed(None, text)?;

// Generate batch embeddings
let texts = vec!["Text 1", "Text 2", "Text 3"];
let embeddings = engine.embed_batch(None, &texts)?;
```

--------------------------------

### Changelog Generation Commands

Source: https://docs.rs/embellama

Commands to generate and preview changelogs using git-cliff. Install git-cliff with `cargo install git-cliff`.

```bash
just changelog              # Regenerate CHANGELOG.md
just changelog-unreleased   # Preview unreleased changes
```

--------------------------------

### Initialize embellama library

Source: https://docs.rs/embellama/0.10.1/embellama/fn.init.html

Call this function once at the start of your application to configure the global tracing subscriber.

```rust
embellama::init();
```

--------------------------------

### Examples of Result Usage

Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html

Demonstrates summing integers in a vector with error handling for negative elements.

```APIDOC
## Examples

### Description
This sums up every integer in a vector, rejecting the sum if a negative element is encountered.

### Method
N/A (code example)

### Endpoint
N/A

### Parameters
None

### Request Example
```rust
let f = |&x: &i32| if x < 0 { Err("Negative element found") } else { Ok(x) };
let v = vec![1, 2];
let res: Result<i32, _> = v.iter().map(f).sum();
assert_eq!(res, Ok(3));
let v = vec![1, -2];
let res: Result<i32, _> = v.iter().map(f).sum();
assert_eq!(res, Err("Negative element found"));
```

### Response
N/A

#### Response Example
N/A
```

--------------------------------

### GET /v1/models

Source: https://docs.rs/embellama/0.10.1/embellama/server/handlers/fn.list_models_handler.html

Handler for listing all available models in an OpenAI-compatible format.

```APIDOC
## GET /v1/models

### Description
Lists all available models in OpenAI-compatible format.

### Method
GET

### Endpoint
/v1/models
```

--------------------------------

### Support

Source: https://docs.rs/embellama/0.10.1/embellama/index.html

Information on how to get support for Embellama.

```APIDOC
### Support

For issues and questions, please use the GitHub issue tracker.
```

--------------------------------

### GET /metrics

Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/index.html

Exports all collected server metrics in Prometheus format.

```APIDOC
## GET /metrics

### Description
Exports all collected metrics in Prometheus format for observability and monitoring.

### Method
GET

### Endpoint
/metrics

### Response
#### Success Response (200)
- **Content-Type** (text/plain) - Prometheus metrics exposition format
```

--------------------------------

### GET /v1/models

Source: https://docs.rs/embellama/0.10.1/embellama/server/handlers/index.html

Handler for listing available models via the OpenAI-compatible API.

```APIDOC
## GET /v1/models

### Description
Handler for listing available models.

### Method
GET

### Endpoint
/v1/models
```

--------------------------------

### Get Version Info

Source: https://docs.rs/embellama/0.10.1/embellama/fn.version_info.html

Retrieves the version information for the embellama crate.

```APIDOC
## GET /version_info

### Description
Get version information for the embellama crate.

### Method
GET

### Endpoint
/version_info

### Response
#### Success Response (200)
- **version** (string) - The version string of the embellama crate.

#### Response Example
```json
{
  "version": "0.10.1"
}
```
```

--------------------------------

### Default BatchProcessorBuilder Implementation

Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessorBuilder.html

Provides a default `BatchProcessorBuilder` instance, which can be used as a starting point for configuration.

```rust
fn default() -> BatchProcessorBuilder
```

--------------------------------

### Get Model Path

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Retrieve the file path to the model file on disk.

```rust
let path = model.path();
```

--------------------------------

### Create New CacheMetrics Instance

Source: https://docs.rs/embellama/0.10.1/embellama/cache/metrics/struct.CacheMetrics.html

Initializes a new CacheMetrics instance with all counters set to zero. This is the default way to start tracking metrics.

```rust
pub fn new() -> Self
```

--------------------------------

### run_server Function

Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.run_server.html

The run_server function is a convenience function that creates the application state, builds the router, and starts the server with graceful shutdown handling.

```APIDOC
## run_server

### Description
Run the server with the provided configuration. This is a convenience function that creates the application state, builds the router, and starts the server with graceful shutdown handling.

### Method
ASYNC FUNCTION

### Endpoint
N/A (This is a function call, not a direct HTTP endpoint)

### Parameters
#### Arguments
- **config** (ServerConfig) - Required - Server configuration

### Returns
Result indicating success or failure

### Errors
Returns an error if:
  * Application state creation fails
  * Server binding fails
  * Server startup fails

### Example
```rust
let config = ServerConfig::builder()
    .model_path("/path/to/model.gguf")
    .model_name("my-model")
    .host("0.0.0.0")
    .port(8080)
    .build()?;

run_server(config).await?;
```
```

--------------------------------

### Get Model Configuration

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Retrieve a reference to the `ModelConfig` used to initialize the embedding model.

```rust
let config = model.config();
```

--------------------------------

### Get Recommended GPU Layers

Source: https://docs.rs/embellama/0.10.1/embellama/enum.BackendType.html

Retrieves the recommended number of GPU layers for the backend. Returns None for CPU backends and effectively all layers for GPU backends.

```rust
pub const fn recommended_gpu_layers(&self) -> Option<u32>
```

--------------------------------

### Get or Initialize Singleton EmbeddingEngine

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html

This function retrieves the singleton EmbeddingEngine instance. If it's not initialized, it creates a new one using the provided configuration. The configuration is only applied on the first call.

```rust
pub fn get_or_init(config: EngineConfig) -> Result<Arc<Mutex<Self>>>
```

--------------------------------

### Build and Run Server Configuration

Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.run_server.html

Demonstrates how to build a ServerConfig and then use it to run the server. Ensure the model path and name are correctly specified.

```rust
let config = ServerConfig::builder()
    .model_path("/path/to/model.gguf")
    .model_name("my-model")
    .host("0.0.0.0")
    .port(8080)
    .build()?;

run_server(config).await?;
```

--------------------------------

### Initialize and extend a router with create_router

Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.create_router.html

Demonstrates initializing the application state and extending the default router with custom routes.

```rust
let config = ServerConfig::builder()
    .model_path("/path/to/model.gguf")
    .build()?;

let state = AppState::new(config)?;
let router = create_router(state);

// Add custom routes
let router = router.route("/custom", axum::routing::get(|| async { "Custom route" }));

// Start server...
```

--------------------------------

### Initialize metrics

Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/fn.init_metrics.html

Call this function during server startup to initialize all metrics.

```rust
pub fn init_metrics()
```

--------------------------------

### Get Compiled Backend - embellama

Source: https://docs.rs/embellama/0.10.1/embellama/fn.get_compiled_backend.html

Retrieves the currently compiled backend based on build features. This function is part of the embellama crate.

```rust
pub fn get_compiled_backend() -> BackendType
```

--------------------------------

### Initialize Metrics

Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/fn.init_metrics.html

Initializes all metrics for the embellama server. This function should be called at server startup.

```APIDOC
## init_metrics

### Description
Initializes all metrics (called at server startup).

### Method
N/A (This is a function call, not an HTTP endpoint)

### Endpoint
N/A

### Parameters
None

### Request Example
N/A

### Response
N/A
```

--------------------------------

### GET /v1/embeddings/prefix/stats

Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_stats_handler.html

Retrieves statistics related to the prefix cache.

```APIDOC
## GET /v1/embeddings/prefix/stats

### Description
Handler for retrieving prefix cache statistics.

### Method
GET

### Endpoint
/v1/embeddings/prefix/stats

### Parameters
#### Path Parameters
- **__arg0** (State<AppState>) - Required - The application state containing cache information.
```

--------------------------------

### GET /v1/embeddings/prefix

Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_list_handler.html

Handler for retrieving a list of all cached prefixes.

```APIDOC
## GET /v1/embeddings/prefix

### Description
Lists all cached prefixes.

### Method
GET

### Endpoint
/v1/embeddings/prefix

### Parameters

### Request Body

### Request Example

### Response
#### Success Response (200)
- **prefixes** (array) - A list of cached prefix strings.

#### Response Example
{
  "prefixes": ["prefix1", "prefix2"]
}
```

--------------------------------

### init function

Source: https://docs.rs/embellama/0.10.1/embellama/fn.init.html

Initializes the library with a default tracing subscriber to ensure proper logging infrastructure.

```APIDOC
## init

### Description
Initializes the library with a default tracing subscriber. This function sets up a global tracing subscriber to ensure logging infrastructure outlives all threads, preventing thread-local storage panics during cleanup.

### Usage
Call this function once at the start of your application.

### Request Example
```rust
embellama::init();
```
```

--------------------------------

### Create FileModelProvider Instance

Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html

Constructs a new FileModelProvider. It takes a model path and a model name as input.

```rust
pub fn new( model_path: impl Into<PathBuf>, model_name: impl Into<String>, ) -> Self
```

--------------------------------

### list_models_handler function signature

Source: https://docs.rs/embellama/0.10.1/embellama/server/handlers/fn.list_models_handler.html

The function signature for the GET /v1/models handler.

```rust
pub async fn list_models_handler(__arg0: State<AppState>) -> Response
```

--------------------------------

### Basic Model and Engine Configuration

Source: https://docs.rs/embellama

Sets up basic model and engine configurations using the builder pattern. Requires specifying the model path and name.

```rust
use embellama::{ModelConfig, EngineConfig};

let model_config = ModelConfig::builder()
    .with_model_path("/path/to/model.gguf")
    .with_model_name("my-model")
    .build()?;

let config = EngineConfig::builder()
    .with_model_config(model_config)
    .build()?;
```

--------------------------------

### Pointer and Initialization Utilities

Source: https://docs.rs/embellama/0.10.1/embellama/enum.TruncateTokens.html

Provides utilities for managing raw pointers, including alignment, initialization, dereferencing, and dropping.

```APIDOC
## Pointer and Initialization Utilities

### Description
Utilities for managing raw pointers, including alignment, initialization, dereferencing, and dropping.

### Constants
- **ALIGN**: `usize` - The alignment of the pointer.

### Functions
- **init**: `unsafe fn init(init: <T as Pointable>::Init) -> usize` - Initializes an object with the given initializer.
- **deref**: `unsafe fn deref<'a>(ptr: usize) -> &'a T` - Dereferences the given pointer to provide an immutable reference.
- **deref_mut**: `unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T` - Mutably dereferences the given pointer to provide a mutable reference.
- **drop**: `unsafe fn drop(ptr: usize)` - Drops the object pointed to by the given pointer.
```

--------------------------------

### AppState Initialization

Source: https://docs.rs/embellama/0.10.1/embellama/server/state/struct.AppState.html

Methods for creating and interacting with the application state.

```APIDOC
## AppState::new

### Description
Creates a new application state instance.

### Parameters
#### Arguments
- **config** (ServerConfig) - Required - Server configuration

### Returns
- **Result<Self>** - A new AppState instance or an error if engine configuration or initialization fails.

## AppState::model_name

### Description
Retrieves the name of the model currently in use.

### Returns
- **&str** - The model name.

## AppState::is_ready

### Description
Checks if the server is ready for requests.

### Returns
- **bool** - True if the server is ready, false otherwise.
```

--------------------------------

### Library Initialization

Source: https://docs.rs/embellama

Functions for initializing the library and retrieving version information.

```APIDOC
## init

### Description
Initialize the library with default tracing subscriber.

## init_with_env_filter

### Description
Initialize the library with a custom environment filter.

## version_info

### Description
Get version information.
```

--------------------------------

### Get Model Name

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Retrieve the name of the embedding model as specified in its configuration.

```rust
let name = model.name();
```

--------------------------------

### Initialize embellama with environment filter

Source: https://docs.rs/embellama/0.10.1/embellama/fn.init_with_env_filter.html

Sets up a global tracing subscriber using the provided filter string to control logging verbosity.

```rust
embellama::init_with_env_filter("embellama=debug,info");
```

--------------------------------

### create_router

Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.create_router.html

Initializes an Axum router with standard routes and middleware using the provided application state.

```APIDOC
## create_router

### Description
Creates an Axum router with all the standard routes and middleware configured. Users can add additional routes or middleware before starting the server.

### Parameters
- **state** (AppState) - Required - Application state containing the embedding engine and configuration

### Returns
- **Router** - Configured Axum router ready to be served

### Request Example
```rust
let config = ServerConfig::builder()
    .model_path("/path/to/model.gguf")
    .build()?;

let state = AppState::new(config)?;
let router = create_router(state);

// Add custom routes
let router = router.route("/custom", axum::routing::get(|| async { "Custom route" }));
```
```

--------------------------------

### GET /cache/stats

Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.cache_stats_handler.html

Retrieves current cache statistics and system memory information.

```APIDOC
## GET /cache/stats

### Description
Handler for retrieving cache statistics and system memory information.

### Method
GET

### Endpoint
/cache/stats

### Parameters
#### Path Parameters
- **__arg0** (State<AppState>) - Required - The application state containing cache and system information.
```

--------------------------------

### Get Type ID

Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixDetector.html

Retrieves the `TypeId` of the implementing type. This is part of the `Any` trait.

```rust
fn type_id(&self) -> TypeId
```

--------------------------------

### Embed Embellama Server in Rust Applications

Source: https://docs.rs/embellama/0.10.1/embellama/server/index.html

Demonstrates how to configure the engine and server, then either run the server directly or integrate it into a custom router.

```rust
use embellama::server::{
    AppState, ServerConfig, create_router, run_server,
    EngineConfig, ModelConfig
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Build engine configuration first
    let engine_config = EngineConfig::builder()
        .with_model_path("/path/to/model.gguf")
        .with_model_name("my-model")
        .build()?;

    // Option 1: Use the convenient run_server function
    let config = ServerConfig::builder()
        .engine_config(engine_config.clone())
        .port(8080)
        .build()?;

    run_server(config.clone()).await?;

    // Option 2: Create router for custom integration
    let state = AppState::new(config)?;
    let router = create_router(state);
    // Add your own routes or middleware...

    Ok(())
}
```

--------------------------------

### Get Cache Size

Source: https://docs.rs/embellama/0.10.1/embellama/cache/trait.CacheStore.html

Returns the current number of entries stored in the cache.

```rust
fn len(&self) -> usize
```

--------------------------------

### Build BatchProcessor Instance

Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessorBuilder.html

Constructs and returns a new `BatchProcessor` instance with the configurations set on the builder.

```rust
pub fn build(self) -> BatchProcessor
```

--------------------------------

### Model Management API

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html

APIs for retrieving, setting, and getting information about embedding models.

```APIDOC
## GET /model/default

### Description
Gets the default model name.

### Method
GET

### Endpoint
/model/default

### Response
#### Success Response (200)
- **model_name** (string) - The default model name if set.

#### Response Example
{
  "model_name": "model_v1"
}
```

```APIDOC
## POST /model/default

### Description
Sets the default model.

### Method
POST

### Endpoint
/model/default

### Request Body
- **model_name** (string) - Required - The name of the model to set as default.

### Request Example
{
  "model_name": "model_v1"
}

### Response
#### Success Response (200)
- **status** (string) - Indicates success or failure.

#### Response Example
{
  "status": "success"
}
```

```APIDOC
## GET /model/{model_name}

### Description
Gets information about a loaded model.

### Method
GET

### Endpoint
/model/{model_name}

### Parameters
#### Path Parameters
- **model_name** (string) - Required - The name of the model.

### Response
#### Success Response (200)
- **ModelInfo** (object) - Information about the model.
  - **name** (string) - The name of the model.
  - **version** (string) - The version of the model.
  - **dimensions** (integer) - The embedding dimensions.

#### Response Example
{
  "name": "model_v1",
  "version": "1.0.0",
  "dimensions": 768
}
```

--------------------------------

### FileModelProvider Constructor

Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html

Creates a new instance of the FileModelProvider.

```APIDOC
## Constructor: FileModelProvider::new

### Description
Creates a new file-based model provider instance.

### Parameters
- **model_path** (impl Into<PathBuf>) - Required - The path to the GGUF model file.
- **model_name** (impl Into<String>) - Required - The name assigned to the model.
```

--------------------------------

### GET /metrics/worker_utilization

Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/static.WORKER_UTILIZATION.html

Retrieves the current gauge value representing the worker thread utilization.

```APIDOC
## GET /metrics/worker_utilization

### Description
Retrieves the current gauge value for worker thread utilization, represented as a float between 0.0 and 1.0.

### Method
GET

### Endpoint
/metrics/worker_utilization

### Response
#### Success Response (200)
- **value** (float) - The current utilization of worker threads (0.0 to 1.0).

#### Response Example
{
  "worker_utilization": 0.45
}
```

--------------------------------

### Function init_with_env_filter

Source: https://docs.rs/embellama/0.10.1/embellama/fn.init_with_env_filter.html

Initializes the library with a custom environment filter. This function sets up a global tracing subscriber with a custom filter string. The global subscriber ensures logging is available during all cleanup operations, preventing thread-local storage issues.

```APIDOC
## Function init_with_env_filter

### Description
Initializes the library with a custom environment filter. This function sets up a global tracing subscriber with a custom filter string. The global subscriber ensures logging is available during all cleanup operations, preventing thread-local storage issues.

### Arguments
- **filter** (string) - Required - Environment filter string (e.g., “info”, “debug”, “embellama=debug”)

### Example
```rust
embellama::init_with_env_filter("embellama=debug,info");
```
```

--------------------------------

### Get Pointer Alignment

Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixCache.html

Defines the alignment of the pointer for the `Pointable` trait. This is a constant value.

```rust
const ALIGN: usize
```

--------------------------------

### EmbeddingEngine Initialization and Singleton Access

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html

Methods for creating and accessing the singleton EmbeddingEngine instance.

```APIDOC
## POST /embedding_engine/get_or_init

### Description
Gets or initializes the singleton embedding engine with the given configuration. If the engine is already initialized, returns the existing instance. The configuration is only used for the first initialization.

### Method
POST

### Endpoint
/embedding_engine/get_or_init

### Parameters
#### Request Body
- **config** (EngineConfig) - Required - The engine configuration (used only on first call)

### Request Example
```json
{
  "config": {
    "model_path": "path/to/model.gguf",
    "model_name": "my-model"
  }
}
```

### Response
#### Success Response (200)
- **engine** (Arc<Mutex<EmbeddingEngine>>) - The singleton embedding engine instance.

#### Response Example
```json
{
  "engine": "..."
}
```

## GET /embedding_engine/instance

### Description
Gets the existing engine instance if it has been initialized.

### Method
GET

### Endpoint
/embedding_engine/instance

### Response
#### Success Response (200)
- **engine** (Option<Arc<Mutex<EmbeddingEngine>>>) - Some(engine) if initialized, None otherwise.

#### Response Example
```json
{
  "engine": "..."
}
```

## POST /embedding_engine/new

### Description
Creates a new embedding engine with the given configuration. Note: This now uses the singleton pattern internally. Use `get_or_init()` for explicit singleton access.

### Method
POST

### Endpoint
/embedding_engine/new

### Parameters
#### Request Body
- **config** (EngineConfig) - Required - The engine configuration

### Request Example
```json
{
  "config": {
    "model_path": "path/to/model.gguf",
    "model_name": "my-model"
  }
}
```

### Response
#### Success Response (200)
- **engine** (EmbeddingEngine) - The newly created embedding engine.

#### Response Example
```json
{
  "engine": "..."
}
```
```

--------------------------------

### Model Loading Behavior: Immediate vs. Lazy

Source: https://docs.rs/embellama

Illustrates the difference between immediate loading for the initial model and lazy loading for additional models. Lazy-loaded models are loaded on their first use.

```rust
// First model - loaded immediately
let mut engine = EmbeddingEngine::new(config)?;
assert!(engine.is_model_loaded_in_thread("model1"));

// Additional model - lazy loaded
engine.load_model(config2)?;
assert!(engine.is_model_registered("model2"));
assert!(!engine.is_model_loaded_in_thread("model2"));  // Not yet loaded

// Triggers actual loading in thread
engine.embed(Some("model2"), "text")?;
assert!(engine.is_model_loaded_in_thread("model2"));  // Now loaded
```

--------------------------------

### GET metadata_cache_size

Source: https://docs.rs/embellama/0.10.1/embellama/fn.metadata_cache_size.html

Retrieves the current size of the GGUF metadata cache for monitoring and debugging purposes.

```APIDOC
## GET metadata_cache_size

### Description
Get the current size of the GGUF metadata cache. This is primarily useful for monitoring and debugging.

### Method
GET

### Endpoint
metadata_cache_size

### Response
#### Success Response (200)
- **size** (usize) - The current size of the GGUF metadata cache.
```

--------------------------------

### Implement CloneToUninit (Nightly)

Source: https://docs.rs/embellama/0.10.1/embellama/cache/memory_monitor/struct.MemoryMonitorConfig.html

Nightly-only experimental API for performing copy-assignment from self to an uninitialized destination.

```rust
unsafe fn clone_to_uninit(&self, dest: *mut u8)
```

--------------------------------

### Backend Auto-Detection and Configuration

Source: https://docs.rs/embellama

Configures the engine to automatically detect the best available backend and specifies the model path. Also shows how to check the selected backend and its features.

```rust
use embellama::{EngineConfig, detect_best_backend, BackendInfo};

// Automatic backend detection
let config = EngineConfig::with_backend_detection()
    .with_model_path("/path/to/model.gguf")
    .with_model_name("my-model")
    .build()?;

// Check which backend was selected
let backend_info = BackendInfo::new();
println!("Using backend: {}", backend_info.backend);
println!("Available features: {:?}", backend_info.available_features);
```

--------------------------------

### Define prefix_stats_handler Function

Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_stats_handler.html

Handler for GET /v1/embeddings/prefix/stats. Returns prefix cache statistics. Requires AppState.

```rust
pub async fn prefix_stats_handler(__arg0: State<AppState>) -> impl IntoResponse
```

--------------------------------

### ClientRateLimiter Methods

Source: https://docs.rs/embellama/0.10.1/embellama/server/rate_limiter/struct.ClientRateLimiter.html

Methods for initializing and managing per-client rate limiters.

```APIDOC
## new(config: RateLimitConfig)

### Description
Create a new per-client rate limiter instance.

### Parameters
- **config** (RateLimitConfig) - Required - Configuration settings for the rate limiter.

---

## get_limiter(client_id: &str)

### Description
Get or create a rate limiter for a specific client.

### Parameters
- **client_id** (&str) - Required - The unique identifier (IP address) for the client.

### Errors
- Panics if rate limiting is not enabled in the configuration.

---

## cleanup(max_age: Duration)

### Description
Clean up old rate limiter entries. This should be called periodically to maintain performance.

### Parameters
- **max_age** (Duration) - Required - The duration after which entries are considered old and eligible for removal.
```

--------------------------------

### Function: detect_best_backend

Source: https://docs.rs/embellama/0.10.1/embellama/fn.detect_best_backend.html

Documentation for the detect_best_backend function which identifies the optimal backend.

```APIDOC
## detect_best_backend

### Description
Detect the best available backend based on compile-time features and runtime capabilities.

### Method
Function Call

### Endpoint
embellama::detect_best_backend()

### Response
- **BackendType** (enum) - The detected backend type based on system configuration.
```

--------------------------------

### prefix_list_handler function signature

Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_list_handler.html

The function signature for the prefix_list_handler, which handles GET requests to list cached prefixes.

```rust
pub async fn prefix_list_handler(__arg0: State<AppState>) -> impl IntoResponse
```

--------------------------------

### Initialize and Use EmbeddingEngine

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html

Use this snippet to create an EmbeddingEngine instance with a specified model configuration and generate embeddings for text. Ensure the model path and name are correctly set.

```rust
use embellama::{EmbeddingEngine, EngineConfig};

let config = EngineConfig::builder()
    .with_model_path("path/to/model.gguf")
    .with_model_name("my-model")
    .build()?;

let engine = EmbeddingEngine::new(config)?;
let embedding = engine.embed("my-model", "Hello, world!")?;
```

--------------------------------

### Retrieve cache statistics handler signature

Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.cache_stats_handler.html

Defines the asynchronous handler function for the GET /cache/stats endpoint.

```rust
pub async fn cache_stats_handler(__arg0: State<AppState>) -> impl IntoResponse
```

--------------------------------

### Conversion and Utility Methods

Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.PrefixListResponse.html

Standard conversion and vector-like zipping operations.

```APIDOC
## fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

### Description
Performs the conversion from type T to U.

## fn vzip(self) -> V

### Description
Performs a multi-lane zip operation on the provided type.
```

--------------------------------

### Get SessionData Age in Seconds

Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.SessionData.html

Calculates and returns the age of the session in seconds based on its creation timestamp.

```rust
pub fn age_seconds(&self) -> u64
```

--------------------------------

### Implement Clone for Usage

Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.Usage.html

Provides methods to duplicate Usage instances. Use `clone` for a simple copy and `clone_from` for in-place assignment.

```rust
fn clone(&self) -> Usage
```

```rust
fn clone_from(&mut self, source: &Self)
```

--------------------------------

### Get Model Path for FileModelProvider

Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html

Retrieves the path to a model file by its name. This is part of the ModelProvider trait implementation.

```rust
fn get_model_path<'life0, 'life1, 'async_trait>( &'life0 self, model_name: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<PathBuf>> + Send + 'async_trait>>
where Self: 'async_trait,
 'life0: 'async_trait,
 'life1: 'async_trait,
```

--------------------------------

### Worker::new

Source: https://docs.rs/embellama/0.10.1/embellama/server/worker/struct.Worker.html

Creates a new worker instance.

```APIDOC
## Worker::new

### Description
Create a new worker

### Method
```rust
pub fn new( id: usize, engine: Arc<Mutex<EmbeddingEngine>>, receiver: Receiver<WorkerRequest>, ) -> Self
```

### Arguments
- **id** (usize) - Worker identifier
- **engine** (Arc<Mutex<EmbeddingEngine>>) - Shared embedding engine instance
- **receiver** (Receiver<WorkerRequest>) - Channel to receive requests

### Returns
A new `Worker` instance
```

--------------------------------

### Get Value from Cache

Source: https://docs.rs/embellama/0.10.1/embellama/cache/trait.CacheStore.html

Retrieves a value associated with a given key from the cache. Returns None if the key is not found.

```rust
fn get(&self, key: &K) -> Option<V>
```

--------------------------------

### Get Maximum Sequences for Batch Processing

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Returns the maximum number of sequences that can be processed in parallel within a single batch.

```rust
let n_seq_max = model.n_seq_max();
```

--------------------------------

### BatchProcessor::builder

Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessor.html

Initiates the creation of a batch processor with custom configuration options.

```APIDOC
## BatchProcessor::builder

### Description
Creates a batch processor with custom configuration.

### Method
`pub fn builder() -> BatchProcessorBuilder`

### Returns
Returns a `BatchProcessorBuilder` instance for further configuration.
```

--------------------------------

### Get Embedding Model Dimensions

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Retrieve the dimensionality of the embedding vectors that the model produces. This indicates the size of the output vectors.

```rust
let dimensions = model.embedding_dimensions();
```

--------------------------------

### Build RateLimiter from Config

Source: https://docs.rs/embellama/0.10.1/embellama/server/rate_limiter/struct.RateLimitConfig.html

Creates a new rate limiter instance based on the provided RateLimitConfig. Panics if requests_per_second or burst_size is 0.

```rust
pub fn build_limiter(&self) -> Option<RateLimiter>
```

--------------------------------

### Dereference Pointer

Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixDetector.html

Dereferences a raw pointer to get an immutable reference to the object. This is an unsafe operation and part of the `Pointable` trait.

```rust
unsafe fn deref<'a>(ptr: usize) -> &'a T
```

--------------------------------

### Chain Fallible Operations with and_then

Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html

Used to chain operations that return a Result. The second example demonstrates usage with filesystem metadata.

```rust
fn sq_then_to_string(x: u32) -> Result<String, &'static str> {
    x.checked_mul(x).map(|sq| sq.to_string()).ok_or("overflowed")
}

assert_eq!(Ok(2).and_then(sq_then_to_string), Ok(4.to_string()));
assert_eq!(Ok(1_000_000).and_then(sq_then_to_string), Err("overflowed"));
assert_eq!(Err("not a number").and_then(sq_then_to_string), Err("not a number"));
```

```rust
use std::{io::ErrorKind, path::Path};

// Note: on Windows "/" maps to "C:\"
let root_modified_time = Path::new("/").metadata().and_then(|md| md.modified());
assert!(root_modified_time.is_ok());

let should_fail = Path::new("/bad/path").metadata().and_then(|md| md.modified());
assert!(should_fail.is_err());
assert_eq!(should_fail.unwrap_err().kind(), ErrorKind::NotFound);
```

--------------------------------

### into_ok() - Unfailing Ok extraction (Nightly)

Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html

Returns the contained `Ok` value, but never panics. This is a nightly-only experimental API.

```APIDOC
## into_ok()

### Description
Returns the contained `Ok` value, but never panics. Unlike `unwrap`, this method is known to never panic on the result types it is implemented for. This is a nightly-only experimental API.

### Method
`into_ok()`

### Parameters
None

### Request Example
```rust
fn only_good_news() -> Result<String, !> {
    Ok("this is fine".into())
}

let s: String = only_good_news().into_ok();
println!("{s}");
```

### Response
#### Success Response (T)
Returns the contained `Ok` value.
```

--------------------------------

### VZip Implementation

Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.ErrorDetail.html

Describes the `VZip` implementation and its `vzip` method.

```APIDOC
## impl<V, T> VZip<V> for T

### Description
Implementation for VZip.

## fn vzip(self) -> V

### Description
Performs a zipped operation.

### Method
N/A (Associated function)

### Endpoint
N/A
```

--------------------------------

### Get Model Metadata

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Access the model's metadata, including its name, file path, vocabulary size, and number of parameters.

```rust
let (name, path, vocab_size, n_params) = model.model_metadata();
```

--------------------------------

### Get Model Size

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Retrieve the approximate memory footprint of the loaded model in bytes. This can be `None` on certain platforms or for very large models.

```rust
let size = model.model_size();
```

--------------------------------

### Create New EmbeddingEngine

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html

Instantiates a new EmbeddingEngine. Note that this method internally uses a singleton pattern, so `get_or_init()` is recommended for explicit singleton access.

```rust
pub fn new(config: EngineConfig) -> Result<Self>
```

--------------------------------

### Get Current Load Level

Source: https://docs.rs/embellama/0.10.1/embellama/server/backpressure/struct.LoadShedder.html

Retrieves the current load level of the load shedder. This value is typically updated by the `update_load` method.

```rust
pub fn load(&self) -> f64
```

--------------------------------

### Detect Best Backend

Source: https://docs.rs/embellama/0.10.1/embellama/fn.detect_best_backend.html

Use this function to automatically select the optimal backend. It considers compile-time features and runtime capabilities.

```rust
pub fn detect_best_backend() -> BackendType
```

--------------------------------

### Get GGUF metadata cache size

Source: https://docs.rs/embellama/0.10.1/embellama/fn.metadata_cache_size.html

Returns the current size of the GGUF metadata cache as a usize. Useful for monitoring and debugging.

```rust
pub fn metadata_cache_size() -> usize
```

--------------------------------

### Retrieve version information

Source: https://docs.rs/embellama/0.10.1/embellama/fn.version_info.html

Returns the current version details for the Embellama crate.

```rust
pub fn version_info() -> VersionInfo
```

--------------------------------

### Dereference Pointer Mutably

Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixDetector.html

Dereferences a raw pointer to get a mutable reference to the object. This is an unsafe operation and part of the `Pointable` trait.

```rust
unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T
```

--------------------------------

### TryFrom and TryInto for Type Conversion

Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.CacheStatsResponse.html

Demonstrates the usage of `TryFrom` and `TryInto` traits for fallible type conversions.

```APIDOC
## TryFrom<U> for T

### Description
Provides a fallible conversion from type `U` to type `T`.

### Methods
#### fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
Performs the conversion.

### Type Alias
#### type Error = Infallible
The type returned in the event of a conversion error.

## TryInto<U> for T

### Description
Provides a fallible conversion from type `T` to type `U`.

### Methods
#### fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>
Performs the conversion.

### Type Alias
#### type Error = <U as TryFrom<T>>::Error
The type returned in the event of a conversion error.
```

--------------------------------

### Get Existing EmbeddingEngine Instance

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html

Retrieves the current singleton EmbeddingEngine instance if it has already been initialized. Returns None if the engine has not been set up yet.

```rust
pub fn instance() -> Option<Arc<Mutex<Self>>>
```

--------------------------------

### Create New PrefixCache

Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixCache.html

Initializes a new PrefixCache instance. Returns an error if the session directory cannot be created. Configuration includes max sessions, TTL, frequency threshold, and an optional session directory path.

```rust
pub fn new( max_sessions: usize, ttl_seconds: u64, frequency_threshold: usize, session_dir: Option<PathBuf>, ) -> Result<Self>
```

--------------------------------

### Get Maximum Sequence Length

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html

Obtain the maximum number of tokens the model can process in a single input sequence. This is a fundamental constraint for inference.

```rust
let max_len = model.max_sequence_length();
```

--------------------------------

### Model Loading Behavior

Source: https://docs.rs/embellama/0.10.1/embellama/index.html

Explains the loading behavior for initial and additional models in Embellama.

```APIDOC
## Model Loading Behavior

### Initial Model Loading

- **Description**: The first model specified during `EmbeddingEngine::new()` is loaded immediately into the current thread.

### Additional Model Loading

- **Description**: Models loaded via `load_model()` are lazily loaded. They are registered but not loaded into memory until their first use (e.g., during an `embed` or `rerank` call).

### Example: Lazy Loading

```rust
use embellama::{EngineConfig, EmbeddingEngine};

// First model - loaded immediately
let config1 = EngineConfig::builder().with_model_path("/path/to/model1.gguf").with_model_name("model1").build()?;
let mut engine = EmbeddingEngine::new(config1)?;
assert!(engine.is_model_loaded_in_thread("model1"));

// Additional model - lazy loaded
let config2 = EngineConfig::builder().with_model_path("/path/to/model2.gguf").with_model_name("model2").build()?;
engine.load_model(config2)?;
assert!(engine.is_model_registered("model2"));
assert!(!engine.is_model_loaded_in_thread("model2")); // Not yet loaded

// Triggers actual loading in thread
let _ = engine.embed(Some("model2"), "some text")?;
assert!(engine.is_model_loaded_in_thread("model2")); // Now loaded
```
```

--------------------------------

### ServerConfig Structure

Source: https://docs.rs/embellama/0.10.1/embellama/server/state/struct.ServerConfig.html

Details the fields and builder pattern for the ServerConfig struct used to initialize the server.

```APIDOC
## ServerConfig

### Description
Represents the configuration settings for the embellama server instance.

### Fields
- **worker_count** (usize) - Number of worker threads.
- **queue_size** (usize) - Maximum pending requests per worker.
- **host** (String) - Server host address.
- **port** (u16) - Server port.
- **request_timeout** (Duration) - Request timeout duration.
- **engine_config** (EngineConfig) - Engine configuration including model settings.

### Methods
- **builder()** -> ServerConfigBuilder: Creates a new builder for server configuration.
```

--------------------------------

### Create Configuration Error

Source: https://docs.rs/embellama/0.10.1/embellama/enum.Error.html

Use this function to create a `ConfigurationError` with a custom message. This is useful for indicating issues with the library's setup or parameters.

```rust
pub fn config(message: impl Into<String>) -> Self
```

--------------------------------

### HttpServerConnExec Implementation

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html

Placeholder for `HttpServerConnExec` implementation.

```APIDOC
### impl<A, B, T> HttpServerConnExec<A, B> for T
where B: Body,

### Source
```

--------------------------------

### Get Cache Statistics

Source: https://docs.rs/embellama/0.10.1/embellama/cache/trait.CacheStore.html

Returns statistics about the cache, such as hit rate, miss rate, or memory usage. The exact content depends on the CacheStats type.

```rust
fn stats(&self) -> CacheStats
```

--------------------------------

### Reset Cache Metrics

Source: https://docs.rs/embellama/0.10.1/embellama/cache/metrics/struct.CacheMetrics.html

Resets all metric counters (hits, misses, evictions, memory_bytes) back to zero. Use this to start fresh metrics collection.

```rust
pub fn reset(&self)
```

--------------------------------

### SystemHealth Methods

Source: https://docs.rs/embellama/0.10.1/embellama/server/backpressure/struct.SystemHealth.html

Methods for initializing and interacting with the SystemHealth monitor.

```APIDOC
## SystemHealth Methods

### Description
Methods to manage and query the health status of the system.

### Methods
- **new(circuit_breaker: Arc<CircuitBreaker>, load_shedder: Arc<LoadShedder>, queue_threshold: usize) -> Self** - Creates a new system health monitor.
- **update_queue_depth(depth: usize)** - Updates the current queue depth.
- **is_healthy() -> bool** - Checks if the system is healthy enough to accept requests.
- **status() -> HealthStatus** - Retrieves the current health status.
```

--------------------------------

### Type Conversion: From/Into

Source: https://docs.rs/embellama/0.10.1/embellama/struct.ModelConfig.html

Demonstrates conversions using the `From` and `Into` traits. `Into<U>` is implemented for `T` when `U: From<T>`.

```APIDOC
## impl<T, U> Into<U> for T
where U: From<T>,

### Description
Provides a way to convert a type `T` into another type `U` if `U` can be created from `T`.

### Method
`into(self) -> U`

### Parameters
- `self` - The instance of type `T` to convert.

### Request Example
```rust
let t_instance: T = ...;
let u_instance: U = t_instance.into();
```

### Response
#### Success Response (200)
- `U` - The converted value of type `U`.

#### Response Example
```rust
// No direct response, conversion happens in place.
```
```

--------------------------------

### Get reference to Result value

Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html

The `as_ref` method converts a `&Result<T, E>` into a `Result<&T, &E>`, providing references to the contained values without consuming the original Result.

```rust
let x: Result<u32, &str> = Ok(2);
assert_eq!(x.as_ref(), Ok(&2));

let x: Result<u32, &str> = Err("Error");
assert_eq!(x.as_ref(), Err(&"Error"));
```

--------------------------------

### List Models for FileModelProvider

Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html

Lists all available models. This is part of the ModelProvider trait implementation.

```rust
fn list_models<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Vec<ModelInfo>>> + Send + 'async_trait>>
where Self: 'async_trait,
 'life0: 'async_trait,
```

--------------------------------

### Get mutable reference to Result value

Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html

The `as_mut` method converts a `&mut Result<T, E>` into a `Result<&mut T, &mut E>`, providing mutable references to the contained values.

```rust
fn mutate(r: &mut Result<i32, i32>) {
    match r.as_mut() {
        Ok(v) => *v = 42,
        Err(e) => *e = 0,
    }
}

let mut x: Result<i32, i32> = Ok(2);
mutate(&mut x);
assert_eq!(x.unwrap(), 42);

let mut x: Result<i32, i32> = Err(13);
mutate(&mut x);
assert_eq!(x.unwrap_err(), 0);
```

--------------------------------

### Create BatchProcessor with Builder

Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessor.html

Initiates the creation of a batch processor using a builder pattern, allowing for custom configuration options before instantiation.

```rust
pub fn builder() -> BatchProcessorBuilder
```

--------------------------------

### EngineConfigBuilder Methods

Source: https://docs.rs/embellama/0.10.1/embellama/struct.EngineConfigBuilder.html

Methods for building and configuring EngineConfig.

```APIDOC
## EngineConfigBuilder

### Description
Builder for creating `EngineConfig` instances.

### Methods

#### `new()`
Create a new builder with default configuration.

#### `with_model_config(model_config: ModelConfig)`
Set the entire model configuration.

#### `with_model_path<P: AsRef<Path>>(path: P)`
Set the model path (convenience method).

#### `with_model_name<S: Into<String>>(name: S)`
Set the model name (convenience method).

#### `with_context_size(size: usize)`
Set the context size (convenience method).

#### `with_n_batch(batch: u32)`
Set the batch size for prompt processing (convenience method).

#### `with_n_ubatch(ubatch: u32)`
Set the micro-batch size for prompt processing (convenience method).

#### `with_n_threads(threads: usize)`
Set the number of threads (convenience method).

#### `with_n_gpu_layers(layers: u32)`
Set the number of GPU layers (convenience method).

#### `with_normalization_mode(mode: NormalizationMode)`
Set the normalization mode for embeddings (convenience method).

#### `with_pooling_strategy(strategy: PoolingStrategy)`
Set the pooling strategy (convenience method).

#### `with_use_mmap(use_mmap: bool)`
Set whether to use memory mapping (convenience method).

#### `with_use_mlock(use_mlock: bool)`
Set whether to use memory locking (convenience method).

#### `with_n_seq_max(n_seq_max: u32)`
Set the maximum number of sequences for batch processing (convenience method). Default: 1, max: 64 (llama.cpp limit).

#### `with_use_gpu(use_gpu: bool)`
Set whether to use GPU.

#### `with_batch_size(size: usize)`
Set the batch size.

#### `with_max_tokens(tokens: usize)`
Set the maximum tokens.

#### `with_memory_limit_mb(limit_mb: usize)`
Set the memory limit in MB.

#### `with_verbose(verbose: bool)`
Set verbose logging.

#### `with_seed(seed: u32)`
Set the seed for reproducibility.

#### `with_temperature(temperature: f32)`
Set the temperature.

#### `with_cache_config(cache: CacheConfig)`
Set cache configuration.

#### `with_cache_enabled()`
Enable caching with default configuration.

#### `with_cache_disabled()`
Disable caching.

#### `with_embedding_config(embedding: EmbeddingConfig)`
Set embedding configuration.

#### `with_truncate_tokens(truncate: TruncateTokens)`
Set truncation strategy (convenience method).

#### `with_truncate_limit(limit: u32)`
Set truncation to a specific limit (convenience method).

#### `build()`
Build the configuration.

##### Errors
Returns an error if the configuration validation fails.
```

--------------------------------

### Enforce Request Size Limit

Source: https://docs.rs/embellama/0.10.1/embellama/server/middleware/fn.limit_request_size.html

Use this middleware to return StatusCode::PAYLOAD_TOO_LARGE if the request body size exceeds MAX_REQUEST_SIZE. No specific setup or imports are required beyond its inclusion in the middleware chain.

```rust
pub async fn limit_request_size(
    request: Request,
    next: Next,
) -> Result<Response, StatusCode>
```

--------------------------------

### VZip::vzip() Method

Source: https://docs.rs/embellama/0.10.1/embellama/struct.GGUFMetadata.html

Zips lanes together.

```APIDOC
## fn vzip(self) -> V

### Description
Zips lanes together.

### Source
[Source Link]
```

--------------------------------

### Calculate Product of Parsed Numbers from Iterator

Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html

This example shows how to use the `product` method on an iterator of `Result`s. If any element fails to parse, the entire operation returns an `Err`. Otherwise, it returns the product of all parsed numbers.

```rust
let nums = vec!["5", "10", "1", "2"];
let total: Result<usize, _> = nums.iter().map(|w| w.parse::<usize>()).product();
assert_eq!(total, Ok(100));
let nums = vec!["5", "10", "one", "2"];
let total: Result<usize, _> = nums.iter().map(|w| w.parse::<usize>()).product();
assert!(total.is_err());
```

--------------------------------

### Sum Integers with Negative Number Check in Rust

Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html

This example demonstrates summing integers in a vector while ensuring no negative numbers are present. It uses a closure to check for negative values and returns an Err if one is found. The `sum()` method on iterators is used with a `Result` type.

```rust
let f = |&x: &i32| if x < 0 { Err("Negative element found") } else { Ok(x) };
let v = vec![1, 2];
let res: Result<i32, _> = v.iter().map(f).sum();
assert_eq!(res, Ok(3));
let v = vec![1, -2];
let res: Result<i32, _> = v.iter().map(f).sum();
assert_eq!(res, Err("Negative element found"));
```

--------------------------------

### PrefixCache Module Overview

Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/index.html

Overview of the structs available in the embellama::cache::prefix_cache module for managing KV cache sessions.

```APIDOC
## Module: embellama::cache::prefix_cache

### Description
This module provides session-based prefix caching to reuse KV cache computations for common text prefixes, achieving significant speedup for repetitive patterns.

### Structs
- **PrefixCache**: Main prefix cache manager.
- **PrefixCacheStats**: Statistics for prefix cache.
- **PrefixDetector**: Prefix detector that analyzes text patterns for caching opportunities.
- **SessionData**: Represents a saved KV cache session state.
```
Tokens:
Raw