### Run Example Cargo Project Source: https://docs.rs/embellama Command to run a specific example from the `examples/` directory. Replace `simple` with the desired example name. ```bash cargo run --example simple ``` -------------------------------- ### Build and Use Embedding Engine Source: https://docs.rs/embellama Quick start guide to build model and engine configurations, create an embedding engine, and generate single or batch embeddings. ```rust use embellama::{ModelConfig, EngineConfig, EmbeddingEngine, NormalizationMode}; // Build model configuration let model_config = ModelConfig::builder() .with_model_path("/path/to/model.gguf") .with_model_name("my-model") .with_normalization_mode(NormalizationMode::L2) .build()?; // Build engine configuration let engine_config = EngineConfig::builder() .with_model_config(model_config) .build()?; // Create engine let engine = EmbeddingEngine::new(engine_config)?; // Generate single embedding let text = "Hello, world!"; let embedding = engine.embed(None, text)?; // Generate batch embeddings let texts = vec!["Text 1", "Text 2", "Text 3"]; let embeddings = engine.embed_batch(None, &texts)?; ``` -------------------------------- ### Changelog Generation Commands Source: https://docs.rs/embellama Commands to generate and preview changelogs using git-cliff. Install git-cliff with `cargo install git-cliff`. ```bash just changelog # Regenerate CHANGELOG.md just changelog-unreleased # Preview unreleased changes ``` -------------------------------- ### Initialize embellama library Source: https://docs.rs/embellama/0.10.1/embellama/fn.init.html Call this function once at the start of your application to configure the global tracing subscriber. ```rust embellama::init(); ``` -------------------------------- ### Examples of Result Usage Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html Demonstrates summing integers in a vector with error handling for negative elements. ```APIDOC ## Examples ### Description This sums up every integer in a vector, rejecting the sum if a negative element is encountered. ### Method N/A (code example) ### Endpoint N/A ### Parameters None ### Request Example ```rust let f = |&x: &i32| if x < 0 { Err("Negative element found") } else { Ok(x) }; let v = vec![1, 2]; let res: Result = v.iter().map(f).sum(); assert_eq!(res, Ok(3)); let v = vec![1, -2]; let res: Result = v.iter().map(f).sum(); assert_eq!(res, Err("Negative element found")); ``` ### Response N/A #### Response Example N/A ``` -------------------------------- ### GET /v1/models Source: https://docs.rs/embellama/0.10.1/embellama/server/handlers/fn.list_models_handler.html Handler for listing all available models in an OpenAI-compatible format. ```APIDOC ## GET /v1/models ### Description Lists all available models in OpenAI-compatible format. ### Method GET ### Endpoint /v1/models ``` -------------------------------- ### Support Source: https://docs.rs/embellama/0.10.1/embellama/index.html Information on how to get support for Embellama. ```APIDOC ### Support For issues and questions, please use the GitHub issue tracker. ``` -------------------------------- ### GET /metrics Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/index.html Exports all collected server metrics in Prometheus format. ```APIDOC ## GET /metrics ### Description Exports all collected metrics in Prometheus format for observability and monitoring. ### Method GET ### Endpoint /metrics ### Response #### Success Response (200) - **Content-Type** (text/plain) - Prometheus metrics exposition format ``` -------------------------------- ### GET /v1/models Source: https://docs.rs/embellama/0.10.1/embellama/server/handlers/index.html Handler for listing available models via the OpenAI-compatible API. ```APIDOC ## GET /v1/models ### Description Handler for listing available models. ### Method GET ### Endpoint /v1/models ``` -------------------------------- ### Get Version Info Source: https://docs.rs/embellama/0.10.1/embellama/fn.version_info.html Retrieves the version information for the embellama crate. ```APIDOC ## GET /version_info ### Description Get version information for the embellama crate. ### Method GET ### Endpoint /version_info ### Response #### Success Response (200) - **version** (string) - The version string of the embellama crate. #### Response Example ```json { "version": "0.10.1" } ``` ``` -------------------------------- ### Default BatchProcessorBuilder Implementation Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessorBuilder.html Provides a default `BatchProcessorBuilder` instance, which can be used as a starting point for configuration. ```rust fn default() -> BatchProcessorBuilder ``` -------------------------------- ### Get Model Path Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Retrieve the file path to the model file on disk. ```rust let path = model.path(); ``` -------------------------------- ### Create New CacheMetrics Instance Source: https://docs.rs/embellama/0.10.1/embellama/cache/metrics/struct.CacheMetrics.html Initializes a new CacheMetrics instance with all counters set to zero. This is the default way to start tracking metrics. ```rust pub fn new() -> Self ``` -------------------------------- ### run_server Function Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.run_server.html The run_server function is a convenience function that creates the application state, builds the router, and starts the server with graceful shutdown handling. ```APIDOC ## run_server ### Description Run the server with the provided configuration. This is a convenience function that creates the application state, builds the router, and starts the server with graceful shutdown handling. ### Method ASYNC FUNCTION ### Endpoint N/A (This is a function call, not a direct HTTP endpoint) ### Parameters #### Arguments - **config** (ServerConfig) - Required - Server configuration ### Returns Result indicating success or failure ### Errors Returns an error if: * Application state creation fails * Server binding fails * Server startup fails ### Example ```rust let config = ServerConfig::builder() .model_path("/path/to/model.gguf") .model_name("my-model") .host("0.0.0.0") .port(8080) .build()?; run_server(config).await?; ``` ``` -------------------------------- ### Get Model Configuration Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Retrieve a reference to the `ModelConfig` used to initialize the embedding model. ```rust let config = model.config(); ``` -------------------------------- ### Get Recommended GPU Layers Source: https://docs.rs/embellama/0.10.1/embellama/enum.BackendType.html Retrieves the recommended number of GPU layers for the backend. Returns None for CPU backends and effectively all layers for GPU backends. ```rust pub const fn recommended_gpu_layers(&self) -> Option ``` -------------------------------- ### Get or Initialize Singleton EmbeddingEngine Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html This function retrieves the singleton EmbeddingEngine instance. If it's not initialized, it creates a new one using the provided configuration. The configuration is only applied on the first call. ```rust pub fn get_or_init(config: EngineConfig) -> Result>> ``` -------------------------------- ### Build and Run Server Configuration Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.run_server.html Demonstrates how to build a ServerConfig and then use it to run the server. Ensure the model path and name are correctly specified. ```rust let config = ServerConfig::builder() .model_path("/path/to/model.gguf") .model_name("my-model") .host("0.0.0.0") .port(8080) .build()?; run_server(config).await?; ``` -------------------------------- ### Initialize and extend a router with create_router Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.create_router.html Demonstrates initializing the application state and extending the default router with custom routes. ```rust let config = ServerConfig::builder() .model_path("/path/to/model.gguf") .build()?; let state = AppState::new(config)?; let router = create_router(state); // Add custom routes let router = router.route("/custom", axum::routing::get(|| async { "Custom route" })); // Start server... ``` -------------------------------- ### Get Compiled Backend - embellama Source: https://docs.rs/embellama/0.10.1/embellama/fn.get_compiled_backend.html Retrieves the currently compiled backend based on build features. This function is part of the embellama crate. ```rust pub fn get_compiled_backend() -> BackendType ``` -------------------------------- ### Initialize metrics Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/fn.init_metrics.html Call this function during server startup to initialize all metrics. ```rust pub fn init_metrics() ``` -------------------------------- ### Initialize Metrics Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/fn.init_metrics.html Initializes all metrics for the embellama server. This function should be called at server startup. ```APIDOC ## init_metrics ### Description Initializes all metrics (called at server startup). ### Method N/A (This is a function call, not an HTTP endpoint) ### Endpoint N/A ### Parameters None ### Request Example N/A ### Response N/A ``` -------------------------------- ### GET /v1/embeddings/prefix/stats Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_stats_handler.html Retrieves statistics related to the prefix cache. ```APIDOC ## GET /v1/embeddings/prefix/stats ### Description Handler for retrieving prefix cache statistics. ### Method GET ### Endpoint /v1/embeddings/prefix/stats ### Parameters #### Path Parameters - **__arg0** (State) - Required - The application state containing cache information. ``` -------------------------------- ### GET /v1/embeddings/prefix Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_list_handler.html Handler for retrieving a list of all cached prefixes. ```APIDOC ## GET /v1/embeddings/prefix ### Description Lists all cached prefixes. ### Method GET ### Endpoint /v1/embeddings/prefix ### Parameters ### Request Body ### Request Example ### Response #### Success Response (200) - **prefixes** (array) - A list of cached prefix strings. #### Response Example { "prefixes": ["prefix1", "prefix2"] } ``` -------------------------------- ### init function Source: https://docs.rs/embellama/0.10.1/embellama/fn.init.html Initializes the library with a default tracing subscriber to ensure proper logging infrastructure. ```APIDOC ## init ### Description Initializes the library with a default tracing subscriber. This function sets up a global tracing subscriber to ensure logging infrastructure outlives all threads, preventing thread-local storage panics during cleanup. ### Usage Call this function once at the start of your application. ### Request Example ```rust embellama::init(); ``` ``` -------------------------------- ### Create FileModelProvider Instance Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html Constructs a new FileModelProvider. It takes a model path and a model name as input. ```rust pub fn new( model_path: impl Into, model_name: impl Into, ) -> Self ``` -------------------------------- ### list_models_handler function signature Source: https://docs.rs/embellama/0.10.1/embellama/server/handlers/fn.list_models_handler.html The function signature for the GET /v1/models handler. ```rust pub async fn list_models_handler(__arg0: State) -> Response ``` -------------------------------- ### Basic Model and Engine Configuration Source: https://docs.rs/embellama Sets up basic model and engine configurations using the builder pattern. Requires specifying the model path and name. ```rust use embellama::{ModelConfig, EngineConfig}; let model_config = ModelConfig::builder() .with_model_path("/path/to/model.gguf") .with_model_name("my-model") .build()?; let config = EngineConfig::builder() .with_model_config(model_config) .build()?; ``` -------------------------------- ### Pointer and Initialization Utilities Source: https://docs.rs/embellama/0.10.1/embellama/enum.TruncateTokens.html Provides utilities for managing raw pointers, including alignment, initialization, dereferencing, and dropping. ```APIDOC ## Pointer and Initialization Utilities ### Description Utilities for managing raw pointers, including alignment, initialization, dereferencing, and dropping. ### Constants - **ALIGN**: `usize` - The alignment of the pointer. ### Functions - **init**: `unsafe fn init(init: ::Init) -> usize` - Initializes an object with the given initializer. - **deref**: `unsafe fn deref<'a>(ptr: usize) -> &'a T` - Dereferences the given pointer to provide an immutable reference. - **deref_mut**: `unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T` - Mutably dereferences the given pointer to provide a mutable reference. - **drop**: `unsafe fn drop(ptr: usize)` - Drops the object pointed to by the given pointer. ``` -------------------------------- ### AppState Initialization Source: https://docs.rs/embellama/0.10.1/embellama/server/state/struct.AppState.html Methods for creating and interacting with the application state. ```APIDOC ## AppState::new ### Description Creates a new application state instance. ### Parameters #### Arguments - **config** (ServerConfig) - Required - Server configuration ### Returns - **Result** - A new AppState instance or an error if engine configuration or initialization fails. ## AppState::model_name ### Description Retrieves the name of the model currently in use. ### Returns - **&str** - The model name. ## AppState::is_ready ### Description Checks if the server is ready for requests. ### Returns - **bool** - True if the server is ready, false otherwise. ``` -------------------------------- ### Library Initialization Source: https://docs.rs/embellama Functions for initializing the library and retrieving version information. ```APIDOC ## init ### Description Initialize the library with default tracing subscriber. ## init_with_env_filter ### Description Initialize the library with a custom environment filter. ## version_info ### Description Get version information. ``` -------------------------------- ### Get Model Name Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Retrieve the name of the embedding model as specified in its configuration. ```rust let name = model.name(); ``` -------------------------------- ### create_router Source: https://docs.rs/embellama/0.10.1/embellama/server/fn.create_router.html Initializes an Axum router with standard routes and middleware using the provided application state. ```APIDOC ## create_router ### Description Creates an Axum router with all the standard routes and middleware configured. Users can add additional routes or middleware before starting the server. ### Parameters - **state** (AppState) - Required - Application state containing the embedding engine and configuration ### Returns - **Router** - Configured Axum router ready to be served ### Request Example ```rust let config = ServerConfig::builder() .model_path("/path/to/model.gguf") .build()?; let state = AppState::new(config)?; let router = create_router(state); // Add custom routes let router = router.route("/custom", axum::routing::get(|| async { "Custom route" })); ``` ``` -------------------------------- ### GET /cache/stats Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.cache_stats_handler.html Retrieves current cache statistics and system memory information. ```APIDOC ## GET /cache/stats ### Description Handler for retrieving cache statistics and system memory information. ### Method GET ### Endpoint /cache/stats ### Parameters #### Path Parameters - **__arg0** (State) - Required - The application state containing cache and system information. ``` -------------------------------- ### Initialize embellama with environment filter Source: https://docs.rs/embellama/0.10.1/embellama/fn.init_with_env_filter.html Sets up a global tracing subscriber using the provided filter string to control logging verbosity. ```rust embellama::init_with_env_filter("embellama=debug,info"); ``` -------------------------------- ### Get Type ID Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixDetector.html Retrieves the `TypeId` of the implementing type. This is part of the `Any` trait. ```rust fn type_id(&self) -> TypeId ``` -------------------------------- ### Embed Embellama Server in Rust Applications Source: https://docs.rs/embellama/0.10.1/embellama/server/index.html Demonstrates how to configure the engine and server, then either run the server directly or integrate it into a custom router. ```rust use embellama::server::{ AppState, ServerConfig, create_router, run_server, EngineConfig, ModelConfig }; #[tokio::main] async fn main() -> Result<(), Box> { // Build engine configuration first let engine_config = EngineConfig::builder() .with_model_path("/path/to/model.gguf") .with_model_name("my-model") .build()?; // Option 1: Use the convenient run_server function let config = ServerConfig::builder() .engine_config(engine_config.clone()) .port(8080) .build()?; run_server(config.clone()).await?; // Option 2: Create router for custom integration let state = AppState::new(config)?; let router = create_router(state); // Add your own routes or middleware... Ok(()) } ``` -------------------------------- ### Get Cache Size Source: https://docs.rs/embellama/0.10.1/embellama/cache/trait.CacheStore.html Returns the current number of entries stored in the cache. ```rust fn len(&self) -> usize ``` -------------------------------- ### Build BatchProcessor Instance Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessorBuilder.html Constructs and returns a new `BatchProcessor` instance with the configurations set on the builder. ```rust pub fn build(self) -> BatchProcessor ``` -------------------------------- ### Model Management API Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html APIs for retrieving, setting, and getting information about embedding models. ```APIDOC ## GET /model/default ### Description Gets the default model name. ### Method GET ### Endpoint /model/default ### Response #### Success Response (200) - **model_name** (string) - The default model name if set. #### Response Example { "model_name": "model_v1" } ``` ```APIDOC ## POST /model/default ### Description Sets the default model. ### Method POST ### Endpoint /model/default ### Request Body - **model_name** (string) - Required - The name of the model to set as default. ### Request Example { "model_name": "model_v1" } ### Response #### Success Response (200) - **status** (string) - Indicates success or failure. #### Response Example { "status": "success" } ``` ```APIDOC ## GET /model/{model_name} ### Description Gets information about a loaded model. ### Method GET ### Endpoint /model/{model_name} ### Parameters #### Path Parameters - **model_name** (string) - Required - The name of the model. ### Response #### Success Response (200) - **ModelInfo** (object) - Information about the model. - **name** (string) - The name of the model. - **version** (string) - The version of the model. - **dimensions** (integer) - The embedding dimensions. #### Response Example { "name": "model_v1", "version": "1.0.0", "dimensions": 768 } ``` -------------------------------- ### FileModelProvider Constructor Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html Creates a new instance of the FileModelProvider. ```APIDOC ## Constructor: FileModelProvider::new ### Description Creates a new file-based model provider instance. ### Parameters - **model_path** (impl Into) - Required - The path to the GGUF model file. - **model_name** (impl Into) - Required - The name assigned to the model. ``` -------------------------------- ### GET /metrics/worker_utilization Source: https://docs.rs/embellama/0.10.1/embellama/server/metrics/static.WORKER_UTILIZATION.html Retrieves the current gauge value representing the worker thread utilization. ```APIDOC ## GET /metrics/worker_utilization ### Description Retrieves the current gauge value for worker thread utilization, represented as a float between 0.0 and 1.0. ### Method GET ### Endpoint /metrics/worker_utilization ### Response #### Success Response (200) - **value** (float) - The current utilization of worker threads (0.0 to 1.0). #### Response Example { "worker_utilization": 0.45 } ``` -------------------------------- ### Function init_with_env_filter Source: https://docs.rs/embellama/0.10.1/embellama/fn.init_with_env_filter.html Initializes the library with a custom environment filter. This function sets up a global tracing subscriber with a custom filter string. The global subscriber ensures logging is available during all cleanup operations, preventing thread-local storage issues. ```APIDOC ## Function init_with_env_filter ### Description Initializes the library with a custom environment filter. This function sets up a global tracing subscriber with a custom filter string. The global subscriber ensures logging is available during all cleanup operations, preventing thread-local storage issues. ### Arguments - **filter** (string) - Required - Environment filter string (e.g., “info”, “debug”, “embellama=debug”) ### Example ```rust embellama::init_with_env_filter("embellama=debug,info"); ``` ``` -------------------------------- ### Get Pointer Alignment Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixCache.html Defines the alignment of the pointer for the `Pointable` trait. This is a constant value. ```rust const ALIGN: usize ``` -------------------------------- ### EmbeddingEngine Initialization and Singleton Access Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html Methods for creating and accessing the singleton EmbeddingEngine instance. ```APIDOC ## POST /embedding_engine/get_or_init ### Description Gets or initializes the singleton embedding engine with the given configuration. If the engine is already initialized, returns the existing instance. The configuration is only used for the first initialization. ### Method POST ### Endpoint /embedding_engine/get_or_init ### Parameters #### Request Body - **config** (EngineConfig) - Required - The engine configuration (used only on first call) ### Request Example ```json { "config": { "model_path": "path/to/model.gguf", "model_name": "my-model" } } ``` ### Response #### Success Response (200) - **engine** (Arc>) - The singleton embedding engine instance. #### Response Example ```json { "engine": "..." } ``` ## GET /embedding_engine/instance ### Description Gets the existing engine instance if it has been initialized. ### Method GET ### Endpoint /embedding_engine/instance ### Response #### Success Response (200) - **engine** (Option>>) - Some(engine) if initialized, None otherwise. #### Response Example ```json { "engine": "..." } ``` ## POST /embedding_engine/new ### Description Creates a new embedding engine with the given configuration. Note: This now uses the singleton pattern internally. Use `get_or_init()` for explicit singleton access. ### Method POST ### Endpoint /embedding_engine/new ### Parameters #### Request Body - **config** (EngineConfig) - Required - The engine configuration ### Request Example ```json { "config": { "model_path": "path/to/model.gguf", "model_name": "my-model" } } ``` ### Response #### Success Response (200) - **engine** (EmbeddingEngine) - The newly created embedding engine. #### Response Example ```json { "engine": "..." } ``` ``` -------------------------------- ### Model Loading Behavior: Immediate vs. Lazy Source: https://docs.rs/embellama Illustrates the difference between immediate loading for the initial model and lazy loading for additional models. Lazy-loaded models are loaded on their first use. ```rust // First model - loaded immediately let mut engine = EmbeddingEngine::new(config)?; assert!(engine.is_model_loaded_in_thread("model1")); // Additional model - lazy loaded engine.load_model(config2)?; assert!(engine.is_model_registered("model2")); assert!(!engine.is_model_loaded_in_thread("model2")); // Not yet loaded // Triggers actual loading in thread engine.embed(Some("model2"), "text")?; assert!(engine.is_model_loaded_in_thread("model2")); // Now loaded ``` -------------------------------- ### GET metadata_cache_size Source: https://docs.rs/embellama/0.10.1/embellama/fn.metadata_cache_size.html Retrieves the current size of the GGUF metadata cache for monitoring and debugging purposes. ```APIDOC ## GET metadata_cache_size ### Description Get the current size of the GGUF metadata cache. This is primarily useful for monitoring and debugging. ### Method GET ### Endpoint metadata_cache_size ### Response #### Success Response (200) - **size** (usize) - The current size of the GGUF metadata cache. ``` -------------------------------- ### Implement CloneToUninit (Nightly) Source: https://docs.rs/embellama/0.10.1/embellama/cache/memory_monitor/struct.MemoryMonitorConfig.html Nightly-only experimental API for performing copy-assignment from self to an uninitialized destination. ```rust unsafe fn clone_to_uninit(&self, dest: *mut u8) ``` -------------------------------- ### Backend Auto-Detection and Configuration Source: https://docs.rs/embellama Configures the engine to automatically detect the best available backend and specifies the model path. Also shows how to check the selected backend and its features. ```rust use embellama::{EngineConfig, detect_best_backend, BackendInfo}; // Automatic backend detection let config = EngineConfig::with_backend_detection() .with_model_path("/path/to/model.gguf") .with_model_name("my-model") .build()?; // Check which backend was selected let backend_info = BackendInfo::new(); println!("Using backend: {}", backend_info.backend); println!("Available features: {:?}", backend_info.available_features); ``` -------------------------------- ### Define prefix_stats_handler Function Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_stats_handler.html Handler for GET /v1/embeddings/prefix/stats. Returns prefix cache statistics. Requires AppState. ```rust pub async fn prefix_stats_handler(__arg0: State) -> impl IntoResponse ``` -------------------------------- ### ClientRateLimiter Methods Source: https://docs.rs/embellama/0.10.1/embellama/server/rate_limiter/struct.ClientRateLimiter.html Methods for initializing and managing per-client rate limiters. ```APIDOC ## new(config: RateLimitConfig) ### Description Create a new per-client rate limiter instance. ### Parameters - **config** (RateLimitConfig) - Required - Configuration settings for the rate limiter. --- ## get_limiter(client_id: &str) ### Description Get or create a rate limiter for a specific client. ### Parameters - **client_id** (&str) - Required - The unique identifier (IP address) for the client. ### Errors - Panics if rate limiting is not enabled in the configuration. --- ## cleanup(max_age: Duration) ### Description Clean up old rate limiter entries. This should be called periodically to maintain performance. ### Parameters - **max_age** (Duration) - Required - The duration after which entries are considered old and eligible for removal. ``` -------------------------------- ### Function: detect_best_backend Source: https://docs.rs/embellama/0.10.1/embellama/fn.detect_best_backend.html Documentation for the detect_best_backend function which identifies the optimal backend. ```APIDOC ## detect_best_backend ### Description Detect the best available backend based on compile-time features and runtime capabilities. ### Method Function Call ### Endpoint embellama::detect_best_backend() ### Response - **BackendType** (enum) - The detected backend type based on system configuration. ``` -------------------------------- ### prefix_list_handler function signature Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.prefix_list_handler.html The function signature for the prefix_list_handler, which handles GET requests to list cached prefixes. ```rust pub async fn prefix_list_handler(__arg0: State) -> impl IntoResponse ``` -------------------------------- ### Initialize and Use EmbeddingEngine Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html Use this snippet to create an EmbeddingEngine instance with a specified model configuration and generate embeddings for text. Ensure the model path and name are correctly set. ```rust use embellama::{EmbeddingEngine, EngineConfig}; let config = EngineConfig::builder() .with_model_path("path/to/model.gguf") .with_model_name("my-model") .build()?; let engine = EmbeddingEngine::new(config)?; let embedding = engine.embed("my-model", "Hello, world!")?; ``` -------------------------------- ### Retrieve cache statistics handler signature Source: https://docs.rs/embellama/0.10.1/embellama/server/cache_handlers/fn.cache_stats_handler.html Defines the asynchronous handler function for the GET /cache/stats endpoint. ```rust pub async fn cache_stats_handler(__arg0: State) -> impl IntoResponse ``` -------------------------------- ### Conversion and Utility Methods Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.PrefixListResponse.html Standard conversion and vector-like zipping operations. ```APIDOC ## fn try_into(self) -> Result>::Error> ### Description Performs the conversion from type T to U. ## fn vzip(self) -> V ### Description Performs a multi-lane zip operation on the provided type. ``` -------------------------------- ### Get SessionData Age in Seconds Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.SessionData.html Calculates and returns the age of the session in seconds based on its creation timestamp. ```rust pub fn age_seconds(&self) -> u64 ``` -------------------------------- ### Implement Clone for Usage Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.Usage.html Provides methods to duplicate Usage instances. Use `clone` for a simple copy and `clone_from` for in-place assignment. ```rust fn clone(&self) -> Usage ``` ```rust fn clone_from(&mut self, source: &Self) ``` -------------------------------- ### Get Model Path for FileModelProvider Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html Retrieves the path to a model file by its name. This is part of the ModelProvider trait implementation. ```rust fn get_model_path<'life0, 'life1, 'async_trait>( &'life0 self, model_name: &'life1 str, ) -> Pin> + Send + 'async_trait>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, ``` -------------------------------- ### Worker::new Source: https://docs.rs/embellama/0.10.1/embellama/server/worker/struct.Worker.html Creates a new worker instance. ```APIDOC ## Worker::new ### Description Create a new worker ### Method ```rust pub fn new( id: usize, engine: Arc>, receiver: Receiver, ) -> Self ``` ### Arguments - **id** (usize) - Worker identifier - **engine** (Arc>) - Shared embedding engine instance - **receiver** (Receiver) - Channel to receive requests ### Returns A new `Worker` instance ``` -------------------------------- ### Get Value from Cache Source: https://docs.rs/embellama/0.10.1/embellama/cache/trait.CacheStore.html Retrieves a value associated with a given key from the cache. Returns None if the key is not found. ```rust fn get(&self, key: &K) -> Option ``` -------------------------------- ### Get Maximum Sequences for Batch Processing Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Returns the maximum number of sequences that can be processed in parallel within a single batch. ```rust let n_seq_max = model.n_seq_max(); ``` -------------------------------- ### BatchProcessor::builder Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessor.html Initiates the creation of a batch processor with custom configuration options. ```APIDOC ## BatchProcessor::builder ### Description Creates a batch processor with custom configuration. ### Method `pub fn builder() -> BatchProcessorBuilder` ### Returns Returns a `BatchProcessorBuilder` instance for further configuration. ``` -------------------------------- ### Get Embedding Model Dimensions Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Retrieve the dimensionality of the embedding vectors that the model produces. This indicates the size of the output vectors. ```rust let dimensions = model.embedding_dimensions(); ``` -------------------------------- ### Build RateLimiter from Config Source: https://docs.rs/embellama/0.10.1/embellama/server/rate_limiter/struct.RateLimitConfig.html Creates a new rate limiter instance based on the provided RateLimitConfig. Panics if requests_per_second or burst_size is 0. ```rust pub fn build_limiter(&self) -> Option ``` -------------------------------- ### Dereference Pointer Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixDetector.html Dereferences a raw pointer to get an immutable reference to the object. This is an unsafe operation and part of the `Pointable` trait. ```rust unsafe fn deref<'a>(ptr: usize) -> &'a T ``` -------------------------------- ### Chain Fallible Operations with and_then Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html Used to chain operations that return a Result. The second example demonstrates usage with filesystem metadata. ```rust fn sq_then_to_string(x: u32) -> Result { x.checked_mul(x).map(|sq| sq.to_string()).ok_or("overflowed") } assert_eq!(Ok(2).and_then(sq_then_to_string), Ok(4.to_string())); assert_eq!(Ok(1_000_000).and_then(sq_then_to_string), Err("overflowed")); assert_eq!(Err("not a number").and_then(sq_then_to_string), Err("not a number")); ``` ```rust use std::{io::ErrorKind, path::Path}; // Note: on Windows "/" maps to "C:\" let root_modified_time = Path::new("/").metadata().and_then(|md| md.modified()); assert!(root_modified_time.is_ok()); let should_fail = Path::new("/bad/path").metadata().and_then(|md| md.modified()); assert!(should_fail.is_err()); assert_eq!(should_fail.unwrap_err().kind(), ErrorKind::NotFound); ``` -------------------------------- ### into_ok() - Unfailing Ok extraction (Nightly) Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html Returns the contained `Ok` value, but never panics. This is a nightly-only experimental API. ```APIDOC ## into_ok() ### Description Returns the contained `Ok` value, but never panics. Unlike `unwrap`, this method is known to never panic on the result types it is implemented for. This is a nightly-only experimental API. ### Method `into_ok()` ### Parameters None ### Request Example ```rust fn only_good_news() -> Result { Ok("this is fine".into()) } let s: String = only_good_news().into_ok(); println!("{s}"); ``` ### Response #### Success Response (T) Returns the contained `Ok` value. ``` -------------------------------- ### VZip Implementation Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.ErrorDetail.html Describes the `VZip` implementation and its `vzip` method. ```APIDOC ## impl VZip for T ### Description Implementation for VZip. ## fn vzip(self) -> V ### Description Performs a zipped operation. ### Method N/A (Associated function) ### Endpoint N/A ``` -------------------------------- ### Get Model Metadata Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Access the model's metadata, including its name, file path, vocabulary size, and number of parameters. ```rust let (name, path, vocab_size, n_params) = model.model_metadata(); ``` -------------------------------- ### Get Model Size Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Retrieve the approximate memory footprint of the loaded model in bytes. This can be `None` on certain platforms or for very large models. ```rust let size = model.model_size(); ``` -------------------------------- ### Create New EmbeddingEngine Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html Instantiates a new EmbeddingEngine. Note that this method internally uses a singleton pattern, so `get_or_init()` is recommended for explicit singleton access. ```rust pub fn new(config: EngineConfig) -> Result ``` -------------------------------- ### Get Current Load Level Source: https://docs.rs/embellama/0.10.1/embellama/server/backpressure/struct.LoadShedder.html Retrieves the current load level of the load shedder. This value is typically updated by the `update_load` method. ```rust pub fn load(&self) -> f64 ``` -------------------------------- ### Detect Best Backend Source: https://docs.rs/embellama/0.10.1/embellama/fn.detect_best_backend.html Use this function to automatically select the optimal backend. It considers compile-time features and runtime capabilities. ```rust pub fn detect_best_backend() -> BackendType ``` -------------------------------- ### Get GGUF metadata cache size Source: https://docs.rs/embellama/0.10.1/embellama/fn.metadata_cache_size.html Returns the current size of the GGUF metadata cache as a usize. Useful for monitoring and debugging. ```rust pub fn metadata_cache_size() -> usize ``` -------------------------------- ### Retrieve version information Source: https://docs.rs/embellama/0.10.1/embellama/fn.version_info.html Returns the current version details for the Embellama crate. ```rust pub fn version_info() -> VersionInfo ``` -------------------------------- ### Dereference Pointer Mutably Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixDetector.html Dereferences a raw pointer to get a mutable reference to the object. This is an unsafe operation and part of the `Pointable` trait. ```rust unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T ``` -------------------------------- ### TryFrom and TryInto for Type Conversion Source: https://docs.rs/embellama/0.10.1/embellama/server/api_types/struct.CacheStatsResponse.html Demonstrates the usage of `TryFrom` and `TryInto` traits for fallible type conversions. ```APIDOC ## TryFrom for T ### Description Provides a fallible conversion from type `U` to type `T`. ### Methods #### fn try_from(value: U) -> Result>::Error> Performs the conversion. ### Type Alias #### type Error = Infallible The type returned in the event of a conversion error. ## TryInto for T ### Description Provides a fallible conversion from type `T` to type `U`. ### Methods #### fn try_into(self) -> Result>::Error> Performs the conversion. ### Type Alias #### type Error = >::Error The type returned in the event of a conversion error. ``` -------------------------------- ### Get Existing EmbeddingEngine Instance Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html Retrieves the current singleton EmbeddingEngine instance if it has already been initialized. Returns None if the engine has not been set up yet. ```rust pub fn instance() -> Option>> ``` -------------------------------- ### Create New PrefixCache Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/struct.PrefixCache.html Initializes a new PrefixCache instance. Returns an error if the session directory cannot be created. Configuration includes max sessions, TTL, frequency threshold, and an optional session directory path. ```rust pub fn new( max_sessions: usize, ttl_seconds: u64, frequency_threshold: usize, session_dir: Option, ) -> Result ``` -------------------------------- ### Get Maximum Sequence Length Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingModel.html Obtain the maximum number of tokens the model can process in a single input sequence. This is a fundamental constraint for inference. ```rust let max_len = model.max_sequence_length(); ``` -------------------------------- ### Model Loading Behavior Source: https://docs.rs/embellama/0.10.1/embellama/index.html Explains the loading behavior for initial and additional models in Embellama. ```APIDOC ## Model Loading Behavior ### Initial Model Loading - **Description**: The first model specified during `EmbeddingEngine::new()` is loaded immediately into the current thread. ### Additional Model Loading - **Description**: Models loaded via `load_model()` are lazily loaded. They are registered but not loaded into memory until their first use (e.g., during an `embed` or `rerank` call). ### Example: Lazy Loading ```rust use embellama::{EngineConfig, EmbeddingEngine}; // First model - loaded immediately let config1 = EngineConfig::builder().with_model_path("/path/to/model1.gguf").with_model_name("model1").build()?; let mut engine = EmbeddingEngine::new(config1)?; assert!(engine.is_model_loaded_in_thread("model1")); // Additional model - lazy loaded let config2 = EngineConfig::builder().with_model_path("/path/to/model2.gguf").with_model_name("model2").build()?; engine.load_model(config2)?; assert!(engine.is_model_registered("model2")); assert!(!engine.is_model_loaded_in_thread("model2")); // Not yet loaded // Triggers actual loading in thread let _ = engine.embed(Some("model2"), "some text")?; assert!(engine.is_model_loaded_in_thread("model2")); // Now loaded ``` ``` -------------------------------- ### ServerConfig Structure Source: https://docs.rs/embellama/0.10.1/embellama/server/state/struct.ServerConfig.html Details the fields and builder pattern for the ServerConfig struct used to initialize the server. ```APIDOC ## ServerConfig ### Description Represents the configuration settings for the embellama server instance. ### Fields - **worker_count** (usize) - Number of worker threads. - **queue_size** (usize) - Maximum pending requests per worker. - **host** (String) - Server host address. - **port** (u16) - Server port. - **request_timeout** (Duration) - Request timeout duration. - **engine_config** (EngineConfig) - Engine configuration including model settings. ### Methods - **builder()** -> ServerConfigBuilder: Creates a new builder for server configuration. ``` -------------------------------- ### Create Configuration Error Source: https://docs.rs/embellama/0.10.1/embellama/enum.Error.html Use this function to create a `ConfigurationError` with a custom message. This is useful for indicating issues with the library's setup or parameters. ```rust pub fn config(message: impl Into) -> Self ``` -------------------------------- ### HttpServerConnExec Implementation Source: https://docs.rs/embellama/0.10.1/embellama/struct.EmbeddingEngine.html Placeholder for `HttpServerConnExec` implementation. ```APIDOC ### impl HttpServerConnExec for T where B: Body, ### Source ``` -------------------------------- ### Get Cache Statistics Source: https://docs.rs/embellama/0.10.1/embellama/cache/trait.CacheStore.html Returns statistics about the cache, such as hit rate, miss rate, or memory usage. The exact content depends on the CacheStats type. ```rust fn stats(&self) -> CacheStats ``` -------------------------------- ### Reset Cache Metrics Source: https://docs.rs/embellama/0.10.1/embellama/cache/metrics/struct.CacheMetrics.html Resets all metric counters (hits, misses, evictions, memory_bytes) back to zero. Use this to start fresh metrics collection. ```rust pub fn reset(&self) ``` -------------------------------- ### SystemHealth Methods Source: https://docs.rs/embellama/0.10.1/embellama/server/backpressure/struct.SystemHealth.html Methods for initializing and interacting with the SystemHealth monitor. ```APIDOC ## SystemHealth Methods ### Description Methods to manage and query the health status of the system. ### Methods - **new(circuit_breaker: Arc, load_shedder: Arc, queue_threshold: usize) -> Self** - Creates a new system health monitor. - **update_queue_depth(depth: usize)** - Updates the current queue depth. - **is_healthy() -> bool** - Checks if the system is healthy enough to accept requests. - **status() -> HealthStatus** - Retrieves the current health status. ``` -------------------------------- ### Type Conversion: From/Into Source: https://docs.rs/embellama/0.10.1/embellama/struct.ModelConfig.html Demonstrates conversions using the `From` and `Into` traits. `Into` is implemented for `T` when `U: From`. ```APIDOC ## impl Into for T where U: From, ### Description Provides a way to convert a type `T` into another type `U` if `U` can be created from `T`. ### Method `into(self) -> U` ### Parameters - `self` - The instance of type `T` to convert. ### Request Example ```rust let t_instance: T = ...; let u_instance: U = t_instance.into(); ``` ### Response #### Success Response (200) - `U` - The converted value of type `U`. #### Response Example ```rust // No direct response, conversion happens in place. ``` ``` -------------------------------- ### Get reference to Result value Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html The `as_ref` method converts a `&Result` into a `Result<&T, &E>`, providing references to the contained values without consuming the original Result. ```rust let x: Result = Ok(2); assert_eq!(x.as_ref(), Ok(&2)); let x: Result = Err("Error"); assert_eq!(x.as_ref(), Err(&"Error")); ``` -------------------------------- ### List Models for FileModelProvider Source: https://docs.rs/embellama/0.10.1/embellama/server/struct.FileModelProvider.html Lists all available models. This is part of the ModelProvider trait implementation. ```rust fn list_models<'life0, 'async_trait>( &'life0 self, ) -> Pin>> + Send + 'async_trait>> where Self: 'async_trait, 'life0: 'async_trait, ``` -------------------------------- ### Get mutable reference to Result value Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html The `as_mut` method converts a `&mut Result` into a `Result<&mut T, &mut E>`, providing mutable references to the contained values. ```rust fn mutate(r: &mut Result) { match r.as_mut() { Ok(v) => *v = 42, Err(e) => *e = 0, } } let mut x: Result = Ok(2); mutate(&mut x); assert_eq!(x.unwrap(), 42); let mut x: Result = Err(13); mutate(&mut x); assert_eq!(x.unwrap_err(), 0); ``` -------------------------------- ### EngineConfigBuilder Methods Source: https://docs.rs/embellama/0.10.1/embellama/struct.EngineConfigBuilder.html Methods for building and configuring EngineConfig. ```APIDOC ## EngineConfigBuilder ### Description Builder for creating `EngineConfig` instances. ### Methods #### `new()` Create a new builder with default configuration. #### `with_model_config(model_config: ModelConfig)` Set the entire model configuration. #### `with_model_path>(path: P)` Set the model path (convenience method). #### `with_model_name>(name: S)` Set the model name (convenience method). #### `with_context_size(size: usize)` Set the context size (convenience method). #### `with_n_batch(batch: u32)` Set the batch size for prompt processing (convenience method). #### `with_n_ubatch(ubatch: u32)` Set the micro-batch size for prompt processing (convenience method). #### `with_n_threads(threads: usize)` Set the number of threads (convenience method). #### `with_n_gpu_layers(layers: u32)` Set the number of GPU layers (convenience method). #### `with_normalization_mode(mode: NormalizationMode)` Set the normalization mode for embeddings (convenience method). #### `with_pooling_strategy(strategy: PoolingStrategy)` Set the pooling strategy (convenience method). #### `with_use_mmap(use_mmap: bool)` Set whether to use memory mapping (convenience method). #### `with_use_mlock(use_mlock: bool)` Set whether to use memory locking (convenience method). #### `with_n_seq_max(n_seq_max: u32)` Set the maximum number of sequences for batch processing (convenience method). Default: 1, max: 64 (llama.cpp limit). #### `with_use_gpu(use_gpu: bool)` Set whether to use GPU. #### `with_batch_size(size: usize)` Set the batch size. #### `with_max_tokens(tokens: usize)` Set the maximum tokens. #### `with_memory_limit_mb(limit_mb: usize)` Set the memory limit in MB. #### `with_verbose(verbose: bool)` Set verbose logging. #### `with_seed(seed: u32)` Set the seed for reproducibility. #### `with_temperature(temperature: f32)` Set the temperature. #### `with_cache_config(cache: CacheConfig)` Set cache configuration. #### `with_cache_enabled()` Enable caching with default configuration. #### `with_cache_disabled()` Disable caching. #### `with_embedding_config(embedding: EmbeddingConfig)` Set embedding configuration. #### `with_truncate_tokens(truncate: TruncateTokens)` Set truncation strategy (convenience method). #### `with_truncate_limit(limit: u32)` Set truncation to a specific limit (convenience method). #### `build()` Build the configuration. ##### Errors Returns an error if the configuration validation fails. ``` -------------------------------- ### Create BatchProcessor with Builder Source: https://docs.rs/embellama/0.10.1/embellama/struct.BatchProcessor.html Initiates the creation of a batch processor using a builder pattern, allowing for custom configuration options before instantiation. ```rust pub fn builder() -> BatchProcessorBuilder ``` -------------------------------- ### Enforce Request Size Limit Source: https://docs.rs/embellama/0.10.1/embellama/server/middleware/fn.limit_request_size.html Use this middleware to return StatusCode::PAYLOAD_TOO_LARGE if the request body size exceeds MAX_REQUEST_SIZE. No specific setup or imports are required beyond its inclusion in the middleware chain. ```rust pub async fn limit_request_size( request: Request, next: Next, ) -> Result ``` -------------------------------- ### VZip::vzip() Method Source: https://docs.rs/embellama/0.10.1/embellama/struct.GGUFMetadata.html Zips lanes together. ```APIDOC ## fn vzip(self) -> V ### Description Zips lanes together. ### Source [Source Link] ``` -------------------------------- ### Calculate Product of Parsed Numbers from Iterator Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html This example shows how to use the `product` method on an iterator of `Result`s. If any element fails to parse, the entire operation returns an `Err`. Otherwise, it returns the product of all parsed numbers. ```rust let nums = vec!["5", "10", "1", "2"]; let total: Result = nums.iter().map(|w| w.parse::()).product(); assert_eq!(total, Ok(100)); let nums = vec!["5", "10", "one", "2"]; let total: Result = nums.iter().map(|w| w.parse::()).product(); assert!(total.is_err()); ``` -------------------------------- ### Sum Integers with Negative Number Check in Rust Source: https://docs.rs/embellama/0.10.1/embellama/type.Result.html This example demonstrates summing integers in a vector while ensuring no negative numbers are present. It uses a closure to check for negative values and returns an Err if one is found. The `sum()` method on iterators is used with a `Result` type. ```rust let f = |&x: &i32| if x < 0 { Err("Negative element found") } else { Ok(x) }; let v = vec![1, 2]; let res: Result = v.iter().map(f).sum(); assert_eq!(res, Ok(3)); let v = vec![1, -2]; let res: Result = v.iter().map(f).sum(); assert_eq!(res, Err("Negative element found")); ``` -------------------------------- ### PrefixCache Module Overview Source: https://docs.rs/embellama/0.10.1/embellama/cache/prefix_cache/index.html Overview of the structs available in the embellama::cache::prefix_cache module for managing KV cache sessions. ```APIDOC ## Module: embellama::cache::prefix_cache ### Description This module provides session-based prefix caching to reuse KV cache computations for common text prefixes, achieving significant speedup for repetitive patterns. ### Structs - **PrefixCache**: Main prefix cache manager. - **PrefixCacheStats**: Statistics for prefix cache. - **PrefixDetector**: Prefix detector that analyzes text patterns for caching opportunities. - **SessionData**: Represents a saved KV cache session state. ```