### Test Point Iteration Setup Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Sets up a random dataset and initializes an HNSW graph for testing point iteration. Demonstrates graph construction with specific parameters. ```rust println!("\n\n test_iter_point"); // let mut rng = rand::rng(); let unif = Uniform::::new(0., 1.).unwrap(); let nbcolumn = 5000; let nbrow = 10; let mut xsi; let mut data = Vec::with_capacity(nbcolumn); for j in 0..nbcolumn { data.push(Vec::with_capacity(nbrow)); for _ in 0..nbrow { xsi = rng.sample(unif); data[j].push(xsi); } } // // check insertion let ef_construct = 25; let nb_connection = 10; let start = ProcessTime::now(); let hns = Hnsw::::new( ``` -------------------------------- ### Reload HNSW with mmap Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.HnswIo.html Example demonstrating how to reload a previously dumped HNSW structure using memory-mapping for data files. Ensure HnswIo is not dropped before the reloaded HNSW. ```rust let directory = Path::new("."); let mut reloader = HnswIo::new(directory, "mmapreloadtest"); let options = ReloadOptions::default().set_mmap(true); reloader.set_options(options); let hnsw_loaded : Hnsw= reloader.load_hnsw::().unwrap(); ``` -------------------------------- ### HNSW Utility Functions Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/libext/index.html Utility functions for HNSW, including getting HNSWIO pointers and initializing Rust logging. ```APIDOC ## drop_hnsw_f32 ### Description Safety ## drop_hnsw_u16 ### Description Safety ## get_hnswio ### Description Returns a pointer to a Hnswio args corresponds to string giving base filename of dump, supposed to be in current directory. ## init_rust_log ### Description To initialize rust logging from Julia. ``` -------------------------------- ### Get TypeId of a Type Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/datamap/struct.DataMap.html Gets the `TypeId` of `self`. This is part of the `Any` trait implementation. ```rust fn type_id(&self) -> TypeId ``` -------------------------------- ### DumpInit::new Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.DumpInit.html Creates a new DumpInit instance. It takes a directory path, a default basename, and a boolean indicating whether to overwrite existing files. ```APIDOC ## DumpInit::new ### Description Creates a new `DumpInit` instance. ### Parameters - `dir` (&Path): The directory where the dump will be stored. - `basename_default` (&str): The default base name for the dump files. - `overwrite` (bool): If true, existing files will be overwritten. ### Returns - Self: A new `DumpInit` instance. ``` -------------------------------- ### get_data_dimension Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Gets the dimensionality of the data vectors stored in the HNSW graph. ```APIDOC ## get_data_dimension ### Description Returns the dimensionality of the data vectors stored within the HNSW structure. This is determined by the dimension of the entry point's vector. ### Method `pub fn get_data_dimension(&self) -> usize` ### Returns - `usize`: The dimension of the data vectors, or 0 if the structure is empty. ``` -------------------------------- ### DumpInit constructor Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.DumpInit.html Creates a new DumpInit instance. Specify the directory, a default basename, and whether to overwrite existing files. ```rust pub fn new(dir: &Path, basename_default: &str, overwrite: bool) -> Self ``` -------------------------------- ### Type ID Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.IterPointLayer.html Gets the `TypeId` of `self`. This is part of the `Any` trait implementation. ```APIDOC ## fn type_id(&self) -> TypeId ### Description Gets the `TypeId` of `self`. ### Returns - `TypeId`: The unique identifier for the type of `self`. ``` -------------------------------- ### HnswIo::init Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Initializes the HNSW I/O by opening the graph and data files. It prepares the necessary file handles for subsequent loading operations. ```APIDOC ## HnswIo::init ### Description Initializes the HNSW I/O by opening the graph and data files. It prepares the necessary file handles for subsequent loading operations. ### Method (Implicitly called by `load_hnsw`) ### Parameters None explicitly documented for direct user invocation. ### Returns - `Result`: On success, returns a `LoadInit` struct containing file handles and description. On failure, returns an error indicating inability to open files. ``` -------------------------------- ### Get Distance Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Returns a reference to the distance function used in the HNSW construction. ```rust pub fn get_distance(&self) -> &D { &self.dist_f } ``` -------------------------------- ### Get Ef Construction Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Retrieves the `ef_construction` value used during graph construction. ```rust pub fn get_ef_construction(&self) -> usize { self.ef_construction } ``` -------------------------------- ### Initialize HnswIo with Options Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.HnswIo.html Creates a new HnswIo instance with a specified directory, basename, and custom ReloadOptions, avoiding a separate call to set_options. ```rust pub fn new_with_options( directory: &Path, basename: &str, options: ReloadOptions, ) -> Self ``` -------------------------------- ### Hnsw::new Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Initializes a new Hnsw instance with specified parameters for graph construction. ```APIDOC ## Hnsw::new ### Description Allocation function for creating a new Hnsw instance. ### Parameters * `max_nb_connection` (usize) - Maximum number of neighbors stored per layer. Must be less than 256. * `max_elements` (usize) - Hint for allocation tables, representing the expected number of elements. * `max_layer` (usize) - The maximum number of layers allowed in construction. * `ef_construction` (usize) - Controls the number of neighbors explored during construction. See README or paper. * `f` (D) - The distance function to be used. ### Returns * `Self` - A new Hnsw instance. ``` -------------------------------- ### Get Distance Name Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/datamap.rs.html Returns the name of the distance metric used by the DataMap. ```rust pub fn get_distname(&self) -> String { self.distname.clone() } ``` -------------------------------- ### Get Data Dimension Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.Description.html Returns the dimension of the data as stored in the Description struct. ```rust pub fn get_dimension(&self) -> usize ``` -------------------------------- ### Initialize HnswIo Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.HnswIo.html Creates a new HnswIo instance with a specified directory and basename for dump files. Uses default ReloadOptions. ```rust pub fn new(directory: &Path, basename: &str) -> Self ``` -------------------------------- ### DumpInit::new Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Initializes a structure for dumping HNSW data and graph. It checks for existing dump files and generates a unique filename if overwrite is false to prevent data loss. ```APIDOC ## DumpInit::new ### Description Initializes a structure for dumping HNSW data and graph. It checks for existing dump files and generates a unique filename if overwrite is false to prevent data loss. ### Method `pub fn new(dir: &Path, basename_default: &str, overwrite: bool) -> Self` ### Parameters - **dir** (`&Path`) - The directory where the dump files will be created. - **basename_default** (`&str`) - The default base name for the dump files. - **overwrite** (`bool`) - If true, existing files with the same name will be overwritten. If false, a unique name will be generated if a conflict exists. ### Returns - `Self` - An instance of `DumpInit` with initialized file writers for data and graph. ``` -------------------------------- ### Get Origin ID Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Point.html Retrieves the external or client-provided ID associated with the Point. ```rust pub fn get_origin_id(&self) -> usize ``` -------------------------------- ### Get Point ID Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Point.html Returns the unique identifier for the Point within the index. ```rust pub fn get_point_id(&self) -> PointId ``` -------------------------------- ### Logging Initialization Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/all.html Function to initialize Rust logging. ```APIDOC ## Logging Initialization Initializes the Rust logging system, which can be useful for debugging and monitoring the library's operations. ### `libext::init_rust_log` Initializes the Rust logging facade. ``` -------------------------------- ### Check and Update Entry Point Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Compares a new point's ID with the current entry point and updates the entry point if the new point has a higher ID. Initializes the entry point if it's currently None. ```rust let mut entry_point_ref = self.entry_point.write(); match entry_point_ref.as_ref() { Some(arc_point) => { if new_point.p_id.0 > arc_point.p_id.0 { debug!("Hnsw , inserting entry point {:?} ", new_point.p_id); debug!( "PointIndexation insert setting max level from {:?} to ?", arc_point.p_id.0, new_point.p_id.0 ); *entry_point_ref = Some(Arc::clone(new_point)); } } None => { trace!("initializing entry point"); debug!("Hnsw , inserting entry point {:?} ", new_point.p_id); *entry_point_ref = Some(Arc::clone(new_point)); } } ``` -------------------------------- ### LayerGenerator Get Scale Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Retrieves the current scale factor used for level generation. ```rust fn get_level_scale(&self) -> f64 { self.scale } } ``` -------------------------------- ### Get Distance Function Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Retrieves a reference to the distance function used in the HNSW construction. ```rust pub fn get_distance(&self) -> &D ``` -------------------------------- ### Initialize HNSW Graph and Data Files Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Initializes the graph and data files for HNSW operations. Handles file opening and error checking, returning a LoadInit struct containing file handles and description. ```rust pub fn init(&mut self) -> Result> { // same thing for graph file let mut graphname = self.basename.clone(); graphname.push_str(".hnsw.graph"); let mut graphpath = self.dir.clone(); graphpath.push(graphname); let graphfileres = OpenOptions::new().read(true).open(&graphpath); if graphfileres.is_err() { println!( "HnswIo::init : could not open file {:?}", graphpath.as_os_str() ); error!( "HnswIo::init : could not open file {:?}", graphpath.as_os_str() ); return Err(anyhow!( "HnswIo::init : could not open file {:?}", graphpath.as_os_str() )); } let graphfile = graphfileres.unwrap(); // same thing for data file let mut dataname = self.basename.clone(); dataname.push_str(".hnsw.data"); let mut datapath = self.dir.clone(); datapath.push(dataname); let datafileres = OpenOptions::new().read(true).open(&datapath); if datafileres.is_err() { println!( "HnswIo::init : could not open file {:?}", datapath.as_os_str() ); error!( "HnswIo::init : could not open file {:?}", datapath.as_os_str() ); return Err(anyhow!( "HnswIo::init : could not open file {:?}", datapath.as_os_str() )); } let datafile = datafileres.unwrap(); // let mut graph_in = BufReader::new(graphfile); let data_in = BufReader::new(datafile); // we need to call load_description first to get distance name let hnsw_description = load_description(&mut graph_in).unwrap(); // Ok(LoadInit { descr: hnsw_description, graphfile: graph_in, datafile: data_in, }) } ``` -------------------------------- ### Get Distance Name Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Returns the name of the distance function used during the HNSW construction. ```rust pub fn get_distance_name(&self) -> String ``` -------------------------------- ### Implement Default for DistL2 Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/prelude/struct.DistL2.html Allows DistL2 to be created using its default value. This is useful for initialization. ```rust impl Default for DistL2 { fn default() -> Self { DistL2 } } ``` -------------------------------- ### Get Max Level Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Returns the maximum layer authorized during the construction of the HNSW graph. ```rust pub fn get_max_level(&self) -> usize ``` -------------------------------- ### Create HnswIo Instance with Custom Options Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Constructs a new HnswIo instance with a specified directory, basename, and custom reload options. ```rust pub fn new_with_options(directory: &Path, basename: &str, options: ReloadOptions) -> Self { HnswIo { dir: directory.to_path_buf(), basename: basename.to_string(), options, datamap: None, nb_point_loaded: Arc::new(AtomicUsize::new(0)), initialized: true, } } ``` -------------------------------- ### Get Vector Data Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Point.html Retrieves a reference to the vector data stored within the Point. ```rust pub fn get_v(&self) -> &[T] ``` -------------------------------- ### Implement CloneToUninit for T (Nightly) Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.NoData.html Nightly-only experimental API for performing copy-assignment to uninitialized memory. ```rust unsafe fn clone_to_uninit(&self, dest: *mut u8); ``` -------------------------------- ### Get Nb Point Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Returns the total number of points currently stored in the HNSW structure. ```rust pub fn get_nb_point(&self) -> usize { self.layer_indexed_points.get_nb_point() } ``` -------------------------------- ### New Hnsw Instance Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Initializes a new HNSW graph. Ensure `max_nb_connection` is less than 256. `ef_construction` controls the number of neighbors explored during graph construction. ```rust pub fn new( max_nb_connection: usize, max_elements: usize, max_layer: usize, ef_construction: usize, f: D, ) -> Self ``` -------------------------------- ### Get Layer Iterator Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Provides an iterator over the points stored in a specific layer of the HNSW structure. ```rust pub fn get_layer_iterator<'a>(&'a self, layer: usize) -> IterPointLayer<'a, 'b, T> { IterPointLayer::new(self, layer) } ``` -------------------------------- ### Initialize Rust Logging from Julia Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html Initializes the Rust logging system using environment variables. This function is intended to be called from Julia to set up logging for the Rust backend. ```rust /// to initialize rust logging from Julia #[unsafe(no_mangle)] pub extern "C" fn init_rust_log() { let _res = env_logger::Builder::from_default_env().try_init(); } ``` -------------------------------- ### Initialize Hnsw Graph Loading Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Internal method to prepare for loading an HNSW graph. It constructs the graph file path and attempts to open it for reading. ```rust fn init(&self) -> Result { // info!("reloading from basename : {}", &self.basename); // let mut graphname = self.basename.clone(); graphname.push_str(".hnsw.graph"); let mut graphpath = self.dir.clone(); graphpath.push(graphname); let graphfileres = OpenOptions::new().read(true).open(&graphpath); if graphfileres.is_err() { println!( "HnswIo::init : could not open file {:?}", graphpath.as_os_str() ); error!( "HnswIo::init : could not open file {:?}", graphpath.as_os_str() ); return Err(anyhow!( ``` -------------------------------- ### Get Data Type Name Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/datamap.rs.html Returns the full data type name stored in the DataMap. ```rust pub fn get_data_typename(&self) -> String { self.t_name.clone() } ``` -------------------------------- ### Get Data Type Name Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.Description.html Returns the data typename stored within the Description struct. ```rust pub fn get_typename(&self) -> String ``` -------------------------------- ### HnswIo::new_with_options Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.HnswIo.html Constructs a new HnswIo instance with specified reload options. This allows for custom configurations, such as enabling memory-mapping. ```APIDOC ## HnswIo::new_with_options ### Description Initializes a new `HnswIo` instance with custom `ReloadOptions`. This method is similar to `new` but allows immediate configuration of reload behavior, such as enabling memory-mapping (`mmap`). ### Signature ```rust pub fn new_with_options(directory: &Path, basename: &str, options: ReloadOptions) -> Self ``` ### Parameters * `directory`: A reference to a `Path` specifying the directory containing the dumped HNSW files. * `basename`: A string slice representing the base name for the dump files. * `options`: A `ReloadOptions` struct to configure the reloading process. ``` -------------------------------- ### HnswIo::new Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.HnswIo.html Constructs a new HnswIo instance with default reload options. The directory should contain the dumped HNSW files, and the basename is used to construct the filenames. ```APIDOC ## HnswIo::new ### Description Initializes a new `HnswIo` instance. It takes a directory path where the HNSW data and graph files are stored, and a basename used to construct the filenames (e.g., `$basename.hnsw.data` and `$basename.hnsw.graph`). This constructor uses default `ReloadOptions`. ### Signature ```rust pub fn new(directory: &Path, basename: &str) -> Self ``` ### Parameters * `directory`: A reference to a `Path` specifying the directory containing the dumped HNSW files. * `basename`: A string slice representing the base name for the dump files. ``` -------------------------------- ### Get Max Nb Connection Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Returns the maximum number of connections allowed per point in each layer. ```rust pub fn get_max_nb_connection(&self) -> u8 { self.max_nb_connection as u8 } ``` -------------------------------- ### init_rust_log Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/libext/fn.init_rust_log.html Initializes Rust logging from Julia. This function is marked as unsafe and has no name mangling. ```APIDOC ## Function init_rust_log ### Summary ```rust #[unsafe(no_mangle)] pub extern "C" fn init_rust_log() ``` ### Description Initializes Rust logging from Julia. ``` -------------------------------- ### Get Max Level Observed Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Returns the highest layer level actually observed in the HNSW structure. ```rust pub fn get_max_level_observed(&self) -> u8 { self.layer_indexed_points.get_max_level_observed() } ``` -------------------------------- ### From Implementation for DescriptionFFI Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/libext/struct.DescriptionFFI.html Enables conversion into `DescriptionFFI` from the same type. ```rust fn from(t: T) -> T Returns the argument unchanged. ``` -------------------------------- ### Get Total Number of Points Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Retrieves the total number of points currently stored in the HNSW structure. ```rust pub fn get_nb_point(&self) -> usize { *self.nb_point.read() } ``` -------------------------------- ### Get Data Dimension from Description Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Retrieves the data dimension stored within the HNSW Description struct. ```rust pub fn get_dimension(&self) -> usize { self.dimension } ``` -------------------------------- ### Hnswio Initialization Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html Creates and returns a pointer to a HnswIo structure. This structure is used for managing HNSW data persistence. ```APIDOC ## get_hnswio ### Description Returns a pointer to a Hnswio structure, which is used for managing HNSW data persistence. The filename provided is expected to be in the current directory. ### Safety This function is unsafe because it dereferences a raw pointer and assumes the provided name is a valid C-style string. ### Parameters - **flen** (u64) - The length of the filename string. - **name** (*const u8) - A C-style string representing the base filename for the HNSW dump. ### Returns - (*const HnswIo) - A raw pointer to the newly created HnswIo structure. ``` -------------------------------- ### PointIndexation::new Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html#1-1843 Constructs a new PointIndexation instance. It initializes the layers with expected sizes based on the provided parameters and sets up the layer generator and other internal structures. ```APIDOC ## PointIndexation::new ### Description Creates a new HNSW point indexation. ### Parameters - `max_nb_connection` (usize): The maximum number of connections a point can have. - `max_layer` (usize): The maximum number of layers in the index. - `max_elements` (usize): The maximum number of elements the index can hold. ### Returns - `PointIndexation<'b, T>`: A new instance of PointIndexation. ``` -------------------------------- ### Get Number of Data Entries Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/datamap.rs.html Returns the total number of data entries currently stored in the DataMap. ```rust pub fn get_nb_data(&self) -> usize { self.hmap.len() } ``` -------------------------------- ### Get Number of Points Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Returns the total number of points currently stored within the HNSW structure. ```rust pub fn get_nb_point(&self) -> usize ``` -------------------------------- ### HNSW Initialization (f32) Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html Initializes an HNSW index for f32 data with a specified distance metric. The function takes raw pointers and returns a raw pointer to the initialized HNSW API structure. It supports various predefined distance metrics like DistL1, DistL2, DistDot, etc. ```APIDOC ## init_hnsw_f32 ### Description Initializes an HNSW index for f32 data with a specified distance metric. ### Parameters - `max_nb_conn` (usize): Maximum number of neighbors for each node. - `max_elements` (usize): Maximum number of elements the index can hold. - `max_layer` (usize): Maximum number of layers in the HNSW graph. - `ef_const` (usize): Constant for the efSearch parameter. - `cdistname` (*const i8): A C-style string representing the distance metric name (e.g., "DistL1", "DistL2"). - `namelen` (usize): The length of the `cdistname` string. ### Returns - `*const HnswApif32`: A raw pointer to the initialized HNSW API structure, or a null pointer if the distance metric is unknown. ``` -------------------------------- ### PointIndexation::new Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Constructs a new PointIndexation instance. It initializes the internal layers with expected sizes based on the provided parameters. ```APIDOC ## PointIndexation::new ### Description Initializes a new `PointIndexation` with specified maximum connections, layers, and elements. It pre-allocates capacity for layers based on a calculated fraction of elements per layer. ### Signature ```rust pub fn new(max_nb_connection: usize, max_layer: usize, max_elements: usize) -> Self ``` ### Parameters - `max_nb_connection` (usize): The maximum number of neighbors a point can have. - `max_layer` (usize): The maximum number of layers in the index. - `max_elements` (usize): The maximum number of elements the index is expected to hold. ``` -------------------------------- ### Get Observed Max Level Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Returns the highest layer that has been reached in the HNSW graph's layers. ```rust pub fn get_max_level_observed(&self) -> u8 ``` -------------------------------- ### Initialize Rust Logging Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html Initializes the Rust logging system using environment variables. This function is intended to be called from external environments (e.g., Julia) to configure logging behavior. ```APIDOC ## init_rust_log ### Description Initializes the Rust logging system from the environment. This allows external applications to control Rust logging levels and formats. ### Function Signature `pub extern "C" fn init_rust_log()` ### Usage Call this function once at the beginning of your application's lifecycle to set up logging. ### Example ```rust // In Julia: // extern void init_rust_log(); // ccall(foreign_library_handle, init_rust_log, Cvoid) // In Rust (called from C): init_rust_log(); ``` ``` -------------------------------- ### Get Mmap Configuration Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.ReloadOptions.html Retrieves the current memory mapping configuration, returning a tuple of (datamap, threshold). ```rust pub fn use_mmap(&self) -> (bool, usize) ``` -------------------------------- ### Initialize HNSW with u16 and various distance metrics Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html Initializes a HNSW structure for u16 data with support for different distance metrics like DistL1, DistL2, DistHamming, DistJaccard, and DistLevenshtein. This function is unsafe due to raw pointer dereferencing. ```rust pub unsafe extern "C" fn new_hnsw_u16( max_nb_conn: usize, ef_const: usize, namelen: usize, cdistname: *const u8, max_elements: usize, max_layer: usize, ) -> *const HnswApiu16 { info!("entering init_hnsw_u16"); let slice = unsafe { std::slice::from_raw_parts(cdistname, namelen) }; let dname = String::from_utf8_lossy(slice); // map distname to sthg. This whole block will go to a macro if dname == "DistL1" { info!(" received DistL1"); let h = Hnsw::::new(max_nb_conn, max_elements, max_layer, ef_const, DistL1 {}); let api = HnswApiu16 { opaque: Box::new(h), }; return Box::into_raw(Box::new(api)); } else if dname == "DistL2" { let h = Hnsw::::new(max_nb_conn, max_elements, max_layer, ef_const, DistL2 {}); let api = HnswApiu16 { opaque: Box::new(h), }; return Box::into_raw(Box::new(api)); } else if dname == "DistHamming" { let h = Hnsw::::new( max_nb_conn, max_elements, max_layer, ef_const, DistHamming {}, ); let api = HnswApiu16 { opaque: Box::new(h), }; return Box::into_raw(Box::new(api)); } else if dname == "DistJaccard" { let h = Hnsw::::new( max_nb_conn, max_elements, max_layer, ef_const, DistJaccard {}, ); let api = HnswApiu16 { opaque: Box::new(h), }; return Box::into_raw(Box::new(api)); } else if dname == "DistLevenshtein" { let h = Hnsw::::new( max_nb_conn, max_elements, max_layer, ef_const, DistLevenshtein {}, ); let api = HnswApiu16 { opaque: Box::new(h), }; return Box::into_raw(Box::new(api)); } ptr::null::() } // end of init_hnsw_u16 ``` -------------------------------- ### Get HnswIo Basename Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.HnswIo.html Retrieves the basename used for constructing dump file names from an existing HnswIo instance. ```rust pub fn get_basename(&self) -> &str ``` -------------------------------- ### init_hnsw_i32 Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/libext/fn.init_hnsw_i32.html Initializes the HNSW structure for i32 data. This function is marked as unsafe due to raw pointer dereferencing. ```APIDOC ## Function init_hnsw_i32 ### Signature ```rust #[unsafe(no_mangle)] pub unsafe extern "C" fn init_hnsw_i32( max_nb_conn: usize, ef_const: usize, namelen: usize, cdistname: *const u8, ) -> *const HnswApii32 ``` ### Safety This function is unsafe because it dereferences raw pointers. ``` -------------------------------- ### Create New HNSW Index (f32) Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/libext/fn.new_hnsw_f32.html Initializes a new HNSW index for f32 data. This function is unsafe due to raw pointer dereferencing. It requires parameters defining the index's structure and capacity. ```rust #[unsafe(no_mangle)] pub unsafe extern "C" fn new_hnsw_f32( max_nb_conn: usize, ef_const: usize, namelen: usize, cdistname: *const u8, max_elements: usize, max_layer: usize, ) -> *const HnswApif32 ``` -------------------------------- ### Get Max Nb Connection Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Retrieves the maximum number of links a point can have to other points in each layer of the HNSW graph. ```rust pub fn get_max_nb_connection(&self) -> u8 ``` -------------------------------- ### Point::new Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Constructs a new Point with vector data. Initializes neighbors for all layers. ```APIDOC ## Point::new ### Description Constructs a new `Point` with vector data. Initializes neighbors for all layers up to `NB_LAYER_MAX`. ### Signature ```rust pub fn new(v: Vec, origin_id: usize, p_id: PointId) -> Self ``` ### Parameters * `v` (Vec): The vector data for the point. * `origin_id` (usize): The client-provided unique identifier for the point. * `p_id` (PointId): The internal identifier for the point within the HNSW structure. ``` -------------------------------- ### Get Distance Name from DataMap Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/datamap/struct.DataMap.html Retrieves the full name of the distance metric used in the Hnsw structure as a String. ```rust pub fn get_distname(&self) -> String ``` -------------------------------- ### Create HnswIo Instance Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Constructs a new HnswIo instance with a specified directory and basename for dump files. Uses default reload options. ```rust pub fn new(directory: &Path, basename: &str) -> Self { HnswIo { dir: directory.to_path_buf(), basename: basename.to_string(), options: ReloadOptions::default(), datamap: None, nb_point_loaded: Arc::new(AtomicUsize::new(0)), initialized: true, } } ``` -------------------------------- ### Get Data Type Name from DataMap Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/datamap/struct.DataMap.html Retrieves the full name of the data type stored in the DataMap as a String. ```rust pub fn get_data_typename(&self) -> String ``` -------------------------------- ### DistPtr::new Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/prelude/struct.DistPtr.html Constructs a new DistPtr instance with a given distance function. ```APIDOC ## DistPtr::new ### Description Constructs a new `DistPtr` instance with a given distance function. ### Signature ```rust pub fn new(f: fn(&[T], &[T]) -> F) -> DistPtr ``` ### Parameters * `f`: A function pointer that takes two slices of type `&[T]` and returns a value of type `F` (where `F` implements the `Float` trait), representing the distance between the two slices. ``` -------------------------------- ### enumerate() Method Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.IterPoint.html Creates an iterator that yields pairs of (index, element), where index is the iteration count starting from 0. ```rust fn enumerate(self) -> Enumerate where Self: Sized, ``` -------------------------------- ### Get Data Type Name from Description Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnswio.rs.html Retrieves the data type name stored within the HNSW Description struct. ```rust pub fn get_typename(&self) -> String { self.t_name.clone() } ``` -------------------------------- ### HNSW Constructor Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Initializes a new HNSW graph. `max_nb_connection` must be less than or equal to 256. `max_layer` is capped by `NB_LAYER_MAX`. ```rust pub fn new( max_nb_connection: usize, max_elements: usize, max_layer: usize, ef_construction: usize, f: D, ) -> Self { let adjusted_max_layer = (NB_LAYER_MAX as usize).min(max_layer); let layer_indexed_points = PointIndexation::::new(max_nb_connection, adjusted_max_layer, max_elements); let extend_candidates = false; let keep_pruned = false; // if max_nb_connection > 256 { println!("error max_nb_connection must be less equal than 256"); std::process::exit(1); } // info!("Hnsw max_nb_connection {:?}", max_nb_connection); info!("Hnsw nb elements {:?}", max_elements); info!("Hnsw ef_construction {:?}", ef_construction); info!("Hnsw distance {:?}", type_name::()); info!("Hnsw extend candidates {:?}", extend_candidates); // Hnsw { max_nb_connection, ef_construction, extend_candidates, keep_pruned, max_layer: adjusted_max_layer, layer_indexed_points, data_dimension: 0, dist_f: f, searching: false, datamap_opt: false, } } // end of new ``` -------------------------------- ### Get Point Indexation Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Provides access to the `PointIndexation` structure, which manages the internal indexing of points within the HNSW graph. ```rust pub fn get_point_indexation(&self) -> &PointIndexation<'b, T> ``` -------------------------------- ### Load HNSW Description from File Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html Loads the HNSW graph description from a file specified by a C-style string pointer and length. This function is unsafe due to raw pointer dereferencing. ```rust /// returns a const pointer to a DescriptionFFI from a dump file, given filename length and pointer (*const u8) /// # Safety /// This function is unsafe because it dereferences raw pointers. /// #[unsafe(no_mangle)] pub unsafe extern "C" fn load_hnsw_description( flen: usize, name: *const u8, ) -> *const DescriptionFFI { // opens file let slice = unsafe { std::slice::from_raw_parts(name, flen) }; let filename = String::from_utf8_lossy(slice).into_owned(); let fpath = PathBuf::from(filename); let fileres = OpenOptions::new().read(true).open(&fpath); // let mut ffi_description = DescriptionFFI::new(); match fileres { Ok(file) => { // let mut bufr = BufReader::with_capacity(10000000, file); let res = load_description(&mut bufr); if let Ok(description) = res { let distname = String::clone(&description.distname); let distname_ptr = distname.as_ptr(); let distname_len = distname.len(); std::mem::forget(distname); let t_name = String::clone(&description.t_name); let t_name_ptr = t_name.as_ptr(); let t_name_len = t_name.len(); std::mem::forget(t_name); ffi_description.dumpmode = 1; // CAVEAT ffi_description.max_nb_connection = description.max_nb_connection; ffi_description.nb_layer = description.nb_layer; ffi_description.ef = description.ef; ffi_description.data_dimension = description.dimension; ffi_description.distname_len = distname_len; ffi_description.distname = distname_ptr; ffi_description.t_name_len = t_name_len; ffi_description.t_name = t_name_ptr; Box::into_raw(Box::new(ffi_description)) } else { error!( "could not get descrption of hnsw from file {:?}", fpath.as_os_str() ); println!( "could not get descrption of hnsw from file {:?} ", ``` -------------------------------- ### init_hnsw_u32 Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/libext/fn.init_hnsw_u32.html Initializes an HNSW structure for u32 data. This function is unsafe because it dereferences raw pointers. ```APIDOC ## Function init_hnsw_u32 ### Signature ```rust #[unsafe(no_mangle)] pub unsafe extern "C" fn init_hnsw_u32( max_nb_conn: usize, ef_const: usize, namelen: usize, cdistname: *const u8, ) -> *const HnswApiu32 ``` ### Safety This function is unsafe because it dereferences raw pointers. ``` -------------------------------- ### Get basename for DumpInit Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnswio/struct.DumpInit.html Retrieves the basename used for the dump. This might be a unique name to avoid overwriting previous dumps. ```rust pub fn get_basename(&self) -> &String ``` -------------------------------- ### Initialize HNSW with various distance metrics (f32) Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html This function initializes an HNSW index for f32 data, supporting multiple distance metrics like DistL1, DistL2, DistDot, etc. It takes parameters for maximum connections, elements, layer size, and ef_const. The function returns a raw pointer to the HnswApif32 structure. ```rust let slice = unsafe { std::slice::from_raw_parts(cdistname, namelen) }; let dname = String::from_utf8_lossy(slice); // map distname to sthg. This whole block will go to a macro match dname.as_ref() { "DistL1" => { info!(" received DistL1"); let h = Hnsw::::new(max_nb_conn, max_elements, max_layer, ef_const, DistL1 {}); let api = HnswApif32 { opaque: Box::new(h), }; Box::into_raw(Box::new(api)) } "DistL2" => { let h = Hnsw::::new(max_nb_conn, max_elements, max_layer, ef_const, DistL2 {}); let api = HnswApif32 { opaque: Box::new(h), }; Box::into_raw(Box::new(api)) } "DistDot" => { let h = Hnsw::::new( max_nb_conn, max_elements, max_layer, ef_const, DistDot {}, ); let api = HnswApif32 { opaque: Box::new(h), }; Box::into_raw(Box::new(api)) } "DistHellinger" => { let h = Hnsw::::new( max_nb_conn, max_elements, max_layer, ef_const, DistHellinger {}, ); let api = HnswApif32 { opaque: Box::new(h), }; Box::into_raw(Box::new(api)) } "DistJeffreys" => { let h = Hnsw::::new( max_nb_conn, max_elements, max_layer, ef_const, DistJeffreys {}, ); let api = HnswApif32 { opaque: Box::new(h), }; Box::into_raw(Box::new(api)) } "DistJensenShannon" => { let h = Hnsw::::new( max_nb_conn, max_elements, max_layer, ef_const, DistJensenShannon {}, ); let api = HnswApif32 { opaque: Box::new(h), }; Box::into_raw(Box::new(api)) } _ => { warn!("init_hnsw_f32 received unknow distance {:?} ", dname); ptr::null::() } } // znd match ``` -------------------------------- ### Get Data Dimension Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Determines the dimensionality of the data vectors stored in the HNSW graph. Returns 0 if no entry point is set. ```rust pub fn get_data_dimension(&self) -> usize { let ep = self.entry_point.read(); match ep.as_ref() { Some(point) => point.get_v().len(), None => 0, } } ``` -------------------------------- ### Get HNSW IO Pointer Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/libext.rs.html Creates and returns a raw pointer to a HnswIo struct. The filename is derived from the provided byte slice. ```rust #[unsafe(no_mangle)] pub unsafe extern "C" fn get_hnswio(flen: u64, name: *const u8) -> *const HnswIo { let slice = unsafe { std::slice::from_raw_parts(name, flen as usize) }; let filename = String::from_utf8_lossy(slice).into_owned(); let hnswio = HnswIo::new(std::path::Path::new("."), &filename); Box::into_raw(Box::new(hnswio)) } ``` -------------------------------- ### HNSW Graph Traversal and Candidate Management Source: https://docs.rs/hnsw_rs/latest/src/hnsw_rs/hnsw.rs.html Illustrates the process of traversing the HNSW graph, managing candidate points using a binary heap, and maintaining a set of visited points. This snippet is crucial for understanding the search mechanism within a specific layer of the HNSW graph. ```rust let mut visited_point_id = HashMap::>>::new(); visited_point_id.insert(entry_point.p_id, Arc::clone(&entry_point)); // let mut candidate_points = BinaryHeap::>>::with_capacity(skiplist_size); candidate_points.push(Arc::new(PointWithOrder::new( &entry_point, -dist_to_entry_point, ))); return_points.push(Arc::new(PointWithOrder::new( &entry_point, dist_to_entry_point, ))); // at the beginning candidate_points contains point passed as arg in layer entry_point_id.0 while !candidate_points.is_empty() { // get nearest point in candidate_points let c = candidate_points.pop().unwrap(); // f farthest point to let f = return_points.peek().unwrap(); assert!(f.dist_to_ref >= 0.); assert!(c.dist_to_ref <= 0.); trace!( "Comparaing c : {:?} f : ?", -(c.dist_to_ref), f.dist_to_ref ); if -(c.dist_to_ref) > f.dist_to_ref { // this comparison requires that we are sure that distances compared are distances to the same point : // This is the case we compare distance to point passed as arg. trace!( "Fast return from search_layer, nb points : {:?} \n \t c {:?} \n \t f {:?} dists: {:?} {:?}", return_points.len(), c.point_ref.p_id, f.point_ref.p_id, -(c.dist_to_ref), f.dist_to_ref ); if filter.is_none() { return return_points; } else if return_points.len() >= ef { return_points.retain(|p| { filter .as_ref() .unwrap() .hnsw_filter(&p.point_ref.get_origin_id()) }); } } // now we scan neighborhood of c in layer and increment visited_point, candidate_points // and optimize candidate_points so that it contains points with lowest distances to point arg // let neighbours_c_l = &c.point_ref.neighbours.read()[layer as usize]; let c_pid = c.point_ref.p_id; trace!( " search_layer, {:?} has nb neighbours : {:?} ", c_pid, neighbours_c_l.len() ); for e in neighbours_c_l { // HERE WE sEE THAT neighbours should be stored as PointIdWithOrder !! // CAVEAT what if several point_id with same distance to ref point? if !visited_point_id.contains_key(&e.point_ref.p_id) { visited_point_id.insert(e.point_ref.p_id, Arc::clone(&e.point_ref)); trace!( ``` -------------------------------- ### Get ef_construction Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/hnsw/struct.Hnsw.html Retrieves the `ef_construction` parameter used during graph creation. This value influences the number of neighbors explored during the construction phase. ```rust pub fn get_ef_construction(&self) -> usize ``` -------------------------------- ### init_hnsw_ptrdist_f32 Source: https://docs.rs/hnsw_rs/latest/hnsw_rs/libext/fn.init_hnsw_ptrdist_f32.html Initializes an HNSW structure for f32 data. This function is unsafe and intended for C interoperation. ```APIDOC ## Function init_hnsw_ptrdist_f32 ### Description Initializes an HNSW structure for f32 data with a specified maximum number of connections and efConstruction value. It accepts a custom distance function pointer. ### Signature ```rust #[unsafe(no_mangle)] pub extern "C" fn init_hnsw_ptrdist_f32( max_nb_conn: usize, ef_const: usize, c_func: extern "C" fn(*const f32, *const f32, c_ulonglong) -> f32, ) -> *const HnswApif32 ``` ### Parameters * **max_nb_conn** (`usize`) - The maximum number of neighbors to connect for each node. * **ef_const** (`usize`) - The size of the dynamic list for the construction time. Controls the trade-off between construction speed and index quality. * **c_func** (`extern "C" fn(*const f32, *const f32, c_ulonglong) -> f32`) - A C-compatible function pointer that calculates the distance between two f32 vectors. It takes two pointers to f32 arrays and their size, returning the distance as an f32. ### Returns * `*const HnswApif32` - A raw pointer to the initialized HNSW structure. ```