### Load DiskANN index from file Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Use the from_file classmethod to initialize an index from a saved path. Ensure the index was created with valid tags and matches the original build parameters. ```python 145 f"The file {tags_file} does not exist in {index_directory}", 146 ) 147 vector_dtype, dap_metric, num_vectors, dimensions = _ensure_index_metadata( 148 index_prefix_path, vector_dtype, distance_metric, max_vectors, dimensions, warn_size_exceeded=True 149 ) 150 151 index = cls( 152 distance_metric=dap_metric, # type: ignore 153 vector_dtype=vector_dtype, 154 dimensions=dimensions, 155 max_vectors=max_vectors, 156 complexity=complexity, 157 graph_degree=graph_degree, 158 saturate_graph=saturate_graph, 159 max_occlusion_size=max_occlusion_size, 160 alpha=alpha, 161 num_threads=num_threads, 162 filter_complexity=filter_complexity, 163 num_frozen_points=num_frozen_points, 164 initial_search_complexity=initial_search_complexity, 165 search_threads=search_threads, 166 concurrent_consolidation=concurrent_consolidation, 167 ) 168 index._index.load(index_prefix_path) 169 index._num_vectors = num_vectors # current number of vectors loaded 170 return index ``` -------------------------------- ### StaticDiskIndex.__init__ Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Initializes a new instance of the StaticDiskIndex class, which loads a disk-backed index from the specified directory. ```APIDOC ## StaticDiskIndex(index_directory, num_threads, num_nodes_to_cache, cache_mechanism=1, distance_metric=None, vector_dtype=None, dimensions=None, index_prefix='ann') ### Description Initializes a disk-backed index for searching. The index directory must contain the required index files, or metadata must be provided. ### Parameters - **index_directory** (str) - Required - The directory containing the index files. - **num_threads** (int) - Required - Number of threads to use when searching. 0 uses system default. - **num_nodes_to_cache** (int) - Required - Number of nodes to cache in memory. - **cache_mechanism** (int) - Optional - 1 to initialize cache with sample data, 2 to ready cache without initialization, other values disable caching. - **distance_metric** (str) - Optional - One of {'l2', 'mips', 'cosine'}. Required if metadata file is missing. - **vector_dtype** (Type) - Optional - The vector data type. Required if metadata file is missing. - **dimensions** (int) - Optional - Vector dimensionality. Required if metadata file is missing. - **index_prefix** (str) - Optional - The prefix of the index files. Defaults to 'ann'. ``` -------------------------------- ### StaticDiskIndex.__init__ Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Initializes a new StaticDiskIndex, which is a disk-backed, immutable index. The index directory must contain the required index files. ```APIDOC ## StaticDiskIndex.__init__ ### Description Initializes a new instance of the StaticDiskIndex class. This index is disk-backed and not mutable. ### Parameters - **index_directory** (str) - Required - The directory containing the index files. - **num_threads** (int) - Required - Number of threads to use. - **num_nodes_to_cache** (int) - Required - Number of nodes to cache in memory. - **cache_mechanism** (int) - Optional - The caching mechanism to use. Default is 1. - **distance_metric** (DistanceMetric) - Optional - The distance metric used for the index. - **vector_dtype** (VectorDType) - Optional - The data type of the vectors. - **dimensions** (int) - Optional - The dimensionality of the vectors. - **index_prefix** (str) - Optional - The prefix of the index files. Defaults to "ann". ``` -------------------------------- ### Initialize StaticDiskIndex Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor for the immutable disk-backed index, requiring a directory containing pre-built index files. ```python 34class StaticDiskIndex: 35 """ 36 A StaticDiskIndex is a disk-backed index that is not mutable. 37 """ 38 39 def __init__( 40 self, 41 index_directory: str, 42 num_threads: int, 43 num_nodes_to_cache: int, 44 cache_mechanism: int = 1, 45 distance_metric: Optional[DistanceMetric] = None, 46 vector_dtype: Optional[VectorDType] = None, 47 dimensions: Optional[int] = None, 48 index_prefix: str = "ann", 49 ): 50 """ 51 ### Parameters 52 - **index_directory**: The directory containing the index files. This directory must contain the following 53 files: 54 - `{index_prefix}_sample_data.bin` 55 - `{index_prefix}_mem.index.data` 56 - `{index_prefix}_pq_compressed.bin` 57 - `{index_prefix}_pq_pivots.bin` 58 - `{index_prefix}_sample_ids.bin` 59 - `{index_prefix}_disk.index` 60 61 It may also include the following optional files: ``` -------------------------------- ### Initialize DiskANN Index Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Handles index metadata loading, filter configuration, and native index object instantiation. ```python index_prefix_path = _valid_index_prefix(index_directory, index_prefix) self._labels_map = {} self._labels_metadata = {} if enable_filters: try: with open(f"{index_prefix_path}_labels_map.txt", "r") as labels_map_if: for line in labels_map_if: (key, val) = line.split("\t") self._labels_map[key] = int(val) with open(f"{index_prefix_path}_label_metadata.json", "r") as labels_metadata_if: self._labels_metadata = json.load(labels_metadata_if) except: # noqa: E722 # exceptions are basically presumed to be either file not found or file not formatted correctly raise RuntimeException("Filter labels file was unable to be processed.") vector_dtype, metric, num_points, dims = _ensure_index_metadata( index_prefix_path, vector_dtype, distance_metric, 1, # it doesn't matter because we don't need it in this context anyway dimensions, ) dap_metric = _valid_metric(metric) _assert_is_nonnegative_uint32(num_threads, "num_threads") _assert_is_positive_uint32( initial_search_complexity, "initial_search_complexity" ) self._vector_dtype = vector_dtype self._dimensions = dims if vector_dtype == np.uint8: _index = _native_dap.StaticMemoryUInt8Index elif vector_dtype == np.int8: _index = _native_dap.StaticMemoryInt8Index else: _index = _native_dap.StaticMemoryFloatIndex self._index = _index( distance_metric=dap_metric, num_points=num_points, dimensions=dims, index_path=index_prefix_path, num_threads=num_threads, initial_search_complexity=initial_search_complexity, ) ``` -------------------------------- ### Initialize DynamicMemoryIndex Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor logic for setting up a new DynamicMemoryIndex, including parameter validation and native index instantiation. ```python self._num_vectors = 0 self._removed_num_vectors = 0 dap_metric = _valid_metric(distance_metric) self._dap_metric = dap_metric _assert_dtype(vector_dtype) _assert_is_positive_uint32(dimensions, "dimensions") self._vector_dtype = vector_dtype self._dimensions = dimensions _assert_is_positive_uint32(max_vectors, "max_vectors") _assert_is_positive_uint32(complexity, "complexity") _assert_is_positive_uint32(graph_degree, "graph_degree") _assert( alpha >= 1, "alpha must be >= 1, and realistically should be kept between [1.0, 2.0)", ) _assert_is_nonnegative_uint32(max_occlusion_size, "max_occlusion_size") _assert_is_nonnegative_uint32(num_threads, "num_threads") _assert_is_nonnegative_uint32(filter_complexity, "filter_complexity") _assert_is_nonnegative_uint32(num_frozen_points, "num_frozen_points") _assert_is_nonnegative_uint32( initial_search_complexity, "initial_search_complexity" ) _assert_is_nonnegative_uint32(search_threads, "search_threads") self._max_vectors = max_vectors self._complexity = complexity self._graph_degree = graph_degree if vector_dtype == np.uint8: _index = _native_dap.DynamicMemoryUInt8Index elif vector_dtype == np.int8: _index = _native_dap.DynamicMemoryInt8Index else: _index = _native_dap.DynamicMemoryFloatIndex self._index = _index( distance_metric=dap_metric, dimensions=dimensions, max_vectors=max_vectors, complexity=complexity, graph_degree=graph_degree, saturate_graph=saturate_graph, max_occlusion_size=max_occlusion_size, alpha=alpha, num_threads=num_threads, filter_complexity=filter_complexity, num_frozen_points=num_frozen_points, initial_search_complexity=initial_search_complexity, search_threads=search_threads, concurrent_consolidation=concurrent_consolidation, ) self._points_deleted = False ``` -------------------------------- ### Initialize StaticDiskIndex Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor for the StaticDiskIndex class, which validates index metadata and initializes the native disk-backed index. ```python 39 def __init__( 40 self, 41 index_directory: str, 42 num_threads: int, 43 num_nodes_to_cache: int, 44 cache_mechanism: int = 1, 45 distance_metric: Optional[DistanceMetric] = None, 46 vector_dtype: Optional[VectorDType] = None, 47 dimensions: Optional[int] = None, 48 index_prefix: str = "ann", 49 ): 50 """ 51 ### Parameters 52 - **index_directory**: The directory containing the index files. This directory must contain the following 53 files: 54 - `{index_prefix}_sample_data.bin` 55 - `{index_prefix}_mem.index.data` 56 - `{index_prefix}_pq_compressed.bin` 57 - `{index_prefix}_pq_pivots.bin` 58 - `{index_prefix}_sample_ids.bin` 59 - `{index_prefix}_disk.index` 60 61 It may also include the following optional files: 62 - `{index_prefix}_vectors.bin`: Optional. `diskannpy` builder functions may create this file in the 63 `index_directory` if the index was created from a numpy array 64 - `{index_prefix}_metadata.bin`: Optional. `diskannpy` builder functions create this file to store metadata 65 about the index, such as vector dtype, distance metric, number of vectors and vector dimensionality. 66 If an index is built from the `diskann` cli tools, this file will not exist. 67 - **num_threads**: Number of threads to use when searching this index. (>= 0), 0 = num_threads in system 68 - **num_nodes_to_cache**: Number of nodes to cache in memory (> -1) 69 - **cache_mechanism**: 1 -> use the generated sample_data.bin file for 70 the index to initialize a set of cached nodes, up to `num_nodes_to_cache`, 2 -> ready the cache for up to 71 `num_nodes_to_cache`, but do not initialize it with any nodes. Any other value disables node caching. 72 - **distance_metric**: A `str`, strictly one of {"l2", "mips", "cosine"}. `l2` and `cosine` are supported for all 3 73 vector dtypes, but `mips` is only available for single precision floats. Default is `None`. **This 74 value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it does not exist, 75 you are required to provide it. 76 - **vector_dtype**: The vector dtype this index has been built with. **This value is only used if a 77 `{index_prefix}_metadata.bin` file does not exist.** If it does not exist, you are required to provide it. 78 - **dimensions**: The vector dimensionality of this index. All new vectors inserted must be the same 79 dimensionality. **This value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it 80 does not exist, you are required to provide it. 81 - **index_prefix**: The prefix of the index files. Defaults to "ann". 82 """ 83 index_prefix_path = _valid_index_prefix(index_directory, index_prefix) 84 vector_dtype, metric, _, _ = _ensure_index_metadata( 85 index_prefix_path, 86 vector_dtype, 87 distance_metric, 88 1, # it doesn't matter because we don't need it in this context anyway 89 dimensions, 90 ) 91 dap_metric = _valid_metric(metric) 92 93 _assert_is_nonnegative_uint32(num_threads, "num_threads") 94 _assert_is_nonnegative_uint32(num_nodes_to_cache, "num_nodes_to_cache") 95 96 self._vector_dtype = vector_dtype 97 if vector_dtype == np.uint8: 98 _index = _native_dap.StaticDiskUInt8Index 99 elif vector_dtype == np.int8: 100 _index = _native_dap.StaticDiskInt8Index 101 else: 102 _index = _native_dap.StaticDiskFloatIndex 103 self._index = _index( 104 distance_metric=dap_metric, 105 index_path_prefix=index_prefix_path, 106 num_threads=num_threads, 107 num_nodes_to_cache=num_nodes_to_cache, 108 cache_mechanism=cache_mechanism, 109 ) ``` -------------------------------- ### from_file(index_prefix_path, ...) Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Loads a previously saved DiskANN index from disk. It is recommended to use the same parameters as those used during the index creation process. ```APIDOC ## from_file ### Description Loads a previously saved index from disk. The index must have been created with a valid tags file or tags array. ### Parameters - **index_prefix_path** (str) - Required - The directory path where the index files are stored. - **vector_dtype** (dtype) - Optional - Data type of the vectors. - **distance_metric** (str) - Optional - The distance metric used for the index. - **max_vectors** (int) - Optional - Maximum number of vectors the index can hold. - **dimensions** (int) - Optional - Number of dimensions for the vectors. - **complexity** (int) - Optional - Indexing complexity parameter. - **graph_degree** (int) - Optional - The degree of the graph. - **saturate_graph** (bool) - Optional - Whether to saturate the graph. - **max_occlusion_size** (int) - Optional - Maximum occlusion size. - **alpha** (float) - Optional - Alpha parameter for graph construction. - **num_threads** (int) - Optional - Number of threads for indexing. - **filter_complexity** (int) - Optional - Complexity for filtering. - **num_frozen_points** (int) - Optional - Number of frozen points. - **initial_search_complexity** (int) - Optional - Initial search complexity. - **search_threads** (int) - Optional - Number of threads for searching. - **concurrent_consolidation** (bool) - Optional - Whether to use concurrent consolidation. ``` -------------------------------- ### Load DynamicMemoryIndex from file Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Use this method to initialize a DynamicMemoryIndex from existing index files on disk. Ensure the index_directory contains the required .data, .tags, and index files. ```python def from_file( cls, index_directory: str, max_vectors: int, complexity: int, graph_degree: int, saturate_graph: bool = defaults.SATURATE_GRAPH, max_occlusion_size: int = defaults.MAX_OCCLUSION_SIZE, alpha: float = defaults.ALPHA, num_threads: int = defaults.NUM_THREADS, filter_complexity: int = defaults.FILTER_COMPLEXITY, num_frozen_points: int = defaults.NUM_FROZEN_POINTS_DYNAMIC, initial_search_complexity: int = 0, search_threads: int = 0, concurrent_consolidation: bool = True, index_prefix: str = "ann", distance_metric: Optional[DistanceMetric] = None, vector_dtype: Optional[VectorDType] = None, dimensions: Optional[int] = None, ) -> "DynamicMemoryIndex": ``` -------------------------------- ### StaticMemoryIndex.__init__ Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Initializes a new instance of the StaticMemoryIndex class by loading an existing index from the specified directory. ```APIDOC ## StaticMemoryIndex.__init__ ### Description Initializes an immutable in-memory DiskANN index from a specified directory. The index must contain the required data files, and optional metadata can be provided if the metadata file is missing. ### Parameters - **index_directory** (str) - Required - The directory containing the index files. - **num_threads** (int) - Required - Number of threads to use when searching this index (0 = system default). - **initial_search_complexity** (int) - Required - The initial complexity setting used to allocate scratch memory. - **index_prefix** (str) - Optional - The prefix of the index files (defaults to "ann"). - **distance_metric** (DistanceMetric) - Optional - One of {"l2", "mips", "cosine"}. Required if metadata file is missing. - **vector_dtype** (VectorDType) - Optional - The vector dtype. Required if metadata file is missing. - **dimensions** (int) - Optional - The vector dimensionality. Required if metadata file is missing. - **enable_filters** (bool) - Optional - Whether to enable filtered search. ``` -------------------------------- ### Initialize StaticMemoryIndex Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor for the immutable in-memory DiskANN index. Requires a directory containing index files and configuration for search threads and complexity. ```python def __init__( self, index_directory: str, num_threads: int, initial_search_complexity: int, index_prefix: str = "ann", distance_metric: Optional[DistanceMetric] = None, vector_dtype: Optional[VectorDType] = None, dimensions: Optional[int] = None, enable_filters: bool = False ): ``` -------------------------------- ### Initialize DynamicMemoryIndex Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor for creating a new, empty DynamicMemoryIndex. Use this for building an index from scratch rather than loading an existing one. ```python def __init__( self, distance_metric: DistanceMetric, vector_dtype: VectorDType, dimensions: int, max_vectors: int, complexity: int, graph_degree: int, saturate_graph: bool = defaults.SATURATE_GRAPH, max_occlusion_size: int = defaults.MAX_OCCLUSION_SIZE, alpha: float = defaults.ALPHA, num_threads: int = defaults.NUM_THREADS, filter_complexity: int = defaults.FILTER_COMPLEXITY, num_frozen_points: int = defaults.NUM_FROZEN_POINTS_DYNAMIC, initial_search_complexity: int = 0, search_threads: int = 0, concurrent_consolidation: bool = True, ): ``` -------------------------------- ### diskannpy.DynamicMemoryIndex.__init__ Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor for creating a new, empty DynamicMemoryIndex. ```APIDOC ## diskannpy.DynamicMemoryIndex.__init__ ### Description Initializes a new instance of a mutable DiskANN memory index. ### Parameters - **saturate_graph** (bool) - Optional - If True, the adjacency list of each node will be saturated with neighbors to have exactly graph_degree neighbors. - **max_occlusion_size** (int) - Optional - The maximum number of points that can be considered by occlude_list function. - **alpha** (float) - Optional - Controls the nature and number of points added to the graph (>=1). - **num_threads** (int) - Optional - Number of threads to use when creating this index. - **filter_complexity** (int) - Optional - Complexity to use when using filters. - **num_frozen_points** (int) - Optional - Number of points to freeze. - **initial_search_complexity** (int) - Optional - Expected complexity for search operations. - **search_threads** (int) - Optional - Expected number of threads for search operations. - **concurrent_consolidation** (bool) - Optional - Whether consolidation can run alongside inserts and deletes. ``` -------------------------------- ### StaticMemoryIndex.__init__ Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Initializes a new StaticMemoryIndex object, which represents an immutable in-memory DiskANN index. ```APIDOC ## StaticMemoryIndex(index_directory, num_threads, initial_search_complexity, ...) ### Description Initializes an immutable in-memory DiskANN index from a specified directory. ### Parameters - **index_directory** (str) - Required - The directory containing the index files. - **num_threads** (int) - Required - Number of threads to use when searching this index. - **initial_search_complexity** (int) - Required - The expected search complexity for memory allocation. - **index_prefix** (str) - Optional - The prefix of the index files. Defaults to "ann". - **distance_metric** (str) - Optional - One of {"l2", "mips", "cosine"}. Used if metadata file is missing. - **vector_dtype** (Type) - Optional - The vector dtype. Used if metadata file is missing. - **dimensions** (int) - Optional - The vector dimensionality of the index. - **enable_filters** (bool) - Optional - Whether to enable filtering. ``` -------------------------------- ### DynamicMemoryIndex.__init__ Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Initializes a new, empty DynamicMemoryIndex for storing and searching vectors. ```APIDOC ## DynamicMemoryIndex(distance_metric, vector_dtype, dimensions, max_vectors, complexity, graph_degree, ...) ### Description Creates a new, empty mutable DiskANN memory index. ### Parameters - **distance_metric** (str) - Required - One of {'l2', 'mips', 'cosine'}. - **vector_dtype** (type) - Required - One of {np.float32, np.int8, np.uint8}. - **dimensions** (int) - Required - Vector dimensionality. - **max_vectors** (int) - Required - Capacity of the data store. - **graph_degree** (int) - Required - Maximum degree allowed for a node in the graph. ``` -------------------------------- ### Load DynamicMemoryIndex from disk Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html This method initializes a DynamicMemoryIndex by loading an existing index from the specified directory. It validates the presence of required metadata and tag files before loading. ```python index_prefix_path = _valid_index_prefix(index_directory, index_prefix) # do tags exist? tags_file = index_prefix_path + ".tags" _assert( Path(tags_file).exists(), f"The file {tags_file} does not exist in {index_directory}", ) vector_dtype, dap_metric, num_vectors, dimensions = _ensure_index_metadata( index_prefix_path, vector_dtype, distance_metric, max_vectors, dimensions, warn_size_exceeded=True ) index = cls( distance_metric=dap_metric, # type: ignore vector_dtype=vector_dtype, dimensions=dimensions, max_vectors=max_vectors, complexity=complexity, graph_degree=graph_degree, saturate_graph=saturate_graph, max_occlusion_size=max_occlusion_size, alpha=alpha, num_threads=num_threads, filter_complexity=filter_complexity, num_frozen_points=num_frozen_points, initial_search_complexity=initial_search_complexity, search_threads=search_threads, concurrent_consolidation=concurrent_consolidation, ) index._index.load(index_prefix_path) index._num_vectors = num_vectors # current number of vectors loaded return index ``` -------------------------------- ### diskannpy.DynamicMemoryIndex.__init__ Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor to create a new, empty mutable DiskANN memory index. ```APIDOC ## DynamicMemoryIndex.__init__ ### Description Initializes a new, empty mutable DiskANN memory index. ### Parameters - **distance_metric** (str) - Required - One of {"l2", "mips", "cosine"}. - **vector_dtype** (type) - Required - One of {np.float32, np.int8, np.uint8}. - **dimensions** (int) - Required - Vector dimensionality. - **max_vectors** (int) - Required - Capacity of the data store. - **graph_degree** (int) - Required - Maximum degree allowed for a node in the graph. - **saturate_graph** (bool) - Optional - Whether to saturate the adjacency list of each node. ``` -------------------------------- ### Initialize StaticMemoryIndex Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructor for the StaticMemoryIndex class. Requires an index directory and configuration parameters for search performance and index metadata. ```python 34class StaticMemoryIndex: 35 """ 36 A StaticMemoryIndex is an immutable in-memory DiskANN index. 37 """ 38 39 def __init__( 40 self, 41 index_directory: str, 42 num_threads: int, 43 initial_search_complexity: int, 44 index_prefix: str = "ann", 45 distance_metric: Optional[DistanceMetric] = None, 46 vector_dtype: Optional[VectorDType] = None, 47 dimensions: Optional[int] = None, 48 enable_filters: bool = False 49 ): 50 """ 51 ### Parameters 52 - **index_directory**: The directory containing the index files. This directory must contain the following 53 files: 54 - `{index_prefix}.data` 55 - `{index_prefix}` 56 57 58 It may also include the following optional files: 59 - `{index_prefix}_vectors.bin`: Optional. `diskannpy` builder functions may create this file in the 60 `index_directory` if the index was created from a numpy array 61 - `{index_prefix}_metadata.bin`: Optional. `diskannpy` builder functions create this file to store metadata 62 about the index, such as vector dtype, distance metric, number of vectors and vector dimensionality. 63 If an index is built from the `diskann` cli tools, this file will not exist. 64 - **num_threads**: Number of threads to use when searching this index. (>= 0), 0 = num_threads in system 65 - **initial_search_complexity**: Should be set to the most common `complexity` expected to be used during the 66 life of this `diskannpy.DynamicMemoryIndex` object. The working scratch memory allocated is based off of 67 `initial_search_complexity` * `search_threads`. Note that it may be resized if a `search` or `batch_search` 68 operation requests a space larger than can be accommodated by these values. 69 - **index_prefix**: The prefix of the index files. Defaults to "ann". 70 - **distance_metric**: A `str`, strictly one of {"l2", "mips", "cosine"}. `l2` and `cosine` are supported for all 3 71 vector dtypes, but `mips` is only available for single precision floats. Default is `None`. **This 72 value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it does not exist, 73 you are required to provide it. 74 - **vector_dtype**: The vector dtype this index has been built with. **This value is only used if a 75 `{index_prefix}_metadata.bin` file does not exist.** If it does not exist, you are required to provide it. 76 - **dimensions**: The vector dimensionality of this index. All new vectors inserted must be the same 77 dimensionality. **This value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it 78 does not exist, you are required to provide it. 79 - **enable_filters**: Indexes built with filters can also be used for filtered search. 80 """ 81 index_prefix_path = _valid_index_prefix(index_directory, index_prefix) 82 self._labels_map = {} 83 self._labels_metadata = {} 84 if enable_filters: 85 try: 86 with open(f"{index_prefix_path}_labels_map.txt", "r") as labels_map_if: 87 for line in labels_map_if: 88 (key, val) = line.split("\t") 89 self._labels_map[key] = int(val) 90 with open(f"{index_prefix_path}_label_metadata.json", "r") as labels_metadata_if: 91 self._labels_metadata = json.load(labels_metadata_if) ``` -------------------------------- ### Initialize DiskANN Index Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Internal logic for validating index metadata and initializing the appropriate native index class based on the vector data type. ```python index_prefix_path = _valid_index_prefix(index_directory, index_prefix) vector_dtype, metric, _, _ = _ensure_index_metadata( index_prefix_path, vector_dtype, distance_metric, 1, # it doesn't matter because we don't need it in this context anyway dimensions, ) dap_metric = _valid_metric(metric) _assert_is_nonnegative_uint32(num_threads, "num_threads") _assert_is_nonnegative_uint32(num_nodes_to_cache, "num_nodes_to_cache") self._vector_dtype = vector_dtype if vector_dtype == np.uint8: _index = _native_dap.StaticDiskUInt8Index elif vector_dtype == np.int8: _index = _native_dap.StaticDiskInt8Index else: _index = _native_dap.StaticDiskFloatIndex self._index = _index( distance_metric=dap_metric, index_path_prefix=index_prefix_path, num_threads=num_threads, num_nodes_to_cache=num_nodes_to_cache, cache_mechanism=cache_mechanism, ) ``` -------------------------------- ### build_disk_index Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructs a DiskANN disk-based index, suitable for datasets that exceed available system memory. ```APIDOC ## build_disk_index ### Description Constructs a DiskANN disk index. Disk indices are ideal for very large datasets that are too large to fit in memory. Memory is used to provide precise disk locations for fast retrieval of smaller subsets of the index. ### Parameters - **data** (str | numpy.ndarray) - Required - Path to a vector bin file or a 2D numpy array of vectors. - **distance_metric** (str) - Required - The distance metric to use. Must be one of: 'l2', 'mips', 'cosine'. - **index_directory** (str) - Required - The existing directory path where index files will be saved. - **complexity** (int) - Required - The size of the candidate nearest neighbor list used during index construction. - **graph_degree** (int) - Required - The degree of the graph to be built. - **search_memory_maximum** (float) - Required - Maximum memory to use during search. - **build_memory_maximum** (float) - Required - Maximum memory to use during build. - **num_threads** (int) - Required - Number of threads to use for construction. - **pq_disk_bytes** (int) - Optional - Size of the PQ disk in bytes. - **vector_dtype** (Type) - Optional - The data type of the vectors (float32, int8, uint8). - **index_prefix** (str) - Optional - Prefix for the generated index files (default: 'ann'). ### Returns - **None** - This function does not return a value. ``` -------------------------------- ### Initialize DynamicMemoryIndex Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Initializes the index with specified parameters, performing validation on data types and constraints before instantiating the native index. ```python self._num_vectors = 0 self._removed_num_vectors = 0 dap_metric = _valid_metric(distance_metric) self._dap_metric = dap_metric _assert_dtype(vector_dtype) _assert_is_positive_uint32(dimensions, "dimensions") self._vector_dtype = vector_dtype self._dimensions = dimensions _assert_is_positive_uint32(max_vectors, "max_vectors") _assert_is_positive_uint32(complexity, "complexity") _assert_is_positive_uint32(graph_degree, "graph_degree") _assert( alpha >= 1, "alpha must be >= 1, and realistically should be kept between [1.0, 2.0)", ) _assert_is_nonnegative_uint32(max_occlusion_size, "max_occlusion_size") _assert_is_nonnegative_uint32(num_threads, "num_threads") _assert_is_nonnegative_uint32(filter_complexity, "filter_complexity") _assert_is_nonnegative_uint32(num_frozen_points, "num_frozen_points") _assert_is_nonnegative_uint32( initial_search_complexity, "initial_search_complexity" ) _assert_is_nonnegative_uint32(search_threads, "search_threads") self._max_vectors = max_vectors self._complexity = complexity self._graph_degree = graph_degree if vector_dtype == np.uint8: _index = _native_dap.DynamicMemoryUInt8Index elif vector_dtype == np.int8: _index = _native_dap.DynamicMemoryInt8Index else: _index = _native_dap.DynamicMemoryFloatIndex self._index = _index( distance_metric=dap_metric, dimensions=dimensions, max_vectors=max_vectors, complexity=complexity, graph_degree=graph_degree, saturate_graph=saturate_graph, max_occlusion_size=max_occlusion_size, alpha=alpha, num_threads=num_threads, filter_complexity=filter_complexity, num_frozen_points=num_frozen_points, initial_search_complexity=initial_search_complexity, search_threads=search_threads, concurrent_consolidation=concurrent_consolidation, ) self._points_deleted = False ``` -------------------------------- ### DiskANN Indexing Parameters Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html The following parameters are used to configure the creation of a DiskANN index. ```APIDOC ### Parameters - **data** (str | numpy.ndarray) - Required - Path to a vector bin file or a 2D numpy array of vectors. - **distance_metric** (str) - Required - One of {"l2", "mips", "cosine"}. - **index_directory** (str) - Required - Existing directory path where index files will be saved. - **complexity** (int) - Required - Size of the candidate nearest neighbor list during build. - **graph_degree** (int) - Required - The degree of the graph index. - **num_threads** (int) - Optional - Number of threads to use; 0 uses all available processors. - **alpha** (float) - Optional - Control parameter for graph construction (>=1). - **use_pq_build** (bool) - Optional - Whether to use product quantization during build. Default: True. - **num_pq_bytes** (int) - Optional - Bytes used for PQ compressed data in memory. Default: 0. - **use_opq** (bool) - Optional - Whether to use optimized product quantization. - **vector_dtype** (str) - Required if data is a path - The dtype of the vectors. - **tags** (str | numpy.ndarray) - Optional - Path to .tags file or numpy array of tags. - **filter_labels** (list) - Optional - List of lists containing categories for each vector. - **universal_label** (str) - Optional - Label for vectors to be included in every search. - **filter_complexity** (int) - Optional - Complexity to use when using filters. Default: 0. - **index_prefix** (str) - Optional - Prefix for index files. Default: "ann". ``` -------------------------------- ### Load DynamicMemoryIndex from file Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Use the from_file class method to load a previously saved index from disk. Ensure the index was created with a valid tags file or array and use the same parameters as the original build_memory_index call. ```python 55 @classmethod 56 def from_file( 57 cls, 58 index_directory: str, 59 max_vectors: int, 60 complexity: int, 61 graph_degree: int, 62 saturate_graph: bool = defaults.SATURATE_GRAPH, 63 max_occlusion_size: int = defaults.MAX_OCCLUSION_SIZE, 64 alpha: float = defaults.ALPHA, 65 num_threads: int = defaults.NUM_THREADS, 66 filter_complexity: int = defaults.FILTER_COMPLEXITY, 67 num_frozen_points: int = defaults.NUM_FROZEN_POINTS_DYNAMIC, 68 initial_search_complexity: int = 0, 69 search_threads: int = 0, 70 concurrent_consolidation: bool = True, 71 index_prefix: str = "ann", 72 distance_metric: Optional[DistanceMetric] = None, 73 vector_dtype: Optional[VectorDType] = None, 74 dimensions: Optional[int] = None, 75 ) -> "DynamicMemoryIndex": ``` -------------------------------- ### Import DiskANN modules Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Standard imports for accessing DiskANN functionality and utilities. ```python from . import defaults from ._builder import build_disk_index, build_memory_index from ._common import valid_dtype from ._dynamic_memory_index import DynamicMemoryIndex from ._files import ( Metadata, tags_from_file, tags_to_file, vectors_from_file, vectors_metadata_from_file, vectors_to_file, ) from ._static_disk_index import StaticDiskIndex from ._static_memory_index import StaticMemoryIndex __all__ = [ "build_disk_index", "build_memory_index", "StaticDiskIndex", "StaticMemoryIndex", "DynamicMemoryIndex", "defaults", "DistanceMetric", "VectorDType", "QueryResponse", "QueryResponseBatch", "VectorIdentifier", "VectorIdentifierBatch", "VectorLike", "VectorLikeBatch", "Metadata", "vectors_metadata_from_file", "vectors_to_file", "vectors_from_file", "tags_to_file", "tags_from_file", "valid_dtype", ] ``` -------------------------------- ### Initialize DiskANN Index Memory Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Internal initialization logic for selecting the appropriate memory index type based on vector data type. ```python if vector_dtype == np.uint8: _index = _native_dap.StaticMemoryUInt8Index elif vector_dtype == np.int8: _index = _native_dap.StaticMemoryInt8Index else: _index = _native_dap.StaticMemoryFloatIndex self._index = _index( distance_metric=dap_metric, num_points=num_points, dimensions=dims, index_path=index_prefix_path, num_threads=num_threads, initial_search_complexity=initial_search_complexity, ) ``` -------------------------------- ### build_memory_index Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Constructs a DiskANN memory index. Memory indices are faster than disk indices but are limited by the available system memory. ```APIDOC ## build_memory_index ### Description Constructs a DiskANN memory index. This function supports both static and dynamic memory indices depending on the parameters provided. ### Parameters - **data** (Union[str, numpy.ndarray]) - Required - Path to a vector bin file or a numpy array of supported dtype. - **distance_metric** (Literal['l2', 'mips', 'cosine']) - Required - The distance metric to use. Note: 'mips' is only available for float32. - **index_directory** (str) - Required - Directory path where index files will be saved. - **complexity** (int) - Required - Size of the candidate nearest neighbor list during build. - **graph_degree** (int) - Required - The degree of the graph index. - **num_threads** (int) - Required - Number of threads to use for index creation. - **alpha** (float) - Optional - Default is 1.2000000476837158. - **use_pq_build** (bool) - Optional - Whether to use PQ build. - **num_pq_bytes** (int) - Optional - Number of PQ bytes. - **use_opq** (bool) - Optional - Whether to use OPQ. - **vector_dtype** (Type) - Optional - Required if data is a string path. - **tags** (Union[str, numpy.ndarray]) - Optional - Required if building for DynamicMemoryIndex. - **filter_labels** (list[list[str]]) - Optional - Filter labels for the index. - **universal_label** (str) - Optional - Universal label string. - **filter_complexity** (int) - Optional - Complexity for filtering. - **index_prefix** (str) - Optional - Prefix for index files, defaults to 'ann'. ``` -------------------------------- ### Construct a DiskANN disk index Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html This function validates input parameters and invokes the appropriate native builder based on the vector data type to create a disk-based index. ```python _assert( (isinstance(data, str) and vector_dtype is not None) or isinstance(data, np.ndarray), "vector_dtype is required if data is a str representing a path to the vector bin file", ) dap_metric = _valid_metric(distance_metric) _assert_is_positive_uint32(complexity, "complexity") _assert_is_positive_uint32(graph_degree, "graph_degree") _assert(search_memory_maximum > 0, "search_memory_maximum must be larger than 0") _assert(build_memory_maximum > 0, "build_memory_maximum must be larger than 0") _assert_is_nonnegative_uint32(num_threads, "num_threads") _assert_is_nonnegative_uint32(pq_disk_bytes, "pq_disk_bytes") _assert(index_prefix != "", "index_prefix cannot be an empty string") index_path = Path(index_directory) _assert( index_path.exists() and index_path.is_dir(), "index_directory must both exist and be a directory", ) vector_bin_path, vector_dtype_actual = _valid_path_and_dtype( data, vector_dtype, index_directory, index_prefix ) _assert(dap_metric != _native_dap.COSINE, "Cosine is currently not supported in StaticDiskIndex") if dap_metric == _native_dap.INNER_PRODUCT: _assert( vector_dtype_actual == np.float32, "Integral vector dtypes (np.uint8, np.int8) are not supported with distance metric mips" ) num_points, dimensions = vectors_metadata_from_file(vector_bin_path) if vector_dtype_actual == np.uint8: _builder = _native_dap.build_disk_uint8_index elif vector_dtype_actual == np.int8: _builder = _native_dap.build_disk_int8_index else: _builder = _native_dap.build_disk_float_index index_prefix_path = os.path.join(index_directory, index_prefix) _builder( distance_metric=dap_metric, data_file_path=vector_bin_path, index_prefix_path=index_prefix_path, complexity=complexity, graph_degree=graph_degree, final_index_ram_limit=search_memory_maximum, indexing_ram_budget=build_memory_maximum, num_threads=num_threads, pq_disk_bytes=pq_disk_bytes, ) _write_index_metadata( index_prefix_path, vector_dtype_actual, dap_metric, num_points, dimensions ) ``` -------------------------------- ### Write tags to a binary file Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Writes a 1D NumPy array of uint32 tags to a DiskANN binary file. 2D arrays with one column are supported but will trigger a copy operation. ```python 75def tags_to_file(tags_file: str, tags: VectorIdentifierBatch) -> None: 76 """ 77 Write tags to a DiskANN binary tag file. 78 79 ### Parameters 80 - **tags_file**: The path to the tag file to write the tags to. 81 - **tags**: A 1d array of dtype `numpy.uint32` containing the tags to write. If you have a 2d array of tags with 82 one column, you can pass it here and it will be reshaped and copied to a new array. It is more efficient for you 83 to reshape on your own without copying it first, as it should be a constant time operation vs. linear time 84 85 """ 86 _assert(np.can_cast(tags.dtype, np.uint32), "valid tags must be uint32") 87 _assert( 88 len(tags.shape) == 1 or tags.shape[1] == 1, 89 "tags must be 1d or 2d with 1 column", 90 ) 91 if len(tags.shape) == 2: 92 warnings.warn( 93 "Tags in 2d with one column will be reshaped and copied to a new array. " 94 "It is more efficient for you to reshape without copying first." 95 ) 96 tags = tags.reshape(tags.shape[0], copy=True) 97 with open(tags_file, "wb") as fh: 98 _write_bin(tags.astype(np.uint32), fh) ``` -------------------------------- ### Construct a DiskANN disk index Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Function to build a disk-based index for large datasets. Requires an existing directory and specific memory/complexity parameters. ```python def build_disk_index( data: Union[str, VectorLikeBatch], distance_metric: DistanceMetric, index_directory: str, complexity: int, graph_degree: int, search_memory_maximum: float, build_memory_maximum: float, num_threads: int, pq_disk_bytes: int = defaults.PQ_DISK_BYTES, vector_dtype: Optional[VectorDType] = None, index_prefix: str = "ann", ) -> None: ``` -------------------------------- ### tags_to_file(tags_file: str, tags: numpy.ndarray) Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html Writes tags to a DiskANN binary tag file. ```APIDOC ## tags_to_file(tags_file: str, tags: numpy.ndarray) ### Description Write tags to a DiskANN binary tag file. ### Parameters - **tags_file** (str) - Required - The path to the tag file to write the tags to. - **tags** (numpy.ndarray) - Required - A 1d array of dtype `numpy.uint32` containing the tags to write. If you have a 2d array of tags with one column, it will be reshaped. ```