DiskANN Python

https://microsoft.github.io/DiskANN/docs/python/latest/
Admin
DiskANN Python is a Python wrapper for the DiskANN library, providing functionalities for building...
Tokens:35,799
Snippets:215
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### Load DiskANN index from file

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Use the from_file classmethod to initialize an index from a saved path. Ensure the index was created with valid tags and matches the original build parameters.

```python
145            f"The file {tags_file} does not exist in {index_directory}",
146        )
147        vector_dtype, dap_metric, num_vectors, dimensions = _ensure_index_metadata(
148            index_prefix_path, vector_dtype, distance_metric, max_vectors, dimensions, warn_size_exceeded=True
149        )
150
151        index = cls(
152            distance_metric=dap_metric,  # type: ignore
153            vector_dtype=vector_dtype,
154            dimensions=dimensions,
155            max_vectors=max_vectors,
156            complexity=complexity,
157            graph_degree=graph_degree,
158            saturate_graph=saturate_graph,
159            max_occlusion_size=max_occlusion_size,
160            alpha=alpha,
161            num_threads=num_threads,
162            filter_complexity=filter_complexity,
163            num_frozen_points=num_frozen_points,
164            initial_search_complexity=initial_search_complexity,
165            search_threads=search_threads,
166            concurrent_consolidation=concurrent_consolidation,
167        )
168        index._index.load(index_prefix_path)
169        index._num_vectors = num_vectors  # current number of vectors loaded
170        return index
```

--------------------------------

### StaticDiskIndex.__init__

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Initializes a new instance of the StaticDiskIndex class, which loads a disk-backed index from the specified directory.

```APIDOC
## StaticDiskIndex(index_directory, num_threads, num_nodes_to_cache, cache_mechanism=1, distance_metric=None, vector_dtype=None, dimensions=None, index_prefix='ann')

### Description
Initializes a disk-backed index for searching. The index directory must contain the required index files, or metadata must be provided.

### Parameters
- **index_directory** (str) - Required - The directory containing the index files.
- **num_threads** (int) - Required - Number of threads to use when searching. 0 uses system default.
- **num_nodes_to_cache** (int) - Required - Number of nodes to cache in memory.
- **cache_mechanism** (int) - Optional - 1 to initialize cache with sample data, 2 to ready cache without initialization, other values disable caching.
- **distance_metric** (str) - Optional - One of {'l2', 'mips', 'cosine'}. Required if metadata file is missing.
- **vector_dtype** (Type) - Optional - The vector data type. Required if metadata file is missing.
- **dimensions** (int) - Optional - Vector dimensionality. Required if metadata file is missing.
- **index_prefix** (str) - Optional - The prefix of the index files. Defaults to 'ann'.
```

--------------------------------

### StaticDiskIndex.__init__

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Initializes a new StaticDiskIndex, which is a disk-backed, immutable index. The index directory must contain the required index files.

```APIDOC
## StaticDiskIndex.__init__

### Description
Initializes a new instance of the StaticDiskIndex class. This index is disk-backed and not mutable.

### Parameters
- **index_directory** (str) - Required - The directory containing the index files.
- **num_threads** (int) - Required - Number of threads to use.
- **num_nodes_to_cache** (int) - Required - Number of nodes to cache in memory.
- **cache_mechanism** (int) - Optional - The caching mechanism to use. Default is 1.
- **distance_metric** (DistanceMetric) - Optional - The distance metric used for the index.
- **vector_dtype** (VectorDType) - Optional - The data type of the vectors.
- **dimensions** (int) - Optional - The dimensionality of the vectors.
- **index_prefix** (str) - Optional - The prefix of the index files. Defaults to "ann".
```

--------------------------------

### Initialize StaticDiskIndex

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor for the immutable disk-backed index, requiring a directory containing pre-built index files.

```python
 34class StaticDiskIndex:
 35    """
 36    A StaticDiskIndex is a disk-backed index that is not mutable.
 37    """
 38
 39    def __init__(
 40        self,
 41        index_directory: str,
 42        num_threads: int,
 43        num_nodes_to_cache: int,
 44        cache_mechanism: int = 1,
 45        distance_metric: Optional[DistanceMetric] = None,
 46        vector_dtype: Optional[VectorDType] = None,
 47        dimensions: Optional[int] = None,
 48        index_prefix: str = "ann",
 49    ):
 50        """
 51        ### Parameters
 52        - **index_directory**: The directory containing the index files. This directory must contain the following
 53            files:
 54            - `{index_prefix}_sample_data.bin`
 55            - `{index_prefix}_mem.index.data`
 56            - `{index_prefix}_pq_compressed.bin`
 57            - `{index_prefix}_pq_pivots.bin`
 58            - `{index_prefix}_sample_ids.bin`
 59            - `{index_prefix}_disk.index`
 60
 61          It may also include the following optional files:
```

--------------------------------

### Initialize DiskANN Index

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Handles index metadata loading, filter configuration, and native index object instantiation.

```python
index_prefix_path = _valid_index_prefix(index_directory, index_prefix)
self._labels_map = {}
self._labels_metadata = {}
if enable_filters:
    try:
        with open(f"{index_prefix_path}_labels_map.txt", "r") as labels_map_if:
            for line in labels_map_if:
                (key, val) = line.split("\t")
                self._labels_map[key] = int(val)
        with open(f"{index_prefix_path}_label_metadata.json", "r") as labels_metadata_if:
            self._labels_metadata = json.load(labels_metadata_if)
    except: # noqa: E722
        # exceptions are basically presumed to be either file not found or file not formatted correctly
        raise RuntimeException("Filter labels file was unable to be processed.")
vector_dtype, metric, num_points, dims = _ensure_index_metadata(
    index_prefix_path,
    vector_dtype,
    distance_metric,
    1,  # it doesn't matter because we don't need it in this context anyway
    dimensions,
)
dap_metric = _valid_metric(metric)

_assert_is_nonnegative_uint32(num_threads, "num_threads")
_assert_is_positive_uint32(
    initial_search_complexity, "initial_search_complexity"
)

self._vector_dtype = vector_dtype
self._dimensions = dims

if vector_dtype == np.uint8:
    _index = _native_dap.StaticMemoryUInt8Index
elif vector_dtype == np.int8:
    _index = _native_dap.StaticMemoryInt8Index
else:
    _index = _native_dap.StaticMemoryFloatIndex

self._index = _index(
    distance_metric=dap_metric,
    num_points=num_points,
    dimensions=dims,
    index_path=index_prefix_path,
    num_threads=num_threads,
    initial_search_complexity=initial_search_complexity,
)
```

--------------------------------

### Initialize DynamicMemoryIndex

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor logic for setting up a new DynamicMemoryIndex, including parameter validation and native index instantiation.

```python
self._num_vectors = 0
self._removed_num_vectors = 0
dap_metric = _valid_metric(distance_metric)
self._dap_metric = dap_metric
_assert_dtype(vector_dtype)
_assert_is_positive_uint32(dimensions, "dimensions")

self._vector_dtype = vector_dtype
self._dimensions = dimensions

_assert_is_positive_uint32(max_vectors, "max_vectors")
_assert_is_positive_uint32(complexity, "complexity")
_assert_is_positive_uint32(graph_degree, "graph_degree")
_assert(
    alpha >= 1,
    "alpha must be >= 1, and realistically should be kept between [1.0, 2.0)",
)
_assert_is_nonnegative_uint32(max_occlusion_size, "max_occlusion_size")
_assert_is_nonnegative_uint32(num_threads, "num_threads")
_assert_is_nonnegative_uint32(filter_complexity, "filter_complexity")
_assert_is_nonnegative_uint32(num_frozen_points, "num_frozen_points")
_assert_is_nonnegative_uint32(
    initial_search_complexity, "initial_search_complexity"
)
_assert_is_nonnegative_uint32(search_threads, "search_threads")

self._max_vectors = max_vectors
self._complexity = complexity
self._graph_degree = graph_degree

if vector_dtype == np.uint8:
    _index = _native_dap.DynamicMemoryUInt8Index
elif vector_dtype == np.int8:
    _index = _native_dap.DynamicMemoryInt8Index
else:
    _index = _native_dap.DynamicMemoryFloatIndex

self._index = _index(
    distance_metric=dap_metric,
    dimensions=dimensions,
    max_vectors=max_vectors,
    complexity=complexity,
    graph_degree=graph_degree,
    saturate_graph=saturate_graph,
    max_occlusion_size=max_occlusion_size,
    alpha=alpha,
    num_threads=num_threads,
    filter_complexity=filter_complexity,
    num_frozen_points=num_frozen_points,
    initial_search_complexity=initial_search_complexity,
    search_threads=search_threads,
    concurrent_consolidation=concurrent_consolidation,
)
self._points_deleted = False
```

--------------------------------

### Initialize StaticDiskIndex

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor for the StaticDiskIndex class, which validates index metadata and initializes the native disk-backed index.

```python
 39    def __init__(
 40        self,
 41        index_directory: str,
 42        num_threads: int,
 43        num_nodes_to_cache: int,
 44        cache_mechanism: int = 1,
 45        distance_metric: Optional[DistanceMetric] = None,
 46        vector_dtype: Optional[VectorDType] = None,
 47        dimensions: Optional[int] = None,
 48        index_prefix: str = "ann",
 49    ):
 50        """
 51        ### Parameters
 52        - **index_directory**: The directory containing the index files. This directory must contain the following
 53            files:
 54            - `{index_prefix}_sample_data.bin`
 55            - `{index_prefix}_mem.index.data`
 56            - `{index_prefix}_pq_compressed.bin`
 57            - `{index_prefix}_pq_pivots.bin`
 58            - `{index_prefix}_sample_ids.bin`
 59            - `{index_prefix}_disk.index`
 60
 61          It may also include the following optional files:
 62            - `{index_prefix}_vectors.bin`: Optional. `diskannpy` builder functions may create this file in the
 63              `index_directory` if the index was created from a numpy array
 64            - `{index_prefix}_metadata.bin`: Optional. `diskannpy` builder functions create this file to store metadata
 65            about the index, such as vector dtype, distance metric, number of vectors and vector dimensionality.
 66            If an index is built from the `diskann` cli tools, this file will not exist.
 67        - **num_threads**: Number of threads to use when searching this index. (>= 0), 0 = num_threads in system
 68        - **num_nodes_to_cache**: Number of nodes to cache in memory (> -1)
 69        - **cache_mechanism**: 1 -> use the generated sample_data.bin file for
 70            the index to initialize a set of cached nodes, up to `num_nodes_to_cache`, 2 -> ready the cache for up to
 71            `num_nodes_to_cache`, but do not initialize it with any nodes. Any other value disables node caching.
 72        - **distance_metric**: A `str`, strictly one of {"l2", "mips", "cosine"}. `l2` and `cosine` are supported for all 3
 73          vector dtypes, but `mips` is only available for single precision floats. Default is `None`. **This
 74          value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it does not exist,
 75          you are required to provide it.
 76        - **vector_dtype**: The vector dtype this index has been built with. **This value is only used if a
 77          `{index_prefix}_metadata.bin` file does not exist.** If it does not exist, you are required to provide it.
 78        - **dimensions**: The vector dimensionality of this index. All new vectors inserted must be the same
 79          dimensionality. **This value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it
 80          does not exist, you are required to provide it.
 81        - **index_prefix**: The prefix of the index files. Defaults to "ann".
 82        """
 83        index_prefix_path = _valid_index_prefix(index_directory, index_prefix)
 84        vector_dtype, metric, _, _ = _ensure_index_metadata(
 85            index_prefix_path,
 86            vector_dtype,
 87            distance_metric,
 88            1,  # it doesn't matter because we don't need it in this context anyway
 89            dimensions,
 90        )
 91        dap_metric = _valid_metric(metric)
 92
 93        _assert_is_nonnegative_uint32(num_threads, "num_threads")
 94        _assert_is_nonnegative_uint32(num_nodes_to_cache, "num_nodes_to_cache")
 95
 96        self._vector_dtype = vector_dtype
 97        if vector_dtype == np.uint8:
 98            _index = _native_dap.StaticDiskUInt8Index
 99        elif vector_dtype == np.int8:
100            _index = _native_dap.StaticDiskInt8Index
101        else:
102            _index = _native_dap.StaticDiskFloatIndex
103        self._index = _index(
104            distance_metric=dap_metric,
105            index_path_prefix=index_prefix_path,
106            num_threads=num_threads,
107            num_nodes_to_cache=num_nodes_to_cache,
108            cache_mechanism=cache_mechanism,
109        )
```

--------------------------------

### from_file(index_prefix_path, ...)

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Loads a previously saved DiskANN index from disk. It is recommended to use the same parameters as those used during the index creation process.

```APIDOC
## from_file

### Description
Loads a previously saved index from disk. The index must have been created with a valid tags file or tags array.

### Parameters
- **index_prefix_path** (str) - Required - The directory path where the index files are stored.
- **vector_dtype** (dtype) - Optional - Data type of the vectors.
- **distance_metric** (str) - Optional - The distance metric used for the index.
- **max_vectors** (int) - Optional - Maximum number of vectors the index can hold.
- **dimensions** (int) - Optional - Number of dimensions for the vectors.
- **complexity** (int) - Optional - Indexing complexity parameter.
- **graph_degree** (int) - Optional - The degree of the graph.
- **saturate_graph** (bool) - Optional - Whether to saturate the graph.
- **max_occlusion_size** (int) - Optional - Maximum occlusion size.
- **alpha** (float) - Optional - Alpha parameter for graph construction.
- **num_threads** (int) - Optional - Number of threads for indexing.
- **filter_complexity** (int) - Optional - Complexity for filtering.
- **num_frozen_points** (int) - Optional - Number of frozen points.
- **initial_search_complexity** (int) - Optional - Initial search complexity.
- **search_threads** (int) - Optional - Number of threads for searching.
- **concurrent_consolidation** (bool) - Optional - Whether to use concurrent consolidation.
```

--------------------------------

### Load DynamicMemoryIndex from file

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Use this method to initialize a DynamicMemoryIndex from existing index files on disk. Ensure the index_directory contains the required .data, .tags, and index files.

```python
    def from_file(
        cls,
        index_directory: str,
        max_vectors: int,
        complexity: int,
        graph_degree: int,
        saturate_graph: bool = defaults.SATURATE_GRAPH,
        max_occlusion_size: int = defaults.MAX_OCCLUSION_SIZE,
        alpha: float = defaults.ALPHA,
        num_threads: int = defaults.NUM_THREADS,
        filter_complexity: int = defaults.FILTER_COMPLEXITY,
        num_frozen_points: int = defaults.NUM_FROZEN_POINTS_DYNAMIC,
        initial_search_complexity: int = 0,
        search_threads: int = 0,
        concurrent_consolidation: bool = True,
        index_prefix: str = "ann",
        distance_metric: Optional[DistanceMetric] = None,
        vector_dtype: Optional[VectorDType] = None,
        dimensions: Optional[int] = None,
    ) -> "DynamicMemoryIndex":
```

--------------------------------

### StaticMemoryIndex.__init__

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Initializes a new instance of the StaticMemoryIndex class by loading an existing index from the specified directory.

```APIDOC
## StaticMemoryIndex.__init__

### Description
Initializes an immutable in-memory DiskANN index from a specified directory. The index must contain the required data files, and optional metadata can be provided if the metadata file is missing.

### Parameters
- **index_directory** (str) - Required - The directory containing the index files.
- **num_threads** (int) - Required - Number of threads to use when searching this index (0 = system default).
- **initial_search_complexity** (int) - Required - The initial complexity setting used to allocate scratch memory.
- **index_prefix** (str) - Optional - The prefix of the index files (defaults to "ann").
- **distance_metric** (DistanceMetric) - Optional - One of {"l2", "mips", "cosine"}. Required if metadata file is missing.
- **vector_dtype** (VectorDType) - Optional - The vector dtype. Required if metadata file is missing.
- **dimensions** (int) - Optional - The vector dimensionality. Required if metadata file is missing.
- **enable_filters** (bool) - Optional - Whether to enable filtered search.
```

--------------------------------

### Initialize StaticMemoryIndex

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor for the immutable in-memory DiskANN index. Requires a directory containing index files and configuration for search threads and complexity.

```python
    def __init__(
        self,
        index_directory: str,
        num_threads: int,
        initial_search_complexity: int,
        index_prefix: str = "ann",
        distance_metric: Optional[DistanceMetric] = None,
        vector_dtype: Optional[VectorDType] = None,
        dimensions: Optional[int] = None,
        enable_filters: bool = False
    ):
```

--------------------------------

### Initialize DynamicMemoryIndex

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor for creating a new, empty DynamicMemoryIndex. Use this for building an index from scratch rather than loading an existing one.

```python
def __init__(
    self,
    distance_metric: DistanceMetric,
    vector_dtype: VectorDType,
    dimensions: int,
    max_vectors: int,
    complexity: int,
    graph_degree: int,
    saturate_graph: bool = defaults.SATURATE_GRAPH,
    max_occlusion_size: int = defaults.MAX_OCCLUSION_SIZE,
    alpha: float = defaults.ALPHA,
    num_threads: int = defaults.NUM_THREADS,
    filter_complexity: int = defaults.FILTER_COMPLEXITY,
    num_frozen_points: int = defaults.NUM_FROZEN_POINTS_DYNAMIC,
    initial_search_complexity: int = 0,
    search_threads: int = 0,
    concurrent_consolidation: bool = True,
):
```

--------------------------------

### diskannpy.DynamicMemoryIndex.__init__

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor for creating a new, empty DynamicMemoryIndex.

```APIDOC
## diskannpy.DynamicMemoryIndex.__init__

### Description
Initializes a new instance of a mutable DiskANN memory index.

### Parameters
- **saturate_graph** (bool) - Optional - If True, the adjacency list of each node will be saturated with neighbors to have exactly graph_degree neighbors.
- **max_occlusion_size** (int) - Optional - The maximum number of points that can be considered by occlude_list function.
- **alpha** (float) - Optional - Controls the nature and number of points added to the graph (>=1).
- **num_threads** (int) - Optional - Number of threads to use when creating this index.
- **filter_complexity** (int) - Optional - Complexity to use when using filters.
- **num_frozen_points** (int) - Optional - Number of points to freeze.
- **initial_search_complexity** (int) - Optional - Expected complexity for search operations.
- **search_threads** (int) - Optional - Expected number of threads for search operations.
- **concurrent_consolidation** (bool) - Optional - Whether consolidation can run alongside inserts and deletes.
```

--------------------------------

### StaticMemoryIndex.__init__

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Initializes a new StaticMemoryIndex object, which represents an immutable in-memory DiskANN index.

```APIDOC
## StaticMemoryIndex(index_directory, num_threads, initial_search_complexity, ...)

### Description
Initializes an immutable in-memory DiskANN index from a specified directory.

### Parameters
- **index_directory** (str) - Required - The directory containing the index files.
- **num_threads** (int) - Required - Number of threads to use when searching this index.
- **initial_search_complexity** (int) - Required - The expected search complexity for memory allocation.
- **index_prefix** (str) - Optional - The prefix of the index files. Defaults to "ann".
- **distance_metric** (str) - Optional - One of {"l2", "mips", "cosine"}. Used if metadata file is missing.
- **vector_dtype** (Type) - Optional - The vector dtype. Used if metadata file is missing.
- **dimensions** (int) - Optional - The vector dimensionality of the index.
- **enable_filters** (bool) - Optional - Whether to enable filtering.
```

--------------------------------

### DynamicMemoryIndex.__init__

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Initializes a new, empty DynamicMemoryIndex for storing and searching vectors.

```APIDOC
## DynamicMemoryIndex(distance_metric, vector_dtype, dimensions, max_vectors, complexity, graph_degree, ...)

### Description
Creates a new, empty mutable DiskANN memory index.

### Parameters
- **distance_metric** (str) - Required - One of {'l2', 'mips', 'cosine'}.
- **vector_dtype** (type) - Required - One of {np.float32, np.int8, np.uint8}.
- **dimensions** (int) - Required - Vector dimensionality.
- **max_vectors** (int) - Required - Capacity of the data store.
- **graph_degree** (int) - Required - Maximum degree allowed for a node in the graph.
```

--------------------------------

### Load DynamicMemoryIndex from disk

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

This method initializes a DynamicMemoryIndex by loading an existing index from the specified directory. It validates the presence of required metadata and tag files before loading.

```python
index_prefix_path = _valid_index_prefix(index_directory, index_prefix)

# do tags exist?
tags_file = index_prefix_path + ".tags"
_assert(
    Path(tags_file).exists(),
    f"The file {tags_file} does not exist in {index_directory}",
)
vector_dtype, dap_metric, num_vectors, dimensions = _ensure_index_metadata(
    index_prefix_path, vector_dtype, distance_metric, max_vectors, dimensions, warn_size_exceeded=True
)

index = cls(
    distance_metric=dap_metric,  # type: ignore
    vector_dtype=vector_dtype,
    dimensions=dimensions,
    max_vectors=max_vectors,
    complexity=complexity,
    graph_degree=graph_degree,
    saturate_graph=saturate_graph,
    max_occlusion_size=max_occlusion_size,
    alpha=alpha,
    num_threads=num_threads,
    filter_complexity=filter_complexity,
    num_frozen_points=num_frozen_points,
    initial_search_complexity=initial_search_complexity,
    search_threads=search_threads,
    concurrent_consolidation=concurrent_consolidation,
)
index._index.load(index_prefix_path)
index._num_vectors = num_vectors  # current number of vectors loaded
return index
```

--------------------------------

### diskannpy.DynamicMemoryIndex.__init__

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor to create a new, empty mutable DiskANN memory index.

```APIDOC
## DynamicMemoryIndex.__init__

### Description
Initializes a new, empty mutable DiskANN memory index.

### Parameters
- **distance_metric** (str) - Required - One of {"l2", "mips", "cosine"}.
- **vector_dtype** (type) - Required - One of {np.float32, np.int8, np.uint8}.
- **dimensions** (int) - Required - Vector dimensionality.
- **max_vectors** (int) - Required - Capacity of the data store.
- **graph_degree** (int) - Required - Maximum degree allowed for a node in the graph.
- **saturate_graph** (bool) - Optional - Whether to saturate the adjacency list of each node.
```

--------------------------------

### Initialize StaticMemoryIndex

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructor for the StaticMemoryIndex class. Requires an index directory and configuration parameters for search performance and index metadata.

```python
 34class StaticMemoryIndex:
 35    """
 36    A StaticMemoryIndex is an immutable in-memory DiskANN index.
 37    """
 38
 39    def __init__(
 40        self,
 41        index_directory: str,
 42        num_threads: int,
 43        initial_search_complexity: int,
 44        index_prefix: str = "ann",
 45        distance_metric: Optional[DistanceMetric] = None,
 46        vector_dtype: Optional[VectorDType] = None,
 47        dimensions: Optional[int] = None,
 48        enable_filters: bool = False
 49    ):
 50        """
 51        ### Parameters
 52        - **index_directory**: The directory containing the index files. This directory must contain the following
 53          files:
 54            - `{index_prefix}.data`
 55            - `{index_prefix}`
 56
 57
 58          It may also include the following optional files:
 59            - `{index_prefix}_vectors.bin`: Optional. `diskannpy` builder functions may create this file in the
 60              `index_directory` if the index was created from a numpy array
 61            - `{index_prefix}_metadata.bin`: Optional. `diskannpy` builder functions create this file to store metadata
 62            about the index, such as vector dtype, distance metric, number of vectors and vector dimensionality.
 63            If an index is built from the `diskann` cli tools, this file will not exist.
 64        - **num_threads**: Number of threads to use when searching this index. (>= 0), 0 = num_threads in system
 65        - **initial_search_complexity**: Should be set to the most common `complexity` expected to be used during the
 66          life of this `diskannpy.DynamicMemoryIndex` object. The working scratch memory allocated is based off of
 67          `initial_search_complexity` * `search_threads`. Note that it may be resized if a `search` or `batch_search`
 68          operation requests a space larger than can be accommodated by these values.
 69        - **index_prefix**: The prefix of the index files. Defaults to "ann".
 70        - **distance_metric**: A `str`, strictly one of {"l2", "mips", "cosine"}. `l2` and `cosine` are supported for all 3
 71          vector dtypes, but `mips` is only available for single precision floats. Default is `None`. **This
 72          value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it does not exist,
 73          you are required to provide it.
 74        - **vector_dtype**: The vector dtype this index has been built with. **This value is only used if a
 75          `{index_prefix}_metadata.bin` file does not exist.** If it does not exist, you are required to provide it.
 76        - **dimensions**: The vector dimensionality of this index. All new vectors inserted must be the same
 77          dimensionality. **This value is only used if a `{index_prefix}_metadata.bin` file does not exist.** If it
 78          does not exist, you are required to provide it.
 79        - **enable_filters**: Indexes built with filters can also be used for filtered search.
 80        """
 81        index_prefix_path = _valid_index_prefix(index_directory, index_prefix)
 82        self._labels_map = {}
 83        self._labels_metadata = {}
 84        if enable_filters:
 85            try:
 86                with open(f"{index_prefix_path}_labels_map.txt", "r") as labels_map_if:
 87                    for line in labels_map_if:
 88                        (key, val) = line.split("\t")
 89                        self._labels_map[key] = int(val)
 90                with open(f"{index_prefix_path}_label_metadata.json", "r") as labels_metadata_if:
 91                    self._labels_metadata = json.load(labels_metadata_if)
```

--------------------------------

### Initialize DiskANN Index

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Internal logic for validating index metadata and initializing the appropriate native index class based on the vector data type.

```python
index_prefix_path = _valid_index_prefix(index_directory, index_prefix)
        vector_dtype, metric, _, _ = _ensure_index_metadata(
            index_prefix_path,
            vector_dtype,
            distance_metric,
            1,  # it doesn't matter because we don't need it in this context anyway
            dimensions,
        )
        dap_metric = _valid_metric(metric)

        _assert_is_nonnegative_uint32(num_threads, "num_threads")
        _assert_is_nonnegative_uint32(num_nodes_to_cache, "num_nodes_to_cache")

        self._vector_dtype = vector_dtype
        if vector_dtype == np.uint8:
            _index = _native_dap.StaticDiskUInt8Index
        elif vector_dtype == np.int8:
            _index = _native_dap.StaticDiskInt8Index
        else:
            _index = _native_dap.StaticDiskFloatIndex
        self._index = _index(
            distance_metric=dap_metric,
            index_path_prefix=index_prefix_path,
            num_threads=num_threads,
            num_nodes_to_cache=num_nodes_to_cache,
            cache_mechanism=cache_mechanism,
        )
```

--------------------------------

### build_disk_index

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructs a DiskANN disk-based index, suitable for datasets that exceed available system memory.

```APIDOC
## build_disk_index

### Description
Constructs a DiskANN disk index. Disk indices are ideal for very large datasets that are too large to fit in memory. Memory is used to provide precise disk locations for fast retrieval of smaller subsets of the index.

### Parameters
- **data** (str | numpy.ndarray) - Required - Path to a vector bin file or a 2D numpy array of vectors.
- **distance_metric** (str) - Required - The distance metric to use. Must be one of: 'l2', 'mips', 'cosine'.
- **index_directory** (str) - Required - The existing directory path where index files will be saved.
- **complexity** (int) - Required - The size of the candidate nearest neighbor list used during index construction.
- **graph_degree** (int) - Required - The degree of the graph to be built.
- **search_memory_maximum** (float) - Required - Maximum memory to use during search.
- **build_memory_maximum** (float) - Required - Maximum memory to use during build.
- **num_threads** (int) - Required - Number of threads to use for construction.
- **pq_disk_bytes** (int) - Optional - Size of the PQ disk in bytes.
- **vector_dtype** (Type) - Optional - The data type of the vectors (float32, int8, uint8).
- **index_prefix** (str) - Optional - Prefix for the generated index files (default: 'ann').

### Returns
- **None** - This function does not return a value.
```

--------------------------------

### Initialize DynamicMemoryIndex

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Initializes the index with specified parameters, performing validation on data types and constraints before instantiating the native index.

```python
        self._num_vectors = 0
        self._removed_num_vectors = 0
        dap_metric = _valid_metric(distance_metric)
        self._dap_metric = dap_metric
        _assert_dtype(vector_dtype)
        _assert_is_positive_uint32(dimensions, "dimensions")

        self._vector_dtype = vector_dtype
        self._dimensions = dimensions

        _assert_is_positive_uint32(max_vectors, "max_vectors")
        _assert_is_positive_uint32(complexity, "complexity")
        _assert_is_positive_uint32(graph_degree, "graph_degree")
        _assert(
            alpha >= 1,
            "alpha must be >= 1, and realistically should be kept between [1.0, 2.0)",
        )
        _assert_is_nonnegative_uint32(max_occlusion_size, "max_occlusion_size")
        _assert_is_nonnegative_uint32(num_threads, "num_threads")
        _assert_is_nonnegative_uint32(filter_complexity, "filter_complexity")
        _assert_is_nonnegative_uint32(num_frozen_points, "num_frozen_points")
        _assert_is_nonnegative_uint32(
            initial_search_complexity, "initial_search_complexity"
        )
        _assert_is_nonnegative_uint32(search_threads, "search_threads")

        self._max_vectors = max_vectors
        self._complexity = complexity
        self._graph_degree = graph_degree

        if vector_dtype == np.uint8:
            _index = _native_dap.DynamicMemoryUInt8Index
        elif vector_dtype == np.int8:
            _index = _native_dap.DynamicMemoryInt8Index
        else:
            _index = _native_dap.DynamicMemoryFloatIndex

        self._index = _index(
            distance_metric=dap_metric,
            dimensions=dimensions,
            max_vectors=max_vectors,
            complexity=complexity,
            graph_degree=graph_degree,
            saturate_graph=saturate_graph,
            max_occlusion_size=max_occlusion_size,
            alpha=alpha,
            num_threads=num_threads,
            filter_complexity=filter_complexity,
            num_frozen_points=num_frozen_points,
            initial_search_complexity=initial_search_complexity,
            search_threads=search_threads,
            concurrent_consolidation=concurrent_consolidation,
        )
        self._points_deleted = False
```

--------------------------------

### DiskANN Indexing Parameters

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

The following parameters are used to configure the creation of a DiskANN index.

```APIDOC
### Parameters

- **data** (str | numpy.ndarray) - Required - Path to a vector bin file or a 2D numpy array of vectors.
- **distance_metric** (str) - Required - One of {"l2", "mips", "cosine"}.
- **index_directory** (str) - Required - Existing directory path where index files will be saved.
- **complexity** (int) - Required - Size of the candidate nearest neighbor list during build.
- **graph_degree** (int) - Required - The degree of the graph index.
- **num_threads** (int) - Optional - Number of threads to use; 0 uses all available processors.
- **alpha** (float) - Optional - Control parameter for graph construction (>=1).
- **use_pq_build** (bool) - Optional - Whether to use product quantization during build. Default: True.
- **num_pq_bytes** (int) - Optional - Bytes used for PQ compressed data in memory. Default: 0.
- **use_opq** (bool) - Optional - Whether to use optimized product quantization.
- **vector_dtype** (str) - Required if data is a path - The dtype of the vectors.
- **tags** (str | numpy.ndarray) - Optional - Path to .tags file or numpy array of tags.
- **filter_labels** (list) - Optional - List of lists containing categories for each vector.
- **universal_label** (str) - Optional - Label for vectors to be included in every search.
- **filter_complexity** (int) - Optional - Complexity to use when using filters. Default: 0.
- **index_prefix** (str) - Optional - Prefix for index files. Default: "ann".
```

--------------------------------

### Load DynamicMemoryIndex from file

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Use the from_file class method to load a previously saved index from disk. Ensure the index was created with a valid tags file or array and use the same parameters as the original build_memory_index call.

```python
 55    @classmethod
 56    def from_file(
 57        cls,
 58        index_directory: str,
 59        max_vectors: int,
 60        complexity: int,
 61        graph_degree: int,
 62        saturate_graph: bool = defaults.SATURATE_GRAPH,
 63        max_occlusion_size: int = defaults.MAX_OCCLUSION_SIZE,
 64        alpha: float = defaults.ALPHA,
 65        num_threads: int = defaults.NUM_THREADS,
 66        filter_complexity: int = defaults.FILTER_COMPLEXITY,
 67        num_frozen_points: int = defaults.NUM_FROZEN_POINTS_DYNAMIC,
 68        initial_search_complexity: int = 0,
 69        search_threads: int = 0,
 70        concurrent_consolidation: bool = True,
 71        index_prefix: str = "ann",
 72        distance_metric: Optional[DistanceMetric] = None,
 73        vector_dtype: Optional[VectorDType] = None,
 74        dimensions: Optional[int] = None,
 75    ) -> "DynamicMemoryIndex":
```

--------------------------------

### Import DiskANN modules

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Standard imports for accessing DiskANN functionality and utilities.

```python
from . import defaults
from ._builder import build_disk_index, build_memory_index
from ._common import valid_dtype
from ._dynamic_memory_index import DynamicMemoryIndex
from ._files import (
    Metadata,
    tags_from_file,
    tags_to_file,
    vectors_from_file,
    vectors_metadata_from_file,
    vectors_to_file,
)
from ._static_disk_index import StaticDiskIndex
from ._static_memory_index import StaticMemoryIndex

__all__ = [
    "build_disk_index",
    "build_memory_index",
    "StaticDiskIndex",
    "StaticMemoryIndex",
    "DynamicMemoryIndex",
    "defaults",
    "DistanceMetric",
    "VectorDType",
    "QueryResponse",
    "QueryResponseBatch",
    "VectorIdentifier",
    "VectorIdentifierBatch",
    "VectorLike",
    "VectorLikeBatch",
    "Metadata",
    "vectors_metadata_from_file",
    "vectors_to_file",
    "vectors_from_file",
    "tags_to_file",
    "tags_from_file",
    "valid_dtype",
]
```

--------------------------------

### Initialize DiskANN Index Memory

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Internal initialization logic for selecting the appropriate memory index type based on vector data type.

```python
        if vector_dtype == np.uint8:
            _index = _native_dap.StaticMemoryUInt8Index
        elif vector_dtype == np.int8:
            _index = _native_dap.StaticMemoryInt8Index
        else:
            _index = _native_dap.StaticMemoryFloatIndex

        self._index = _index(
            distance_metric=dap_metric,
            num_points=num_points,
            dimensions=dims,
            index_path=index_prefix_path,
            num_threads=num_threads,
            initial_search_complexity=initial_search_complexity,
        )
```

--------------------------------

### build_memory_index

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Constructs a DiskANN memory index. Memory indices are faster than disk indices but are limited by the available system memory.

```APIDOC
## build_memory_index

### Description
Constructs a DiskANN memory index. This function supports both static and dynamic memory indices depending on the parameters provided.

### Parameters
- **data** (Union[str, numpy.ndarray]) - Required - Path to a vector bin file or a numpy array of supported dtype.
- **distance_metric** (Literal['l2', 'mips', 'cosine']) - Required - The distance metric to use. Note: 'mips' is only available for float32.
- **index_directory** (str) - Required - Directory path where index files will be saved.
- **complexity** (int) - Required - Size of the candidate nearest neighbor list during build.
- **graph_degree** (int) - Required - The degree of the graph index.
- **num_threads** (int) - Required - Number of threads to use for index creation.
- **alpha** (float) - Optional - Default is 1.2000000476837158.
- **use_pq_build** (bool) - Optional - Whether to use PQ build.
- **num_pq_bytes** (int) - Optional - Number of PQ bytes.
- **use_opq** (bool) - Optional - Whether to use OPQ.
- **vector_dtype** (Type) - Optional - Required if data is a string path.
- **tags** (Union[str, numpy.ndarray]) - Optional - Required if building for DynamicMemoryIndex.
- **filter_labels** (list[list[str]]) - Optional - Filter labels for the index.
- **universal_label** (str) - Optional - Universal label string.
- **filter_complexity** (int) - Optional - Complexity for filtering.
- **index_prefix** (str) - Optional - Prefix for index files, defaults to 'ann'.
```

--------------------------------

### Construct a DiskANN disk index

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

This function validates input parameters and invokes the appropriate native builder based on the vector data type to create a disk-based index.

```python
_assert(
    (isinstance(data, str) and vector_dtype is not None)
    or isinstance(data, np.ndarray),
    "vector_dtype is required if data is a str representing a path to the vector bin file",
)
dap_metric = _valid_metric(distance_metric)
_assert_is_positive_uint32(complexity, "complexity")
_assert_is_positive_uint32(graph_degree, "graph_degree")
_assert(search_memory_maximum > 0, "search_memory_maximum must be larger than 0")
_assert(build_memory_maximum > 0, "build_memory_maximum must be larger than 0")
_assert_is_nonnegative_uint32(num_threads, "num_threads")
_assert_is_nonnegative_uint32(pq_disk_bytes, "pq_disk_bytes")
_assert(index_prefix != "", "index_prefix cannot be an empty string")

index_path = Path(index_directory)
_assert(
    index_path.exists() and index_path.is_dir(),
    "index_directory must both exist and be a directory",
)

vector_bin_path, vector_dtype_actual = _valid_path_and_dtype(
    data, vector_dtype, index_directory, index_prefix
)
_assert(dap_metric != _native_dap.COSINE, "Cosine is currently not supported in StaticDiskIndex")
if dap_metric == _native_dap.INNER_PRODUCT:
    _assert(
        vector_dtype_actual == np.float32,
        "Integral vector dtypes (np.uint8, np.int8) are not supported with distance metric mips"
    )

num_points, dimensions = vectors_metadata_from_file(vector_bin_path)

if vector_dtype_actual == np.uint8:
    _builder = _native_dap.build_disk_uint8_index
elif vector_dtype_actual == np.int8:
    _builder = _native_dap.build_disk_int8_index
else:
    _builder = _native_dap.build_disk_float_index

index_prefix_path = os.path.join(index_directory, index_prefix)

_builder(
    distance_metric=dap_metric,
    data_file_path=vector_bin_path,
    index_prefix_path=index_prefix_path,
    complexity=complexity,
    graph_degree=graph_degree,
    final_index_ram_limit=search_memory_maximum,
    indexing_ram_budget=build_memory_maximum,
    num_threads=num_threads,
    pq_disk_bytes=pq_disk_bytes,
)
_write_index_metadata(
    index_prefix_path, vector_dtype_actual, dap_metric, num_points, dimensions
)
```

--------------------------------

### Write tags to a binary file

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Writes a 1D NumPy array of uint32 tags to a DiskANN binary file. 2D arrays with one column are supported but will trigger a copy operation.

```python
75def tags_to_file(tags_file: str, tags: VectorIdentifierBatch) -> None:
76    """
77    Write tags to a DiskANN binary tag file.
78
79    ### Parameters
80    - **tags_file**: The path to the tag file to write the tags to.
81    - **tags**: A 1d array of dtype `numpy.uint32` containing the tags to write. If you have a 2d array of tags with
82      one column, you can pass it here and it will be reshaped and copied to a new array. It is more efficient for you
83      to reshape on your own without copying it first, as it should be a constant time operation vs. linear time
84
85    """
86    _assert(np.can_cast(tags.dtype, np.uint32), "valid tags must be uint32")
87    _assert(
88        len(tags.shape) == 1 or tags.shape[1] == 1,
89        "tags must be 1d or 2d with 1 column",
90    )
91    if len(tags.shape) == 2:
92        warnings.warn(
93            "Tags in 2d with one column will be reshaped and copied to a new array. "
94            "It is more efficient for you to reshape without copying first."
95        )
96        tags = tags.reshape(tags.shape[0], copy=True)
97    with open(tags_file, "wb") as fh:
98        _write_bin(tags.astype(np.uint32), fh)
```

--------------------------------

### Construct a DiskANN disk index

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Function to build a disk-based index for large datasets. Requires an existing directory and specific memory/complexity parameters.

```python
def build_disk_index(
    data: Union[str, VectorLikeBatch],
    distance_metric: DistanceMetric,
    index_directory: str,
    complexity: int,
    graph_degree: int,
    search_memory_maximum: float,
    build_memory_maximum: float,
    num_threads: int,
    pq_disk_bytes: int = defaults.PQ_DISK_BYTES,
    vector_dtype: Optional[VectorDType] = None,
    index_prefix: str = "ann",
) -> None:
```

--------------------------------

### tags_to_file(tags_file: str, tags: numpy.ndarray)

Source: https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html

Writes tags to a DiskANN binary tag file.

```APIDOC
## tags_to_file(tags_file: str, tags: numpy.ndarray)

### Description
Write tags to a DiskANN binary tag file.

### Parameters
- **tags_file** (str) - Required - The path to the tag file to write the tags to.
- **tags** (numpy.ndarray) - Required - A 1d array of dtype `numpy.uint32` containing the tags to write. If you have a 2d array of tags with one column, it will be reshaped.
```
Tokens:
Raw