### Install sqlitesearch Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Install the library using uv or pip. ```bash uv add sqlitesearch # or: pip install sqlitesearch ``` -------------------------------- ### Install libSQL support Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Install the necessary dependencies to use the libSQL backend. ```bash pip install "sqlitesearch[libsql]" ``` -------------------------------- ### Configure VectorSearchIndex parameters Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/WRITEUP.md Examples of initializing the VectorSearchIndex with different configurations for default recall, exact-bucket probing, and high-recall settings. ```python # Default: better recall (0.89 at 100K) index = VectorSearchIndex(db_path="vectors.db") # Exact-bucket probing: fast, low recall index = VectorSearchIndex(n_probe=0, db_path="vectors.db") # Best recall (0.95 at 100K, slower inserts, larger DB) index = VectorSearchIndex(n_tables=16, n_probe=2, db_path="vectors.db") ``` -------------------------------- ### Correct uv add command Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/AGENTS.md Shows the correct way to install Python packages using uv. Run from the backend-django directory. ```bash cd backend-django uv add djangorestframework ``` -------------------------------- ### Incorrect pip install command Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/AGENTS.md Shows the wrong way to install Python packages using pip. Avoid using pip; use uv instead. ```bash pip install djangorestframework ``` -------------------------------- ### Initialize VectorSearchIndex with libSQL Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Configure the index to use libSQL for remote persistence with Turso Cloud. ```python # Embedded replica: reads run against a local file that syncs to Turso Cloud. index = VectorSearchIndex( keyword_fields=["category"], db_path="local-replica.db", # local embedded-replica cache sync_url="libsql://your-db.turso.io", # Turso database URL auth_token="...", # Turso auth token ) ``` -------------------------------- ### Initialize Disk-Backed HNSW Index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Configures an HNSW index with disk_backed=True to reduce serving RAM by using a file-backed memmap for graph-walk vectors. ```python index = VectorSearchIndex( mode="hnsw", disk_backed=True, # drop the vector cache; serve nav vectors from disk db_path="vectors.db" ) ``` -------------------------------- ### Initialize and Use LSH Index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Sets up an LSH index with multiple hash tables and multi-probe lookup, followed by fitting data and performing a search. ```python import numpy as np from sqlitesearch import VectorSearchIndex index = VectorSearchIndex( mode="lsh", keyword_fields=["category"], n_tables=8, # Number of hash tables (more = better recall) hash_size=16, # Bits per hash (more = better precision) n_probe=2, # Multi-probe bit flips (0-2, higher = better recall) db_path="vectors.db" ) vectors = np.random.rand(100, 384) documents = [{"category": "test"} for _ in range(100)] index.fit(vectors, documents) query = np.random.rand(384) results = index.search(query) ``` -------------------------------- ### Seed Turso database from local file Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Use the Turso CLI to create a database from a pre-built local SQLite file. ```bash turso db create my-index --from-file local.db ``` -------------------------------- ### Load all FAQ documents Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/notebooks/faq_search.ipynb Iterates through the courses index to fetch and aggregate all FAQ documents into a single list. ```python documents = [] for course in courses_index: course_path = course['path'] course_url = f'{base_faq_url}/{course_path}' course_data = requests.get(course_url).json() documents.extend(course_data) len(documents) ``` ```python documents[5] ``` -------------------------------- ### Create the search index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/notebooks/faq_search.ipynb Initializes the TextSearchIndex with specified text and keyword fields, then fits the index using the loaded documents. ```python from sqlitesearch import TextSearchIndex index = TextSearchIndex( text_fields=['section', 'question', 'answer'], keyword_fields=['course'], db_path='faq.db' ) index.fit(documents) ``` -------------------------------- ### Initialize IVF Index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Configures an IVF index using k-means clustering to balance build speed and recall. ```python index = VectorSearchIndex( mode="ivf", n_clusters=None, # Auto-scales (sqrt(n), capped at 256) n_probe_clusters=8, # Clusters to search (more = better recall, slower) db_path="vectors.db" ) ``` -------------------------------- ### Ingest and Serve pattern for ephemeral disks Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Split the workflow into an offline ingestion script and a read-only service to optimize performance on ephemeral disks. ```python # ingest.py — run once, writes to Turso index = VectorSearchIndex(keyword_fields=["category"], db_path="cache.db", sync_url=URL, auth_token=TOKEN) index.fit(vectors, docs) # app.py — the running service, read-only index = VectorSearchIndex(keyword_fields=["category"], db_path="cache.db", sync_url=URL, auth_token=TOKEN) results = index.search(query_vector) ``` -------------------------------- ### Run text search benchmarks Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/README.md Commands to download and parse the Simple English Wikipedia dataset, then execute text search benchmarks. ```bash # Download and parse Wikipedia dump pip install mwparserfromhell git clone --depth 1 https://github.com/alexeygrigorev/minsearch.git /tmp/minsearch uv run python /tmp/minsearch/benchmark/download_wikipedia.py uv run python /tmp/minsearch/benchmark/parse_wikipedia.py benchmark/data/simplewiki-*.xml.bz2 # Run benchmark (subset) uv run python benchmark/run_benchmark.py -n 10000 -q 100 # Full benchmark (all dataset sizes) uv run python benchmark/run_full_benchmark.py ``` -------------------------------- ### Implement Hybrid Search with shared database Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Demonstrates initializing both VectorSearchIndex and TextSearchIndex with the same db_path and id_field to enable hybrid search capabilities. ```python from sqlitesearch import TextSearchIndex, VectorSearchIndex docs = [{"doc_id": 1, "title": "Python", "description": "...", "category": "dev"}, ...] # The vector index stores the documents (and their vectors)... vector_index = VectorSearchIndex(keyword_fields=["category"], id_field="doc_id", db_path="hybrid.db") vector_index.fit(query_vectors, docs) # ...and the text index builds its full-text index over the *same* rows. text_index = TextSearchIndex(text_fields=["title", "description"], keyword_fields=["category"], id_field="doc_id", db_path="hybrid.db") text_index.fit(docs) text_results = text_index.search("python tutorial", output_ids=True) vector_results = vector_index.search(query_vector, output_ids=True) # Combine the two result lists by their shared id (e.g. reciprocal rank fusion). ``` -------------------------------- ### Initialize HNSW Index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Configures the default HNSW index with specific parameters for graph construction and search beam widths. ```python index = VectorSearchIndex( mode="hnsw", # the default; shown for clarity m=16, # Max connections per node (more = better recall) ef_construction=200, # Build-time beam width (more = better graph) ef_search=50, # Search-time beam width (more = better recall) db_path="vectors.db" ) ``` -------------------------------- ### Run storage backend benchmark Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/README.md Executes the benchmark script to compare vector ingest time and search latency across storage backends for specified dataset sizes and dimensions. ```bash uv run python benchmark/bench_backends.py --n 1000 10000 100000 --dim 64 --mode lsh ``` -------------------------------- ### Import requests library Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/notebooks/faq_search.ipynb Initial import required for fetching course data. ```python import requests ``` -------------------------------- ### Run vector search benchmarks Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/README.md Commands to compare vector search modes, instrument filtered search, and tune HNSW or LSH parameters using the Cohere-1M dataset. ```bash # Compare all vector search modes at 100K uv run python benchmark/bench_modes.py --n-vectors 100000 # Only HNSW configs uv run python benchmark/bench_modes.py --n-vectors 100000 --modes hnsw # Instrument filtered search hot paths uv run python benchmark/profile_filtered_search.py --modes hnsw --n-vectors 100000 # HNSW recall/latency sweep (builds once, tests multiple ef_search) uv run python benchmark/tune_hnsw_search.py --n-vectors 100000 --ef-search 200 300 500 # LSH parameter tuning uv run python benchmark/tune_recall.py --n-vectors 100000 ``` -------------------------------- ### Basic Text Search Usage Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Initialize a TextSearchIndex, index documents, and perform a search. ```python from sqlitesearch import TextSearchIndex # Create an index index = TextSearchIndex( text_fields=["title", "description"], keyword_fields=["category"], db_path="search.db" ) # Index documents in bulk documents = [ {"id": 1, "title": "Python Tutorial", "description": "Learn Python basics", "category": "tutorial"}, {"id": 2, "title": "Java Guide", "description": "Java programming guide", "category": "guide"}, ] index.fit(documents) # Or add one at a time index.add({"id": 3, "title": "Advanced Python", "description": "Deep dive into Python", "category": "tutorial"}) # Search results = index.search("python programming") for result in results: print(result["title"], result["score"]) ``` -------------------------------- ### Search with filtering and boosting Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/notebooks/faq_search.ipynb Performs a search query with specific filters for the course and boosts for certain fields. ```python question = 'I just discovered the course. Can I join now?' filter_dict = { 'course': 'llm-zoomcamp' } boost_dict = { 'question': 3, 'section': 0.5, } search_results = index.search( question, filter_dict=filter_dict, boost_dict=boost_dict, num_results=5 ) search_results ``` -------------------------------- ### Run Django commands with uv Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/AGENTS.md Commands to run Django management tasks using uv run. Includes makemigrations, migrate, and test. ```bash cd backend-django uv run python manage.py makemigrations uv run python manage.py migrate uv run python manage.py test ``` -------------------------------- ### Persistence of the search index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/notebooks/faq_search.ipynb Demonstrates closing the index and reopening it from the existing 'faq.db' file to resume searching. ```python # Close the current index index.close() # Reopen the existing index index = TextSearchIndex( text_fields=['section', 'question', 'answer'], keyword_fields=['course'], db_path='faq.db' ) # Ready to search immediately! search('certificate') ``` -------------------------------- ### HNSW with faster build using ef_construction=16 Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/WRITEUP.md Configures an HNSW index with ef_construction set to 16 for faster graph building, resulting in lower graph quality. ```python # Faster build, lower graph quality index = VectorSearchIndex(mode="hnsw", ef_construction=16, db_path="vectors.db") ``` -------------------------------- ### VectorSearchIndex - HNSW mode Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Initializes a VectorSearchIndex with HNSW mode, the default. Supports parameters m, ef_construction, ef_search, and db_path. Optionally, disk_backed=True can be used to reduce RAM usage. ```APIDOC ## VectorSearchIndex - HNSW mode ### Description Initializes a VectorSearchIndex with HNSW (Hierarchical Navigable Small World) mode, the default. Builds a multi-layer proximity graph for high recall and fast search. ### Constructor `VectorSearchIndex(mode="hnsw", m=16, ef_construction=200, ef_search=50, db_path="vectors.db", disk_backed=False)` ### Parameters - **mode** (string) - Required - Search mode, set to "hnsw" (default). - **m** (int) - Optional - Max connections per node (more = better recall). Default: 16. - **ef_construction** (int) - Optional - Build-time beam width (more = better graph). Default: 200. - **ef_search** (int) - Optional - Search-time beam width (more = better recall). Default: 50. - **db_path** (string) - Optional - Path to the SQLite database file. Default: "vectors.db". - **disk_backed** (bool) - Optional - If True, drops the float32 vector cache and serves navigation vectors from a file-backed memmap to reduce RAM. Default: False. ### Example ```python index = VectorSearchIndex( mode="hnsw", # the default; shown for clarity m=16, # Max connections per node (more = better recall) ef_construction=200, # Build-time beam width (more = better graph) ef_search=50, # Search-time beam width (more = better recall) db_path="vectors.db" ) ``` ### Example with disk_backed ```python index = VectorSearchIndex( mode="hnsw", disk_backed=True, # drop the vector cache; serve nav vectors from disk db_path="vectors.db" ) ``` ``` -------------------------------- ### Initialize LSH_INT8 Index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Configures an LSH_INT8 index to reduce RAM usage by using an int8-quantized, disk-backed copy of normalized vectors. ```python index = VectorSearchIndex( mode="lsh_int8", keyword_fields=["category"], n_tables=8, hash_size=16, n_probe=2, db_path="vectors.db" ) ``` -------------------------------- ### HNSW default configuration for 100K-scale vector search Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/WRITEUP.md Creates an HNSW index with default parameters, providing a good balance for local 100K-scale vector search. Uses mode='hnsw' and specifies the database path. ```python # HNSW default (good balance for local 100K-scale vector search) index = VectorSearchIndex(mode="hnsw", db_path="vectors.db") ``` -------------------------------- ### IVF with 8 probes (balanced) Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/WRITEUP.md Creates an IVF index with 8 probe clusters for a balanced tradeoff between recall and speed. ```python index = VectorSearchIndex(mode="ivf", n_probe_clusters=8, db_path="vectors.db") ``` -------------------------------- ### Configure Tokenizer and Stemming Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Customize query processing using stemmers and stop word removal. ```python from sqlitesearch import TextSearchIndex, Tokenizer # Built-in Porter stemming: "running" matches "run", "courses" matches "course" index = TextSearchIndex( text_fields=["title", "description"], stemming=True, # disabled by default to match minsearch behavior db_path="search.db" ) # Custom tokenizer: no stop words index = TextSearchIndex( text_fields=["title", "description"], tokenizer=Tokenizer(), db_path="search.db" ) # Custom tokenizer: custom stop words + custom stemmer (any callable(str) -> str) from stemlite import porter_stemmer # pip install "sqlitesearch[stemming]" index = TextSearchIndex( text_fields=["title", "description"], tokenizer=Tokenizer(stop_words={"custom", "words"}, stemmer=porter_stemmer), db_path="search.db" ) # Or pick a stemmer by name (porter / snowball / lancaster) via get_stemmer: from stemlite import get_stemmer index = TextSearchIndex( text_fields=["title", "description"], tokenizer=Tokenizer(stop_words="english", stemmer=get_stemmer("snowball")), db_path="search.db" ) ``` -------------------------------- ### Run filtered search profiler Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/WRITEUP.md Commands to execute the filtered search profiler with specific vector counts and query counts. ```bash uv run python benchmark/profile_filtered_search.py --n-vectors 30000 --n-queries 20 uv run python benchmark/profile_filtered_search.py --modes hnsw --n-vectors 100000 --n-queries 20 ``` -------------------------------- ### HNSW with higher recall using ef_search=1000 Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/WRITEUP.md Configures an HNSW index with ef_search set to 1000 for higher recall, at the cost of slower search performance. ```python # Higher recall at the cost of slower search index = VectorSearchIndex(mode="hnsw", ef_search=1000, db_path="vectors.db") ``` -------------------------------- ### VectorSearchIndex - LSH_INT8 mode Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Initializes a VectorSearchIndex with LSH_INT8 mode, a lower-RAM variant of LSH that uses int8-quantized, disk-backed vectors. ```APIDOC ## VectorSearchIndex - LSH_INT8 mode ### Description Initializes a VectorSearchIndex with LSH_INT8 mode. Identical algorithm and results to plain LSH, but uses an int8-quantized, disk-backed copy of normalized vectors to reduce RAM usage. ### Constructor `VectorSearchIndex(mode="lsh_int8", keyword_fields=[], n_tables=8, hash_size=16, n_probe=2, db_path="vectors.db")` ### Parameters - **mode** (string) - Required - Search mode, set to "lsh_int8". - **keyword_fields** (list) - Optional - List of keyword field names for filtering. Default: []. - **n_tables** (int) - Optional - Number of hash tables (more = better recall). Default: 8. - **hash_size** (int) - Optional - Bits per hash (more = better precision). Default: 16. - **n_probe** (int) - Optional - Multi-probe bit flips (0-2, higher = better recall). Default: 2. - **db_path** (string) - Optional - Path to the SQLite database file. Default: "vectors.db". ### Example ```python index = VectorSearchIndex( mode="lsh_int8", keyword_fields=["category"], n_tables=8, hash_size=16, n_probe=2, db_path="vectors.db" ) ``` ``` -------------------------------- ### IVF with 16 probe clusters (best recall) Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/WRITEUP.md Creates an IVF index with 16 probe clusters for best recall. The n_probe_clusters parameter controls how many clusters are searched; more probes increase recall but slow search. ```python index = VectorSearchIndex(mode="ivf", n_probe_clusters=16, db_path="vectors.db") ``` -------------------------------- ### Fetch the courses index Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/notebooks/faq_search.ipynb Retrieves the list of available courses from the DataTalks Club FAQ endpoint. ```python base_faq_url = 'https://datatalks.club/faq' courses_index_url = f'{base_faq_url}/json/courses.json' courses_index = requests.get(courses_index_url).json() courses_index ``` -------------------------------- ### VectorSearchIndex - LSH mode Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Initializes a VectorSearchIndex with LSH mode. Uses random hyperplane projections and multi-probe lookup. Includes fit and search methods. ```APIDOC ## VectorSearchIndex - LSH mode ### Description Initializes a VectorSearchIndex with LSH (Locality-Sensitive Hashing) mode. Each vector is hashed into buckets using random hyperplane projections. At query time, it looks up matching buckets and reranks candidates by exact cosine similarity. ### Constructor `VectorSearchIndex(mode="lsh", keyword_fields=[], n_tables=8, hash_size=16, n_probe=2, db_path="vectors.db")` ### Parameters - **mode** (string) - Required - Search mode, set to "lsh". - **keyword_fields** (list) - Optional - List of keyword field names for filtering. Default: []. - **n_tables** (int) - Optional - Number of hash tables (more = better recall). Default: 8. - **hash_size** (int) - Optional - Bits per hash (more = better precision). Default: 16. - **n_probe** (int) - Optional - Multi-probe bit flips (0-2, higher = better recall). Default: 2. - **db_path** (string) - Optional - Path to the SQLite database file. Default: "vectors.db". ### Methods #### fit(vectors, documents) Fits the index with the given vectors and associated documents. **Parameters:** - **vectors** (numpy.ndarray) - Required - Array of vectors to index. - **documents** (list) - Required - List of document dictionaries corresponding to each vector. #### search(query) Searches the index for vectors similar to the query vector. **Parameters:** - **query** (numpy.ndarray) - Required - Query vector. **Returns:** - Search results (list) - List of results with similarity scores. ### Example ```python import numpy as np from sqlitesearch import VectorSearchIndex index = VectorSearchIndex( mode="lsh", keyword_fields=["category"], n_tables=8, # Number of hash tables (more = better recall) hash_size=16, # Bits per hash (more = better precision) n_probe=2, # Multi-probe bit flips (0-2, higher = better recall) db_path="vectors.db" ) vectors = np.random.rand(100, 384) documents = [{"category": "test"} for _ in range(100)] index.fit(vectors, documents) query = np.random.rand(384) results = index.search(query) ``` ``` -------------------------------- ### VectorSearchIndex - IVF mode Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Initializes a VectorSearchIndex with IVF mode. Uses k-means clustering and searches only the nearest clusters at query time. ```APIDOC ## VectorSearchIndex - IVF mode ### Description Initializes a VectorSearchIndex with IVF (Inverted File Index) mode. Clusters vectors using k-means, then searches only the nearest clusters at query time. ### Constructor `VectorSearchIndex(mode="ivf", n_clusters=None, n_probe_clusters=8, db_path="vectors.db")` ### Parameters - **mode** (string) - Required - Search mode, set to "ivf". - **n_clusters** (int or None) - Optional - Number of clusters. If None, auto-scales (sqrt(n), capped at 256). Default: None. - **n_probe_clusters** (int) - Optional - Number of clusters to search (more = better recall, slower). Default: 8. - **db_path** (string) - Optional - Path to the SQLite database file. Default: "vectors.db". ### Example ```python index = VectorSearchIndex( mode="ivf", n_clusters=None, # Auto-scales (sqrt(n), capped at 256) n_probe_clusters=8, # Clusters to search (more = better recall, slower) db_path="vectors.db" ) ``` ``` -------------------------------- ### Boost Search Fields Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Assign weights to specific fields to influence search ranking. ```python # Boost title matches higher than description results = index.search("python", boost_dict={"title": 2.0, "description": 1.0}) ``` -------------------------------- ### Define a search function Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/notebooks/faq_search.ipynb Encapsulates the search logic into a reusable function and demonstrates its usage with various queries. ```python def search(question): filter_dict = { 'course': 'llm-zoomcamp' } boost_dict = { 'question': 3, 'section': 0.5, } search_results = index.search( question, filter_dict=filter_dict, boost_dict=boost_dict, num_results=5 ) return search_results ``` ```python search('How do I get a certificate?') ``` ```python search('What are the prerequisites?') ``` ```python search('homework deadline') ``` -------------------------------- ### Profile filtered vector search Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/benchmark/README.md Commands to run the focused profiler for filtered vector queries at different vector counts and modes. ```bash # 30K, all modes uv run python benchmark/profile_filtered_search.py --n-vectors 30000 --n-queries 20 # 100K, HNSW only uv run python benchmark/profile_filtered_search.py --modes hnsw --n-vectors 100000 --n-queries 20 ``` -------------------------------- ### Use Custom ID Field Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Define a custom field name for document identifiers. ```python index = TextSearchIndex( text_fields=["title", "description"], id_field="doc_id", db_path="search.db" ) results = index.search("python", output_ids=True) # Results will include 'id' field with the doc_id value ``` -------------------------------- ### Filter Search Results Source: https://github.com/alexeygrigorev/sqlitesearch/blob/main/README.md Apply keyword, numeric, or date range filters to search queries. ```python # Filter by keyword fields results = index.search("python", filter_dict={"category": "tutorial"}) # Multi-value (IN/OR) keyword filter: match any of the listed values results = index.search("python", filter_dict={"category": ["tutorial", "guide"]}) # Filter by numeric range results = index.search("python", filter_dict={"price": [('>=', 50), ('<', 200)]}) # Exact numeric match results = index.search("python", filter_dict={"price": 100}) # Filter by date range from datetime import date results = index.search("python", filter_dict={ "created_at": [('>=', date(2024, 1, 1)), ('<', date(2024, 12, 31))] }) ```