### Setup MySQL for Testing Source: https://github.com/ankane/neighbor/blob/master/README.md Starts a MySQL Docker container and runs the test suite. Ensure Docker is installed and running. ```sh docker run -e MYSQL_ALLOW_EMPTY_PASSWORD=1 -e MYSQL_DATABASE=neighbor_test -p 3306:3306 mysql:9.7 bundle exec rake test:mysql ``` -------------------------------- ### Setup MariaDB for Testing Source: https://github.com/ankane/neighbor/blob/master/README.md Starts a MariaDB Docker container and runs the test suite. Ensure Docker is installed and running. ```sh docker run -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -e MARIADB_DATABASE=neighbor_test -p 3307:3306 mariadb:11.8 bundle exec rake test:mariadb ``` -------------------------------- ### Setup PostgreSQL for Testing Source: https://github.com/ankane/neighbor/blob/master/README.md Creates a PostgreSQL database for testing and runs the test suite. Requires PostgreSQL to be installed and running. ```sh createdb neighbor_test bundle exec rake test:postgresql ``` -------------------------------- ### Setup SQLite for Testing Source: https://github.com/ankane/neighbor/blob/master/README.md Runs the test suite using SQLite. This is a simpler setup if PostgreSQL is not available. ```sh bundle exec rake test:sqlite bundle exec rake test:sqlitevec bundle exec rake test:vec1 ``` -------------------------------- ### Vector Type Migration Example Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Example of how to add a vector column to a database table using migrations. This demonstrates the usage of the custom vector type with a specified limit for dimensions. ```APIDOC ## Vector Type Migration Example ### Description Column migration for adding a vector type column. ### Usage ```ruby add_column :items, :embedding, :vector, limit: 3 ``` ### Supported Features - **Dimensions**: Specified by `limit` (e.g., 3, 1536). - **Distances**: Supports Euclidean and Cosine distances for MariaDB and MySQL HeatWave. - **Serialization**: Uses IEEE 754 single-precision binary (little-endian, `"e*"` pack format). ``` -------------------------------- ### Cube for Recommendations Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/postgresql.md Set up a model for recommendations using the 'cube' type with specified dimensions and normalization. This example shows how to find the nearest neighbors for a given record. ```ruby class Movie < ApplicationRecord has_neighbors :factors, dimensions: 20, normalize: true end movie = Movie.find(1) similar = movie.nearest_neighbors(:factors, distance: "cosine").first(5) ``` -------------------------------- ### Basic Neighbor Setup in Rails Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Set up a vector column in your migration, define the neighbor relationship in your model, and demonstrate basic usage for finding similar records. ```ruby # Migration add_column :documents, :embedding, :vector, limit: 1536 # Model class Document < ApplicationRecord has_neighbors :embedding end # Usage doc = Document.first similar = doc.nearest_neighbors(:embedding, distance: "cosine").first(5) ``` -------------------------------- ### Usage Example: vec1 Extension Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Demonstrates setting up and querying with a vec1 extension. This speeds up euclidean and cosine distance calculations. ```ruby # In config/initializers/neighbor.rb Neighbor::SQLite.initialize!(extension: "/path/to/vec1.so") # This speeds up euclidean and cosine distance calculations Item.nearest_neighbors(:embedding, [1.0, 2.0, 3.0], distance: "cosine").first(5) ``` -------------------------------- ### Full Neighbor Configuration Example Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Demonstrates a comprehensive configuration of `has_neighbors` with multiple vector columns, including dimensions, normalization, and specific type for SQLite. ```ruby class Document < ApplicationRecord has_neighbors :full_embedding, dimensions: 1536 has_neighbors :normalized_embedding, normalize: true, dimensions: 384 has_neighbors :compressed, dimensions: 256, type: :int8 # SQLite only end ``` -------------------------------- ### Install Neighbor Gem Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Add the Neighbor gem to your Gemfile for installation. ```ruby gem "neighbor" ``` -------------------------------- ### Usage Example: sqlite-vec Gem Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Demonstrates setting up and querying with the sqlite-vec gem extension. Queries use the optimized sqlite-vec extension for nearest neighbor searches. ```ruby # In config/initializers/neighbor.rb Neighbor::SQLite.initialize!(extension: :sqlite_vec) # In your model class Item < ApplicationRecord has_neighbors :embedding end # Queries now use the optimized sqlite-vec extension Item.nearest_neighbors(:embedding, [1.0, 2.0, 3.0], distance: "euclidean").first(5) ``` -------------------------------- ### Usage Example: Virtual Tables with vec1 Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Illustrates creating and querying virtual tables with the vec1 extension. Use for advanced performance tuning when bypassing the ORM query system. ```ruby class CreateItems < ActiveRecord::Migration[8.1] def change create_virtual_table :items, :vec1, ["embedding", "id"] end end # Ignore shadow tables ActiveRecord::SchemaDumper.ignore_tables += [ "items_base", "items_config", "items_idx", "items_meta", "items_model" ] # Query with vec1 syntax Item.find_by_sql("SELECT * FROM items(vec1_from_json(?), ?)", [[1, 2, 3].to_json, {k: 5}.to_json]) ``` -------------------------------- ### Install PostgreSQL Vector Extension Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Ensure the vector extension is installed in your PostgreSQL database before using pgvector. ```sql CREATE EXTENSION IF NOT EXISTS vector; -- For pgvector CREATE EXTENSION IF NOT EXISTS cube; -- For cube ``` -------------------------------- ### Usage Example: Virtual Tables with sqlite-vec Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Shows how to create and query using virtual tables with the sqlite-vec extension. This bypasses the ORM query system for advanced performance tuning. ```ruby class CreateItems < ActiveRecord::Migration[8.1] def change create_virtual_table :items, :vec0, [ "id integer PRIMARY KEY AUTOINCREMENT NOT NULL", "embedding float[3] distance_metric=L2" ] end end # In schema.rb, ignore the generated shadow tables ActiveRecord::SchemaDumper.ignore_tables += [ "items_chunks", "items_rowids", "items_vector_chunks00" ] # Query with virtual table syntax Item.where("embedding MATCH ?", [1, 2, 3].to_s).where(k: 5).order(:distance) ``` -------------------------------- ### Get Nearest Neighbors by Hamming Distance (MariaDB) Source: https://github.com/ankane/neighbor/blob/master/README.md Retrieve the nearest neighbors using Hamming distance for a binary vector column. This example shows how to query for the top 5 nearest neighbors. ```ruby Item.nearest_neighbors(:embedding, 5, distance: "hamming").first(5) ``` -------------------------------- ### Fit Disco Recommender Source: https://github.com/ankane/neighbor/blob/master/README.md Loads MovieLens data and fits a Disco recommender model. Ensure the Disco gem is installed and data is accessible. ```ruby data = Disco.load_movielens recommender = Disco::Recommender.new(factors: 20) recommender.fit(data) ``` -------------------------------- ### Neighbor::SQLite Source: https://github.com/ankane/neighbor/blob/master/_autodocs/00-MANIFEST.txt Documentation for the Neighbor::SQLite module, covering setup and usage of the SQLite extension for vector search. ```APIDOC ## Neighbor::SQLite ### Description Neighbor::SQLite module documentation. initialize! method for extension setup. initialize_adapter!, vec1?, sqlite_vec? methods. extensions property. Usage examples: sqlite-vec gem, vec1 extension. Virtual tables with sqlite-vec and vec1. Custom distance functions documentation. Notes on extension compatibility. Source location. ### Methods - **initialize!**: Sets up the SQLite extension for Neighbor. - **initialize_adapter!**: Initializes the adapter for SQLite. - **vec1?**: Checks if the vec1 extension is available. - **sqlite_vec?**: Checks if the sqlite-vec extension is available. ### Properties - **extensions**: (array) - List of available SQLite extensions. ### Usage Examples - Using the `sqlite-vec` gem. - Integrating with the `vec1` extension. - Creating virtual tables. - Defining custom distance functions. ``` -------------------------------- ### Recommendation System with Neighbor Gem Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Generate recommendations by finding nearest neighbors based on item factors. This example configures the `has_neighbors` association with specific dimensions and normalization. It finds similar movies to a given movie. ```ruby class Movie < ApplicationRecord has_neighbors :factors, dimensions: 20, normalize: true end movie = Movie.find_by(name: "Star Wars") similar = movie.nearest_neighbors(:factors, distance: "cosine").first(5) ``` -------------------------------- ### Get Registered Extensions Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Returns the list of registered extensions (:sqlite_vec symbols and/or string paths). ```ruby Neighbor::SQLite.extensions ``` -------------------------------- ### Binary Vectors on MySQL Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Handle binary vectors in MySQL, storing them as strings. This example demonstrates creating and querying binary vectors using Hamming distance. ```ruby class Document < ApplicationRecord has_neighbors :binary_embedding end # Binary as string doc = Document.create!(binary_embedding: "\x05\x0f") # Query results = Document.nearest_neighbors(:binary_embedding, "\x05\x0f", distance: "hamming").first(5) ``` -------------------------------- ### Convert Dense Array to SparseVector and Inspect Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sparse_vector.md This example shows converting a dense array to a sparse vector and then inspecting its resulting indices and values. ```ruby vector = Neighbor::SparseVector.new([0.9, 0.0, 1.3]) vector.dimensions # => 3 vector.indices # => [0, 2] vector.values # => [0.9, 1.3] ``` -------------------------------- ### Get Nearest Neighbors by Hamming Distance (MySQL) Source: https://github.com/ankane/neighbor/blob/master/README.md Retrieve the nearest neighbors using Hamming distance for a binary vector column in MySQL. This example queries for the top 5 nearest neighbors using a binary literal. ```ruby Item.nearest_neighbors(:embedding, "\x05", distance: "hamming").first(5) ``` -------------------------------- ### Query MariaDB Binary Vector by Hamming Distance Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Example of querying binary vectors using Hamming distance in MariaDB. ```ruby Item.nearest_neighbors(:binary_embedding, 5, distance: "hamming").first(5) ``` -------------------------------- ### Initialize PostgreSQL Module Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/postgresql.md Automatically called when the `pg` gem is available. This method registers custom type classes, defines column methods for migrations, installs type map overrides, and patches array type handling for vector arrays. ```ruby Neighbor::PostgreSQL.initialize! ``` -------------------------------- ### Neighbor::MySQL.initialize! Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md This method is automatically called on ActiveRecord load. It handles the registration of the vector type with the MySQL adapter, defines column methods for migrations, and installs type map overrides for proper type resolution. It is not typically called directly by users. ```APIDOC ## Neighbor::MySQL.initialize! ### Description Automatically called on ActiveRecord load. Performs: 1. Requires the MysqlVector type class 2. Registers vector type with AbstractMysqlAdapter 3. Defines column methods for migrations 4. Installs type map overrides for proper type resolution ### Method `Neighbor::MySQL.initialize!` ### Returns nil ### Access Not typically called directly (automatic) ``` -------------------------------- ### Get Similar Movies Source: https://github.com/ankane/neighbor/blob/master/README.md Finds a specific movie and retrieves its nearest neighbors using cosine distance. This demonstrates how to query for recommendations. ```ruby movie = Movie.find_by(name: "Star Wars (1977)") movie.nearest_neighbors(:factors, distance: "cosine").first(5).map(&:name) ``` -------------------------------- ### Query Nearest Neighbors with Sparse Embeddings Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sparse_vector.md Demonstrates how to use Neighbor::SparseVector embeddings with a model that has a `has_neighbors` association. This example queries for the nearest documents based on an embedding. ```ruby class Document < ApplicationRecord has_neighbors :embedding end embedding = Neighbor::SparseVector.new({0 => 0.9, 1 => 1.3, 2 => 1.1}, 30522) results = Document.nearest_neighbors(:embedding, embedding, distance: "inner_product").first(5) ``` -------------------------------- ### Initialize Neighbor::MySQL Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Automatically called on ActiveRecord load. This method initializes the MySQL adapter by requiring the MysqlVector type class, registering it with AbstractMysqlAdapter, defining column methods for migrations, and installing type map overrides. ```ruby Neighbor::MySQL.initialize! ``` -------------------------------- ### Detect Adapter and Get Operator Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/utils.md Determine the database adapter and retrieve the appropriate operator for vector operations based on the model, column type, and distance metric. The operator varies by database (e.g., "<=" for PostgreSQL, "COSINE" for MySQL). ```ruby model = Document adapter = Neighbor::Utils.adapter(model) column_type = model.columns_hash[:embedding].type distance = "cosine" operator = Neighbor::Utils.operator(adapter, column_type, distance) # => "<=" for PostgreSQL, "COSINE" for MySQL, etc. ``` -------------------------------- ### ActiveRecord Type Casting Example Source: https://github.com/ankane/neighbor/blob/master/_autodocs/types.md Demonstrates setting a vector attribute on an ActiveRecord record, retrieving its deserialized value, and saving it to the database. Assumes a 'record' object with an 'embedding' attribute is already defined. ```ruby record.embedding = [1.0, 2.0, 3.0] # Calls type.cast record.embedding # Returns deserialized value record.save # Calls type.serialize ``` -------------------------------- ### Mocking and Querying Vectors in Rails Tests Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md In tests, mock vector embeddings as arrays and use `nearest_neighbors` to query for similar items. This example demonstrates mocking a `Document` model and performing a cosine distance search. ```ruby # Mock vectors as arrays document = Document.new(embedding: [0.1, 0.2, 0.3]) assert document.valid? # Test queries results = Document.nearest_neighbors(:embedding, [0.1, 0.2, 0.3], distance: "cosine") assert_equal 1, results.count ``` -------------------------------- ### Basic Vector Storage and Querying Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/has_neighbors.md Demonstrates how to set up a model with a single vector column and perform basic nearest neighbor searches. Vectors are automatically serialized and deserialized. ```ruby class Document < ApplicationRecord has_neighbors :embedding end # Store a vector document = Document.create!(content: "Hello world", embedding: [0.1, 0.2, 0.3]) # Get nearest neighbors nearest = document.nearest_neighbors(:embedding, distance: "cosine").first(5) ``` -------------------------------- ### Configure IVFFlat Probes Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/postgresql.md Set the IVFFlat probe parameter using `execute`. Higher values increase accuracy but decrease search speed. ```ruby Item.connection.execute("SET ivfflat.probes = 3") ``` -------------------------------- ### Initialize Neighbor SQLite Configuration Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Configure Neighbor for SQLite by specifying the extension to use. This is typically done in an initializer file. ```ruby # config/initializers/neighbor.rb # Option 1: Use sqlite-vec gem Neighbor::SQLite.initialize!(extension: :sqlite_vec) # Option 2: Use vec1 extension Neighbor::SQLite.initialize!(extension: "/path/to/vec1.so") # Option 3: Use built-in functions (no setup needed) # Just omit the initialize! call ``` -------------------------------- ### Vector Normalization Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/has_neighbors.md Illustrates setting up a model with vector normalization enabled. Normalization is automatically applied during saving and querying. ```ruby class Movie < ApplicationRecord has_neighbors :factors, normalize: true, dimensions: 20 end # Normalization happens automatically on save and query ``` -------------------------------- ### Add Embedding Column to ActiveRecord Model Source: https://github.com/ankane/neighbor/blob/master/README.md Example migration for adding an embedding column to an ActiveRecord model, with variations for different database systems. ```ruby class AddEmbeddingToItems < ActiveRecord::Migration[8.1] def change # pgvector, MariaDB, and MySQL add_column :items, :embedding, :vector, limit: 3 # dimensions # cube add_column :items, :embedding, :cube # SQLite add_column :items, :embedding, :binary end end ``` -------------------------------- ### Initialize SQLite Extension Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Call this once in `config/initializers/neighbor.rb` to configure SQLite extensions. Supports loading the `sqlite-vec` gem or a custom compiled extension path. ```ruby Neighbor::SQLite.initialize!(extension: :sqlite_vec) ``` ```ruby Neighbor::SQLite.initialize!(extension: "/usr/local/lib/vec1.so") ``` -------------------------------- ### Generate Neighbor Migration for Postgres (cube) Source: https://github.com/ankane/neighbor/blob/master/README.md Run this command to generate a migration for adding a cube column when using the cube extension with Postgres. ```sh rails generate neighbor:cube rails db:migrate ``` -------------------------------- ### Initialize SQLite Adapter Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Internal method called automatically to register SQLite type classes. Not intended for direct use. ```ruby Neighbor::SQLite.initialize_adapter! ``` -------------------------------- ### Initialize SQLite Vector Extension Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Initialize the Neighbor SQLite extension for vector search acceleration. This is for exact search only as SQLite does not support indexing directly. ```ruby # No indexing; use extensions for acceleration Neighbor::SQLite.initialize!(extension: :sqlite_vec) ``` -------------------------------- ### Query MariaDB Vector by Cosine Distance Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Example of querying vector embeddings using cosine distance in MariaDB. Ensure the model has `has_neighbors` defined for the embedding column. ```ruby class Document < ApplicationRecord has_neighbors :embedding end ``` ```ruby results = Document.nearest_neighbors(:embedding, [0.1, 0.2, 0.3], distance: "cosine").first(5) ``` -------------------------------- ### Initialize Neighbor SQLite Extension Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Optional: Initialize the Neighbor SQLite extension in an initializer file if you are using SQLite. ```ruby # config/initializers/neighbor.rb (for SQLite extensions, optional) Neighbor::SQLite.initialize!(extension: :sqlite_vec) ``` -------------------------------- ### Multiple Vector Columns Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/has_neighbors.md Shows how to configure a model to handle multiple vector columns for nearest neighbor searches. ```ruby class Product < ApplicationRecord has_neighbors :embedding, :thumbnail_embedding end ``` -------------------------------- ### Initialize SQLite Extension with Valid Type Source: https://github.com/ankane/neighbor/blob/master/_autodocs/errors.md Calling `Neighbor::SQLite.initialize!` with an invalid extension type raises an `ArgumentError`. Use `:sqlite_vec` for the gem or a string path for `.so` files. ```ruby Neighbor::SQLite.initialize!(extension: :invalid) # Raises: ArgumentError "Unsupported extension" ``` -------------------------------- ### MariaDB/MySQL Binary Serialization Source: https://github.com/ankane/neighbor/blob/master/_autodocs/types.md Vectors are stored in a binary format using IEEE 754 single-precision floats. This example shows a 3-element vector resulting in 12 bytes of storage. ```ruby vector = [1.0, 2.0, 3.0] # Stored as binary with 12 bytes (3 floats × 4 bytes) ``` -------------------------------- ### Initialize SQLite Extension Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Registers a SQLite extension for vector operations. Call this in a Rails initializer. ```ruby Neighbor::SQLite.initialize!(extension: :sqlite_vec) ``` -------------------------------- ### Create Documents for Hybrid Search Source: https://github.com/ankane/neighbor/blob/master/README.md Instantiate and save new Document records with sample content. This prepares the data for embedding generation and subsequent searching. ```ruby Document.create!(content: "The dog is barking") Document.create!(content: "The cat is purring") Document.create!(content: "The bear is growling") ``` -------------------------------- ### Catch Configuration Errors Source: https://github.com/ankane/neighbor/blob/master/_autodocs/errors.md Handle `Neighbor::Error` exceptions that arise from incorrect gem configuration, such as specifying an unsupported data type for embeddings. This ensures proper setup of the `has_neighbors` association. ```ruby begin class Item < ApplicationRecord has_neighbors :embedding, type: :int8 end rescue Neighbor::Error => e puts "Configuration error: #{e.message}" end ``` -------------------------------- ### Initialize Vec1 Extension for SQLite Source: https://github.com/ankane/neighbor/blob/master/README.md Initialize the Vec1 SQLite extension for improved performance with Euclidean and cosine distances. Provide the path to the compiled extension. ```ruby Neighbor::SQLite.initialize!(extension: "/path/to/vec1.so") ``` -------------------------------- ### SQLite Binary Blob Serialization (int8) Source: https://github.com/ankane/neighbor/blob/master/_autodocs/types.md Vectors can be stored as binary blobs using the :int8 type, where each element uses 1 byte. This example shows a 3-element integer vector. ```ruby # With type: :int8 vector = [1, 2, 3] # Stored as packed binary "c*" format ``` -------------------------------- ### Neighbor::PostgreSQL Source: https://github.com/ankane/neighbor/blob/master/_autodocs/00-MANIFEST.txt Documentation for the Neighbor::PostgreSQL module, detailing automatic initialization, supported types, and indexing options. ```APIDOC ## Neighbor::PostgreSQL ### Description Neighbor::PostgreSQL module documentation. Overview of automatic initialization. initialize! method. Registered types: vector, halfvec, sparsevec, cube. Type registration and column definition. Indexing support: HNSW and IVFFlat. Configuration for HNSW and IVFFlat parameters. Array type support. Model usage examples. Internal implementation details. Requirements. Source location. ### Methods - **initialize!**: Initializes Neighbor for PostgreSQL, often automatically. ### Registered Types - **vector**: Standard vector type. - **halfvec**: Half-precision vector type. - **sparsevec**: Sparse vector type. - **cube**: Cube data type. ### Indexing Support - **HNSW**: Hierarchical Navigable Small Worlds. - **IVFFlat**: Inverted File Flat. ### Configuration - Options for configuring HNSW and IVFFlat parameters. ### Usage Examples - Defining vector columns in migrations. - Using Neighbor with ActiveRecord models. - Examples demonstrating array type support. ``` -------------------------------- ### Recommendation System with Neighbor Source: https://github.com/ankane/neighbor/blob/master/_autodocs/INDEX.md Implement a recommendation system by finding items similar to a given item based on their factor vectors. Normalize vectors for better performance and specify the dimensions. ```ruby class Movie < ApplicationRecord has_neighbors :factors, normalize: true, dimensions: 20 end movie = Movie.find(1) similar = movie.nearest_neighbors(:factors, distance: "cosine").first(5) ``` -------------------------------- ### SQLite Binary Blob Serialization (float32) Source: https://github.com/ankane/neighbor/blob/master/_autodocs/types.md Vectors are stored as binary blobs. The default format uses float32 (4 bytes per element). This example shows a 3-element vector. ```ruby # Default (float32) vector = [1.0, 2.0, 3.0] # Stored as packed binary "f*" format ``` -------------------------------- ### Query MySQL HeatWave Binary Vector by Hamming Distance Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Example of querying binary vectors using Hamming distance in MySQL HeatWave. The `DISTANCE()` function is only available with HeatWave service enabled. ```ruby Item.nearest_neighbors(:binary_embedding, "\x05", distance: "hamming").first(5) ``` -------------------------------- ### Initialize Embedding Model for Sparse Search Source: https://github.com/ankane/neighbor/blob/master/README.md Load a pre-trained masked language model and its tokenizer using Transformers.rb to generate sparse embeddings. ```ruby class EmbeddingModel def initialize(model_id) @model = Transformers::AutoModelForMaskedLM.from_pretrained(model_id) @tokenizer = Transformers::AutoTokenizer.from_pretrained(model_id) @special_token_ids = @tokenizer.special_tokens_map.map { |_, token| @tokenizer.vocab[token] } end def embed(input) feature = @tokenizer.(input, padding: true, truncation: true, return_tensors: "pt", return_token_type_ids: false) output = @model.(**feature)[0] values = Torch.max(output * feature[:attention_mask].unsqueeze(-1), dim: 1)[0] values = Torch.log(1 + Torch.relu(values)) values[0.., @special_token_ids] = 0 values.to_a end end model = EmbeddingModel.new("opensearch-project/opensearch-neural-sparse-encoding-v1") ``` -------------------------------- ### Create MariaDB Vector Index Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Use `null: false` on vector columns to enable indexing in MariaDB. ```ruby class CreateItems < ActiveRecord::Migration[8.1] def change create_table :items do |t| t.vector :embedding, limit: 3, null: false t.index :embedding, type: :vector end end end ``` -------------------------------- ### Configure PostgreSQL HNSW Parameters Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Set HNSW search parameters in PostgreSQL using raw SQL execution. This is typically done in a migration or seed file. ```ruby # In a migration or seed file Item.connection.execute("SET hnsw.ef_search = 100") ``` -------------------------------- ### Binary Embeddings Search Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Perform nearest neighbor search using binary embeddings, suitable for models like Cohere. This example uses the 'hamming' distance metric. Ensure the Document model has `has_neighbors` configured for the `:embedding` attribute. ```ruby class Document < ApplicationRecord has_neighbors :embedding end binary_string = embedding_model.call(query) results = Document.nearest_neighbors(:embedding, binary_string, distance: "hamming").first(10) ``` -------------------------------- ### Hybrid Search with Keyword and Semantic Results Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Combine keyword search results with semantic search results for a more comprehensive search experience. This example uses RRF for reranking. Ensure the Document model has `has_neighbors` configured for the `:embedding` attribute. ```ruby keyword_results = Document.where("content ILIKE ?", "%#{query}%").limit(10) embedding = embedding_model.call(query) semantic_results = Document.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(10) combined = Neighbor::Reranking.rrf(keyword_results, semantic_results).first(10).map { |h| h[:result] } ``` -------------------------------- ### Generate cube Migration Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Use this generator to create a migration for cube. Run `rails db:migrate` afterwards. ```bash rails generate neighbor:cube rails db:migrate ``` -------------------------------- ### Add Binary Quantization Index Source: https://github.com/ankane/neighbor/blob/master/README.md Create an index for binary vectors using expression indexing and binary quantization. This approach optimizes searches by applying a quantization function before indexing. ```ruby class AddIndexToItemsEmbedding < ActiveRecord::Migration[8.1] def change add_index :items, "(binary_quantize(embedding)::bit(3)) bit_hamming_ops", using: :hnsw end end ``` -------------------------------- ### Sparse Embeddings Search Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Execute nearest neighbor search with sparse embeddings, useful for models like Neural Sparse. This example requires creating a `Neighbor::SparseVector` object and uses the 'inner_product' distance. Ensure the Document model has `has_neighbors` configured for the `:embedding` attribute. ```ruby class Document < ApplicationRecord has_neighbors :embedding end embedding_dict = sparse_embedding_model.call(query) # {0 => 0.9, 2 => 1.3, ...} sparse_vec = Neighbor::SparseVector.new(embedding_dict, 30522) results = Document.nearest_neighbors(:embedding, sparse_vec, distance: "inner_product").first(10) ``` -------------------------------- ### Configure HNSW Search Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/postgresql.md Adjust the HNSW search parameter `ef_search` using `execute`. Larger values improve accuracy at the cost of speed. ```ruby Item.connection.execute("SET hnsw.ef_search = 100") ``` -------------------------------- ### SQLite with Binary Vectors Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/has_neighbors.md Shows how to configure `has_neighbors` for SQLite using the `:bit` vector type, suitable for binary data. Dimensions must be specified. ```ruby class Document < ApplicationRecord has_neighbors :embedding, dimensions: 64, type: :bit end Document.nearest_neighbors(:embedding, "\x05\x0f", distance: "hamming").first(5) ``` -------------------------------- ### Clone Neighbor Repository Source: https://github.com/ankane/neighbor/blob/master/README.md Clones the Neighbor project repository from GitHub. This is the first step for local development. ```sh git clone https://github.com/ankane/neighbor.git cd neighbor bundle install ``` -------------------------------- ### Instance Method Signature Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/nearest_neighbors.md Defines the instance method signature for finding nearest neighbors. Use this for querying within a specific record's context. ```ruby def nearest_neighbors(attribute_name, **options) ``` -------------------------------- ### Basic Vector Storage with ActiveRecord Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/postgresql.md Define an ActiveRecord model with a vector embedding and create an instance. ```ruby class Document < ApplicationRecord has_neighbors :embedding end doc = Document.create!(content: "text", embedding: [0.1, 0.2, 0.3]) doc.embedding # => [0.1, 0.2, 0.3] ``` -------------------------------- ### Neighbor::MySQL Source: https://github.com/ankane/neighbor/blob/master/_autodocs/00-MANIFEST.txt Documentation for the Neighbor::MySQL module, covering integration with MariaDB and MySQL HeatWave, including vector types and indexing. ```APIDOC ## Neighbor::MySQL ### Description Neighbor::MySQL module documentation. Overview for MariaDB and MySQL HeatWave. initialize! method. Registered vector type. Column definition in migrations. Indexing for MariaDB (requires null: false). Binary vectors for MariaDB and MySQL. Distance metrics for each database. Model usage examples with various vector types. Internal implementation details. Database compatibility matrix. Performance notes. Source location. ### Methods - **initialize!**: Initializes Neighbor for MySQL/MariaDB. ### Vector Type - **Registered vector type**: Details on the specific vector type supported. ### Indexing - **MariaDB Indexing**: Requirements and configuration for indexing (e.g., `null: false`). ### Binary Vectors - Support and usage of binary vectors for MariaDB and MySQL. ### Distance Metrics - Supported distance metrics for each database version. ### Usage Examples - Defining vector columns in migrations. - Using Neighbor with ActiveRecord models for MySQL/MariaDB. - Examples with different vector types and distance metrics. ``` -------------------------------- ### Convert Sparse to Dense and Back Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sparse_vector.md Demonstrates the round trip conversion from a sparse vector to a dense array and back to a sparse vector. This is useful for verifying data or when an operation requires a dense format. ```ruby sparse = Neighbor::SparseVector.new({0 => 1.5, 2 => 2.5}, 3) dense = sparse.to_a # => [1.5, 0.0, 2.5] sparse2 = Neighbor::SparseVector.new(dense) sparse2.indices # => [0, 2] ``` -------------------------------- ### Neighbor::SQLite.initialize! Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Registers a SQLite extension for vector operations. This method should be called in a Rails initializer. ```APIDOC ## Neighbor::SQLite.initialize! ### Description Registers a SQLite extension for vector operations. Call this in a Rails initializer. ### Method `Neighbor::SQLite.initialize!(extension: :sqlite_vec)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **extension** (Symbol, String) - Optional - Default: `:sqlite_vec` - Extension to load; `:sqlite_vec` (gem) or path to `.so` file ### Returns None ### Raises - **ArgumentError**: if extension is not `:sqlite_vec` or a string path - **Gem::LoadError**: if sqlite_vec gem is not installed when using `:sqlite_vec` ``` -------------------------------- ### Generate Neighbor Migration for Postgres (pgvector) Source: https://github.com/ankane/neighbor/blob/master/README.md Run this command to generate a migration for adding a vector column when using the pgvector extension with Postgres. ```sh rails generate neighbor:vector rails db:migrate ``` -------------------------------- ### Pass Input for OpenAI Embeddings Source: https://github.com/ankane/neighbor/blob/master/README.md Prepares an array of input strings for embedding generation using the `embed` method. ```ruby input = [ "The dog is barking", "The cat is purring", "The bear is growling" ] embeddings = embed(input) ``` -------------------------------- ### Create Vector Index (IVFFlat) Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/postgresql.md Add an IVFFlat index to a vector column using `add_index` in migrations. Specify the operation class for distance calculation. ```ruby add_index :items, :embedding, using: :ivfflat, opclass: :vector_l2_ops ``` -------------------------------- ### Neighbor::SparseVector Initialization from Hash Source: https://github.com/ankane/neighbor/blob/master/_autodocs/types.md Create a sparse vector using a hash where keys are 0-indexed dimensions and values are the corresponding non-zero elements. Specify the total dimensions. ```ruby Neighbor::SparseVector.new({0 => 0.9, 2 => 1.3}, 3) # from hash ``` -------------------------------- ### Add MariaDB vector Column and Index Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Add a `vector` column with native indexing to a MariaDB table. Ensure `null: false` is used for the vector column. ```ruby add_column :items, :embedding, :vector, limit: 3, null: false add_index :items, :embedding, type: :vector ``` -------------------------------- ### Neighbor::SparseVector Initialization from Array Source: https://github.com/ankane/neighbor/blob/master/_autodocs/types.md Create a sparse vector from a dense array representation. Missing elements are assumed to be zero. ```ruby Neighbor::SparseVector.new([0.9, 0.0, 1.3]) # from array ``` -------------------------------- ### Perform Keyword Search Source: https://github.com/ankane/neighbor/blob/master/README.md Execute a keyword search on the Document model using the defined `search` scope. `load_async` can be used for performance. ```ruby query = "growling bear" keyword_results = Document.search(query).limit(20).load_async ``` -------------------------------- ### Create Table with Vector Column for MariaDB Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Define a table with a vector column for MariaDB 11.8+. Ensure binary vectors use `null: false` for indexing. ```ruby # Migration create_table :items do |t| t.vector :embedding, limit: 3, null: false t.index :embedding, type: :vector end ``` -------------------------------- ### Vector with Indexing on MariaDB Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Create a migration to add a vector column with an index for efficient searching. This requires the `pgvector` or a compatible extension. ```ruby class Item < ActiveRecord::Migration[8.1] def change create_table :items do |t| t.string :name t.vector :embedding, limit: 1536, null: false t.timestamps end add_index :items, :embedding, type: :vector end end class Item < ApplicationRecord has_neighbors :embedding end ``` -------------------------------- ### Pass Input for Cohere Embeddings Source: https://github.com/ankane/neighbor/blob/master/README.md Prepares an array of input strings for embedding generation using the `embed` method with 'search_document' input type. ```ruby input = [ "The dog is barking", "The cat is purring", "The bear is growling" ] embeddings = embed(input, "search_document") ``` -------------------------------- ### Supported Distance Metrics Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/nearest_neighbors.md This section details the supported distance metrics for nearest neighbor searches across various database adapters. ```APIDOC ## Supported Distance Metrics ### PostgreSQL (pgvector) - `"euclidean"` — Euclidean distance (L2) - `"inner_product"` — Negative inner product - `"cosine"` — Cosine distance - `"taxicab"` — Manhattan distance (L1) - `"hamming"` — Hamming distance (for bit type) - `"jaccard"` — Jaccard distance (for bit type) ### PostgreSQL (cube) - `"euclidean"` — Euclidean distance - `"cosine"` — Cosine distance (requires normalized vectors) - `"taxicab"` — Manhattan distance - `"chebyshev"` — Chebyshev distance ### MariaDB 11.8+ - `"euclidean"` — Euclidean distance - `"cosine"` — Cosine distance - `"hamming"` — Hamming distance (for binary vectors stored as bigint) ### MySQL 9.7+ (HeatWave only) - `"euclidean"` — Euclidean distance - `"cosine"` — Cosine distance - `"hamming"` — Hamming distance (for binary vectors) ### SQLite - `"euclidean"` — Euclidean distance (L2) - `"inner_product"` — Negative inner product - `"cosine"` — Cosine distance - `"taxicab"` — Manhattan distance (L1) - `"hamming"` — Hamming distance (for binary vectors) - `"jaccard"` — Jaccard distance (for binary vectors) ``` -------------------------------- ### Create Vec1 Virtual Table (Rails 8+) Source: https://github.com/ankane/neighbor/blob/master/README.md Create a virtual table using the Vec1 extension for Rails 8 and later. This allows efficient querying of neighbor data. ```ruby class CreateItems < ActiveRecord::Migration[8.1] def change # Rails 8+ create_virtual_table :items, :vec1, ["embedding", "id"] # Rails < 8 execute "CREATE VIRTUAL TABLE items USING vec1(embedding, id)" end end ``` -------------------------------- ### Neighbor::SQLite.extensions Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Returns the list of registered extensions. ```APIDOC ## Neighbor::SQLite.extensions ### Description Returns the list of registered extensions (`:sqlite_vec` symbols and/or string paths). ### Method `Neighbor::SQLite.extensions` ### Returns Array of registered extensions ``` -------------------------------- ### Generate sqlite-vec Migration Source: https://github.com/ankane/neighbor/blob/master/README.md Run the Rails generator to create the necessary migration files for the sqlite-vec extension. ```sh rails generate neighbor:sqlite ``` -------------------------------- ### Neighbor::SQLite.vec1? Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Checks if a vec1 extension (by path) is loaded. ```APIDOC ## Neighbor::SQLite.vec1? ### Description Checks if a vec1 extension (by path) is loaded. ### Method `Neighbor::SQLite.vec1?` ### Returns `true` if any string extension path is registered ``` -------------------------------- ### Add PostgreSQL halfvec Column and Index Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Add a `halfvec` column for half-precision embeddings with HNSW indexing to a PostgreSQL table. ```ruby add_column :items, :embedding, :halfvec, limit: 1536 add_index :items, :embedding, using: :hnsw, opclass: :halfvec_l2_ops ``` -------------------------------- ### Halfvec with Normalization Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/postgresql.md Configure a model to store normalized vectors using the 'normalize: true' option. The vector will be automatically normalized upon storage. ```ruby class Document < ApplicationRecord has_neighbors :embedding, normalize: true end doc = Document.create!(embedding: [1.0, 2.0, 3.0]) # Stored as normalized vector ``` -------------------------------- ### Basic Vector Search on MariaDB Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/mysql.md Define a model with a vector attribute and perform a basic similarity search. Ensure the `neighbors` gem is configured. ```ruby class Document < ApplicationRecord has_neighbors :embedding end # Create with vector doc = Document.create!( content: "sample text", embedding: [0.1, 0.2, 0.3] ) # Query by similarity results = Document.nearest_neighbors(:embedding, [0.1, 0.2, 0.3], distance: "cosine").first(5) ``` -------------------------------- ### Create Cohere Embeddings API Method Source: https://github.com/ankane/neighbor/blob/master/README.md Defines a method to call the Cohere embed API, handling authentication and request/response parsing. Requires CO_API_KEY environment variable. Returns binary embeddings. ```ruby def embed(input, input_type) url = "https://api.cohere.com/v2/embed" headers = { "Authorization" => "Bearer #{ENV.fetch("CO_API_KEY")}", "Content-Type" => "application/json" } data = { texts: input, model: "embed-v4.0", input_type: input_type, embedding_types: ["ubinary"] } response = Net::HTTP.post(URI(url), data.to_json, headers).tap(&:value) JSON.parse(response.body)["embeddings"]["ubinary"].map { |e| e.map { |v| v.chr.unpack1("B*") }.join } end ``` -------------------------------- ### Neighbor::SparseVector Initialization from Text Source: https://github.com/ankane/neighbor/blob/master/_autodocs/types.md Parse a string representation of a sparse vector into a Neighbor::SparseVector object. The format is {index:value,...}/dimensions. ```ruby Neighbor::SparseVector.from_text("{1:0.9,2:1.3}/3") # from text ``` -------------------------------- ### Configure Schema Dumping for Virtual Tables Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md When using `sqlite-vec` or `vec1` virtual tables, configure schema dumping to ignore shadow tables. Add this to `config/initializers/neighbor.rb`. ```ruby ActiveRecord::SchemaDumper.ignore_tables += [ # For sqlite-vec "items_chunks", "items_rowids", "items_vector_chunks00", # For vec1 "items_base", "items_config", "items_idx", "items_meta", "items_model" ] ``` -------------------------------- ### Add IVFFlat Index to PostgreSQL Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Use this to add an IVFFlat index for approximate nearest neighbor search in PostgreSQL. Suitable for moderate vector dimensions. ```ruby add_index :documents, :embedding, using: :ivfflat, opclass: :vector_l2_ops ``` -------------------------------- ### Query k Nearest Neighbors with Vec1 Source: https://github.com/ankane/neighbor/blob/master/README.md Retrieve the k nearest neighbors using a raw SQL query with the Vec1 extension. Pass vector data and options as JSON. ```ruby Item.find_by_sql("SELECT * FROM items(vec1_from_json(?), ?)", [[1, 2, 3].to_json, {k: 5}.to_json]) ``` -------------------------------- ### Add PostgreSQL pgvector Column and Index Source: https://github.com/ankane/neighbor/blob/master/_autodocs/configuration.md Add a `vector` column with HNSW indexing to a PostgreSQL table. ```ruby add_column :items, :embedding, :vector, limit: 1536 add_index :items, :embedding, using: :hnsw, opclass: :vector_l2_ops ``` -------------------------------- ### Neighbor::SQLite.sqlite_vec? Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Checks if the sqlite-vec gem extension is loaded. ```APIDOC ## Neighbor::SQLite.sqlite_vec? ### Description Checks if the sqlite-vec gem extension is loaded. ### Method `Neighbor::SQLite.sqlite_vec?` ### Returns `true` if `:sqlite_vec` is registered ``` -------------------------------- ### Detect Database Adapter Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/utils.md Identifies the database adapter used by an ActiveRecord model's connection. It specifically detects MariaDB via its `mariadb?()` method and defaults to PostgreSQL for unrecognized adapters. ```ruby Neighbor::Utils.adapter(model) ``` -------------------------------- ### Store Item Factors in Database Source: https://github.com/ankane/neighbor/blob/master/README.md Iterates through recommender item factors and stores them in the Movie model. This prepares the data for querying nearest neighbors. ```ruby movies = [] recommender.item_ids.each do |item_id| movies << {name: item_id, factors: recommender.item_factors(item_id)} end Movie.create!(movies) ``` -------------------------------- ### Store OpenAI Embeddings in Database Source: https://github.com/ankane/neighbor/blob/master/README.md Stores the generated content and embeddings into the Document table using `insert_all!`. ```ruby documents = [] input.zip(embeddings) do |content, embedding| documents << {content: content, embedding: embedding} end Document.insert_all!(documents) ``` -------------------------------- ### Binary Vectors for Nearest Neighbors (SQLite) Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/nearest_neighbors.md Perform nearest neighbor search with binary vectors stored in SQLite. The 'hamming' distance is used for binary comparisons. ```ruby # SQLite with binary vector results = Document.nearest_neighbors(:embedding, "\x05", distance: "hamming").first(5) ``` -------------------------------- ### Perform Cosine Search with Half-Precision Source: https://github.com/ankane/neighbor/blob/master/_autodocs/README.md Execute a nearest neighbor search using the cosine distance metric with half-precision for reduced query cost. The database maintains full precision. ```ruby results = Document.nearest_neighbors(:embedding, vec, distance: "cosine", precision: "half") ``` -------------------------------- ### Query k Nearest Neighbors with sqlite-vec Source: https://github.com/ankane/neighbor/blob/master/README.md Retrieve the k nearest neighbors using the sqlite-vec extension. Filter by embedding and order by distance. ```ruby Item.where("embedding MATCH ?", [1, 2, 3].to_s).where(k: 5).order(:distance) ``` -------------------------------- ### Neighbor::Reranking Source: https://github.com/ankane/neighbor/blob/master/_autodocs/00-MANIFEST.txt Documentation for the Neighbor::Reranking module, focusing on the Reciprocal Rank Fusion (RRF) method for combining search results. ```APIDOC ## Neighbor::Reranking ### Description Neighbor::Reranking module documentation. rrf method: signature and parameters. Returns value structure. RRF algorithm explanation. Usage examples: hybrid search, custom k parameter. Extracting results and scores. Notes on ranking combination. Source location. ### Method - **rrf(results, k: nil, **options)**: Applies Reciprocal Rank Fusion to combine search results. ### Parameters #### Path Parameters - **results** (array) - Required - A list of search result sets to be reranked. - **k** (integer) - Optional - The number of top results to consider after reranking. #### Query Parameters - **options** (hash) - Additional options for the RRF algorithm. ### Returns Value - **reranked_results** (array) - The combined and reranked search results. ### Usage Examples - Examples for hybrid search scenarios. - Using a custom `k` parameter. - Extracting final scores and results. ``` -------------------------------- ### Check vec1 Extension Loaded Source: https://github.com/ankane/neighbor/blob/master/_autodocs/api-reference/sqlite.md Checks if a vec1 extension (by path) is loaded. Returns true if any string extension path is registered. ```ruby Neighbor::SQLite.vec1? ```