### Run an Example Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Execute a loading example after setting up the example database. ```sh createdb pgvector_example cd examples go mod tidy go run ./loading ``` -------------------------------- ### Install pgvector-go Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Install the pgvector-go library using go get. ```sh go get github.com/pgvector/pgvector-go ``` -------------------------------- ### Example Organization Structure Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md The examples directory showcases various use cases, including OpenAI and Cohere embeddings, hybrid search, sparse vectors, recommendations, distributed examples, and bulk loading. ```go examples/ ├── openai/main.go - OpenAI embeddings example ├── cohere/main.go - Cohere embeddings example ├── hybrid/main.go - Hybrid search (vector + BM25) ├── sparse/main.go - Sparse vector example ├── disco/main.go - Recommendation example ├── citus/main.go - Distributed example └── loading/main.go - Bulk loading example ``` -------------------------------- ### sqlx Installation Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Install sqlx and pgvector-go. ```sh go get github.com/pgvector/pgvector-go go get github.com/jmoiron/sqlx ``` -------------------------------- ### Install pgvector-go and Ent extension Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Install the pgvector-go library and its Ent extension using go get. ```sh go get github.com/pgvector/pgvector-go go get github.com/pgvector/pgvector-go/ent ``` -------------------------------- ### Set up Development Environment Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Clone the repository, install dependencies, and set up the test database. ```sh git clone https://github.com/pgvector/pgvector-go.git cd pgvector-go go mod tidy createdb pgvector_go_test cd test go mod tidy go generate ./ent go mod tidy go test -v ``` -------------------------------- ### GORM Installation Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Install GORM, the PostgreSQL driver, and pgvector-go. ```sh go get github.com/pgvector/pgvector-go go get gorm.io/gorm go get gorm.io/driver/postgres ``` -------------------------------- ### Ent Vector Query Setup and Usage Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/ent.md This Go code sets up an Ent client with pgvector support and demonstrates querying for nearest neighbors using L2 and Cosine distances. Ensure Ent client initialization with pgvector support is done prior to these examples. ```go package main import ( "context" "log" "sql" "entgo.io/ent/dialect/sql" pgxvec "github.com/pgvector/pgvector-go/pgx" "github.com/pgvector/pgvector-go" entvec "github.com/pgvector/pgvector-go/ent" entsql "entgo.io/ent/dialect/sql" ) func main() { // Setup: client initialization with pgvector support // (Ent client creation code...) // Example: Find 5 nearest neighbors using L2 distance ctx := context.Background() queryVec := pgvector.NewVector([]float32{1, 2, 3}) items, err := client.Item. Query(). Order(func(s *entsql.Selector) { s.OrderExpr(entvec.L2Distance("embedding", queryVec)) }). Limit(5). All(ctx) if err != nil { log.Fatal(err) } // Example: Find items using cosine distance items2, err := client.Item. Query(). Order(func(s *entsql.Selector) { s.OrderExpr(entvec.CosineDistance("embedding", queryVec)) }). Limit(10). All(ctx) if err != nil { log.Fatal(err) } } ``` -------------------------------- ### Install pgx and pgvector-go pgx package Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Install the necessary packages for pgx integration. This includes the main pgvector-go library and its pgx-specific module. ```sh go get github.com/pgvector/pgvector-go go get github.com/pgvector/pgvector-go/pgx ``` -------------------------------- ### Error Handling Examples Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Demonstrates common error handling patterns for database operations across various Go libraries. ```go // pg err := db.Model(&items).Select() if err != nil { // Handle error } // Bun err = db.NewSelect().Model(&items).Scan(ctx) if err != nil { // Handle error } // Ent items, err := client.Item.Query().All(ctx) if err != nil { // Handle error } // GORM result := db.Find(&items) if result.Error != nil { // Handle error } // sqlx err = db.Select(&items, query) if err != nil { // Handle error } ``` -------------------------------- ### Define Vector Column Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Examples of defining a vector column type for different libraries. ```go // pg Embedding pgvector.Vector `pg:"type:vector(3)"` // Bun Embedding pgvector.Vector `bun:"type:vector(3)"` // Ent field.Other("embedding", pgvector.Vector{}). SchemaType(map[string]string{dialect.Postgres: "vector(3)"}) // GORM Embedding pgvector.Vector `gorm:"type:vector(3)"` // sqlx Embedding pgvector.Vector `db:"embedding"` ``` -------------------------------- ### Example: Register pgvector Types with a pgx Connection Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/pgx.md Demonstrates how to register pgvector types with a standard pgx connection. Ensure the pgvector extension is enabled in your database. ```go package main import ( "context" "log" "github.com/jackc/pgx/v5" pgxvec "github.com/pgvector/pgvector-go/pgx" ) func main() { conn, err := pgx.Connect(context.Background(), "postgres://user:password@localhost/dbname") if err != nil { log.Fatal(err) } defer conn.Close(context.Background()) // Register pgvector types err = pgxvec.RegisterTypes(context.Background(), conn) if err != nil { log.Fatal(err) } // Now can use pgvector types with this connection } ``` -------------------------------- ### Vector String Format Example Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/types.md Illustrates the string representation for a Vector type. This format is used for data exchange and storage. ```text [1,2.5,3] ``` -------------------------------- ### Handling PostgreSQL Registration Errors Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/errors.md Provides a comprehensive example of handling potential errors during database connection, enabling the pgvector extension, and registering pgvector types with pgx. ```Go ctx := context.Background() conn, err := pgx.Connect(ctx, dsn) if err != nil { return err } deferr conn.Close(ctx) // Ensure extension exists _, err = conn.Exec(ctx, "CREATE EXTENSION IF NOT EXISTS vector") if err != nil { return fmt.Errorf("failed to enable vector extension: %w", err) } // Register types er = pgxvec.RegisterTypes(ctx, conn) if err != nil { return fmt.Errorf("failed to register pgvector types: %w", err) } ``` -------------------------------- ### Vector Operations with pgx Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/pgx.md This Go code snippet demonstrates a full usage pattern for pgvector-go with the pgx driver. It includes connecting to the database, registering pgvector types, creating a table, inserting a vector, and performing a nearest neighbor search. Ensure you have the pgx driver and pgvector-go library installed. ```go package main import ( "context" "log" "github.com/jackc/pgx/v5" pgxvec "github.com/pgvector/pgvector-go/pgx" "github.com/pgvector/pgvector-go" ) type Item struct { ID int64 Embedding pgvector.Vector } func main() { ctx := context.Background() conn, err := pgx.Connect(ctx, "postgres://user:password@localhost/dbname") if err != nil { log.Fatal(err) } defer conn.Close(ctx) // Register pgvector types if err := pgxvec.RegisterTypes(ctx, conn); err != nil { log.Fatal(err) } // Create table _, err = conn.Exec(ctx, "CREATE EXTENSION IF NOT EXISTS vector") if err != nil { log.Fatal(err) } _, err = conn.Exec(ctx, "CREATE TABLE IF NOT EXISTS items (id bigserial PRIMARY KEY, embedding vector(3))") if err != nil { log.Fatal(err) } // Insert a vector embedding := pgvector.NewVector([]float32{1, 2, 3}) var id int64 err = conn.QueryRow(ctx, "INSERT INTO items (embedding) VALUES ($1) RETURNING id", embedding).Scan(&id) if err != nil { log.Fatal(err) } // Query nearest neighbors queryVec := pgvector.NewVector([]float32{1, 2, 3}) rows, err := conn.Query(ctx, "SELECT id, embedding FROM items ORDER BY embedding <-> $1 LIMIT 5", queryVec) if err != nil { log.Fatal(err) } defer rows.Close() var items []Item for rows.Next() { var item Item if err := rows.Scan(&item.ID, &item.Embedding); err != nil { log.Fatal(err) } items = append(items, item) } if err := rows.Err(); err != nil { log.Fatal(err) } } ``` -------------------------------- ### SparseVector String Format Example Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/types.md Demonstrates the string representation for a SparseVector type, including indices and dimension. Note that indices are 1-based in this string format. ```text {1:1,3:2,5:3}/6 ``` -------------------------------- ### Unsupported Data Type Error Example Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/errors.md Demonstrates triggering an 'unsupported data type' error when attempting to scan non-byte slice or string data into a Vector type. ```Go var vec pgvector.Vector // Passing an integer err := vec.Scan(123) // Error: "unsupported data type: int" // Passing a float err := vec.Scan(1.5) // Error: "unsupported data type: float64" // Passing a struct type CustomType struct{} er := vec.Scan(CustomType{}) // Error: "unsupported data type: CustomType" ``` -------------------------------- ### Enable pgvector extension with pgx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Enable the pgvector extension in your PostgreSQL database using a pgx connection. This is a one-time setup per database. ```go _, err := conn.Exec(ctx, "CREATE EXTENSION IF NOT EXISTS vector") ``` -------------------------------- ### Example: Register pgvector Types with a pgx Connection Pool Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/pgx.md Shows how to automatically register pgvector types for all new connections established by a pgx connection pool using the AfterConnect callback. ```go config, err := pgx.ParseConfig("postgres://user:password@localhost/dbname") if err != nil { log.Fatal(err) } config.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error { return pgxvec.RegisterTypes(ctx, conn) } pool, err := pgx.NewConnPool(context.Background(), config) if err != nil { log.Fatal(err) } ``` -------------------------------- ### Query Items Ordered by Jaccard Distance Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/ent.md This example demonstrates how to query for items and order them by Jaccard distance using the `entvec.JaccardDistance` function. It limits the results to the top 5 nearest neighbors. ```go items, err := client.Item. Query(). Order(func(s *sql.Selector) { s.OrderExpr(entvec.JaccardDistance("embedding", pgvector.NewVector([]float32{1, 2, 3}))) }). Limit(5). All(ctx) ``` -------------------------------- ### Updating pgvector-go Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md Use 'go get -u' to update the main pgvector-go package and its optional submodules for pgx or Ent integration. ```bash go get -u github.com/pgvector/pgvector-go go get -u github.com/pgvector/pgvector-go/pgx # if using pgx go get -u github.com/pgvector/pgvector-go/ent # if using ent ``` -------------------------------- ### String Representation of HalfVector Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/halfvector.md Use the `String` method to get a human-readable string representation of the HalfVector in the format `[v1,v2,v3,...]`. This is helpful for debugging and logging. ```go vec := pgvector.NewHalfVector([]float32{1, 2, 3}) fmt.Println(vec.String()) // Output: [1,2,3] ``` -------------------------------- ### Get SparseVector String Representation Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/sparsevector.md The String() method returns a string representation of the sparse vector in the format {i1:v1,i2:v2,...}/dim. Indices are 1-based for PostgreSQL compatibility. ```go package main import ( "fmt" "github.com/pgvector/pgvector-go" ) func main() { vec := pgvector.NewSparseVector([]float32{1, 0, 2, 0, 3, 0}) fmt.Println(vec.String()) // Output: {1:1,3:2,5:3}/6 } ``` -------------------------------- ### Basic Operations with Bun Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Connect to the database, enable the vector extension, create a table, insert an item, and query nearest neighbors using Bun. ```go // Connect db := bun.NewDB(sql.Open("postgres", dsn)) // Enable extension db.Exec("CREATE EXTENSION IF NOT EXISTS vector") // Create table db.NewCreateTable().Model((*Item)(nil)).Exec(ctx) // Insert item := Item{ Embedding: pgvector.NewVector([]float32{1, 2, 3}), } db.NewInsert().Model(&item).Exec(ctx) // Query nearest neighbors var items []Item db.NewSelect(). Model(&items). OrderExpr("embedding <-> ?", pgvector.NewVector([]float32{1, 2, 3})). Limit(5). Scan(ctx) ``` -------------------------------- ### Create and Access Vector Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/README.md Demonstrates how to create a dense vector, access its components, and serialize it to a string representation. ```go package main import ( "github.com/pgvector/pgvector-go" ) func main() { // Create a vector vec := pgvector.NewVector([]float32{1, 2, 3}) // Access components slice := vec.Slice() // Serialize to string str := vec.String() // "[1,2,3]" } ``` -------------------------------- ### Get dimensions of a SparseVector Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Retrieve the total number of dimensions of a pgvector.SparseVector object. ```go dim := vec.Dimensions() ``` -------------------------------- ### Get slice from a vector Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Retrieve the underlying float32 slice from a pgvector.Vector object. ```go slice := vec.Slice() ``` -------------------------------- ### Basic Operations with go-pg Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Connect to the database, enable the vector extension, create a table, insert an item, and query nearest neighbors using go-pg. ```go // Connect db := pg.Connect(&pg.Options{ Addr: "localhost:5432", User: "user", Database: "dbname", }) // Enable extension db.Exec("CREATE EXTENSION IF NOT EXISTS vector") // Create table db.Model((*Item)(nil)).CreateTable(&orm.CreateTableOptions{}) // Insert item := Item{ Embedding: pgvector.NewVector([]float32{1, 2, 3}), } db.Model(&item).Insert() // Query nearest neighbors var items []Item db.Model(&items). OrderExpr("embedding <-> ?", pgvector.NewVector([]float32{1, 2, 3})). Limit(5). Select() ``` -------------------------------- ### Bulk Loading with SQL COPY Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Demonstrates the SQL COPY syntax for high-performance bulk inserts and how to prepare vector data in pgvector's text format. ```sql -- SQL COPY syntax COPY items (id, embedding) FROM STDIN ``` ```go // Prepare vector data in pgvector text format vec := pgvector.NewVector([]float32{1, 2, 3}) vecString := vec.String() // "[1,2,3]" ``` -------------------------------- ### Get non-zero values of a SparseVector Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Retrieve the values of the non-zero elements from a pgvector.SparseVector object. ```go values := vec.Values() ``` -------------------------------- ### Vector Type Initialization Patterns Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/types.md Demonstrates common patterns for initializing Vector, SparseVector, and HalfVector types using provided constructor functions. ```APIDOC ## Vector Type Initialization Patterns ### Creating Vectors from Dense Arrays ```go // From float32 slice vec := pgvector.NewVector([]float32{1.0, 2.0, 3.0}) // From float64 slice (requires conversion) float64vals := []float64{1.0, 2.0, 3.0} float32vals := make([]float32, len(float64vals)) for i, v := range float64vals { float32vals[i] = float32(v) } vec := pgvector.NewVector(float32vals) ``` ### Creating Sparse Vectors ```go // From dense array (non-zeros extracted automatically) densevec := []float32{1, 0, 2, 0, 3, 0} sparsevec := pgvector.NewSparseVector(densevec) // From map of indices to values elements := map[int32]float32{ 0: 1.0, 2: 2.0, 4: 3.0, } sparsevec := pgvector.NewSparseVectorFromMap(elements, 6) ``` ### Creating Half Vectors ```go // From float32 slice (stored as float16 in database) vec := pgvector.NewHalfVector([]float32{1.0, 2.0, 3.0}) ``` ``` -------------------------------- ### Get non-zero indices of a SparseVector Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Retrieve the indices of the non-zero elements from a pgvector.SparseVector object. ```go indices := vec.Indices() ``` -------------------------------- ### Get SparseVector Dimensions Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/sparsevector.md The Dimensions() method returns the total number of dimensions of the sparse vector. ```go vec := pgvector.NewSparseVector([]float32{1, 0, 2, 0, 3, 0}) dim := vec.Dimensions() // Returns 6 ``` -------------------------------- ### Enable vector extension with sqlx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Enable the vector extension in PostgreSQL using a sqlx database instance. ```go db.MustExec("CREATE EXTENSION IF NOT EXISTS vector") ``` -------------------------------- ### Migrating Vector Columns Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Shows the SQL commands to add a new vector column, copy data using the compatible Vector type, and then drop the old column during a database library migration. ```go // Create new column db.Exec("ALTER TABLE items ADD COLUMN new_embedding vector(3)") ``` ```go // Copy data (same format across all libraries) db.Exec("UPDATE items SET new_embedding = embedding") ``` ```go // Drop old column db.Exec("ALTER TABLE items DROP COLUMN embedding") ``` -------------------------------- ### Get SparseVector Indices Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/sparsevector.md The Indices() method returns a slice containing the 0-based indices of the non-zero elements in the sparse vector. ```go vec := pgvector.NewSparseVector([]float32{1, 0, 2, 0, 3, 0}) indices := vec.Indices() // Returns []int32{0, 2, 4} ``` -------------------------------- ### Get Vector String Representation Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Generates a string representation of a pgvector.Vector or pgvector.SparseVector. The sparse vector format includes 1-based indices. ```go vec := pgvector.NewVector([]float32{1, 2, 3}) str := vec.String() // "[1,2,3]" sparse := pgvector.NewSparseVector([]float32{1, 0, 2, 0, 3, 0}) str := sparse.String() // "{1:1,3:2,5:3}/6" (1-based indices) ``` -------------------------------- ### Query with Different Distance Metrics Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Demonstrates querying using various distance metrics available in pgvector: inner product (<#>), cosine (<=>), L1 (<+>), Hamming (<~>), and Jaccard (<%>). ```go // Inner product (maximize similarity) conn.Query(ctx, "... ORDER BY embedding <#> $1", queryVec) // Cosine distance conn.Query(ctx, "... ORDER BY embedding <=> $1", queryVec) // L1 distance (Manhattan) conn.Query(ctx, "... ORDER BY embedding <+> $1", queryVec) // Hamming distance (binary vectors) conn.Query(ctx, "... ORDER BY embedding <~> $1", queryVec) // Jaccard distance conn.Query(ctx, "... ORDER BY embedding <%> $1", queryVec) ``` -------------------------------- ### sqlx Basic Operations Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Perform basic database operations including connection, extension enabling, insertion, and nearest neighbor querying with sqlx. ```go // Connect db, err := sqlx.Open("postgres", dsn) // Enable extension db.MustExec("CREATE EXTENSION IF NOT EXISTS vector") // Insert item := Item{ Embedding: pgvector.NewVector([]float32{1, 2, 3}), } db.NamedExec(`INSERT INTO items (embedding) VALUES (:embedding)`, item) // Query nearest neighbors var items []Item db.Select(&items, "SELECT id, embedding FROM items ORDER BY embedding <-> $1 LIMIT 5", pgvector.NewVector([]float32{1, 2, 3})) ``` -------------------------------- ### Get SparseVector Values Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/sparsevector.md The Values() method returns a slice containing the values of the non-zero elements, corresponding to the indices returned by Indices(). ```go vec := pgvector.NewSparseVector([]float32{1, 0, 2, 0, 3, 0}) values := vec.Values() // Returns []float32{1, 2, 3} ``` -------------------------------- ### GORM Basic Operations Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Perform basic database operations including connection, extension enabling, table creation, insertion, and nearest neighbor querying with GORM. ```go // Connect db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{}) // Enable extension db.Exec("CREATE EXTENSION IF NOT EXISTS vector") // Create table db.AutoMigrate(&Item{}) // Insert item := Item{ Embedding: pgvector.NewVector([]float32{1, 2, 3}), } db.Create(&item) // Query nearest neighbors var items []Item db.Clauses(clause.OrderBy{ Expression: clause.Expr{ SQL: "embedding <-> ?", Vars: []interface{}{pgvector.NewVector([]float32{1, 2, 3})}, }, }).Limit(5).Find(&items) ``` -------------------------------- ### pgx Integration Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md Import the pgx submodule and register its types with a pgx connection for seamless integration. This requires the pgx library. ```go import ( "github.com/pgvector/pgvector-go" pgxvec "github.com/pgvector/pgvector-go/pgx" ) // Register types and use with pgx connection err := pgxvec.RegisterTypes(ctx, conn) ``` -------------------------------- ### Get Underlying Slice from HalfVector Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/halfvector.md The `Slice` method returns the original `[]float32` slice from a HalfVector. This is useful for accessing or modifying the vector's data directly. ```go vec := pgvector.NewHalfVector([]float32{1.0, 2.5, 3.5}) slice := vec.Slice() // []float32{1.0, 2.5, 3.5} ``` -------------------------------- ### pgx Integration for Vector Operations Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/README.md Shows how to register pgvector types with pgx, insert vectors into the database, and query for nearest neighbors using the L2 distance operator. ```go import ( "context" "github.com/jackc/pgx/v5" pgxvec "github.com/pgvector/pgvector-go/pgx" "github.com/pgvector/pgvector-go" ) conn, _ := pgx.Connect(context.Background(), "postgres://...") pgxvec.RegisterTypes(context.Background(), conn) // Insert vec := pgvector.NewVector([]float32{1, 2, 3}) conn.Exec(context.Background(), "INSERT INTO items (embedding) VALUES ($1)", vec) // Query nearest neighbors query := pgvector.NewVector([]float32{1, 2, 3}) rows, _ := conn.Query(context.Background(), "SELECT id, embedding FROM items ORDER BY embedding <-> $1 LIMIT 5", query) ``` -------------------------------- ### Enabling pgvector Extension and Registering Types Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/errors.md Provides the correct sequence to enable the pgvector extension in PostgreSQL and then register the pgvector types with pgx. ```Go _, err := conn.Exec(ctx, "CREATE EXTENSION IF NOT EXISTS vector") if err != nil { return err } er = pgxvec.RegisterTypes(ctx, conn) ``` -------------------------------- ### Enable pgvector Extension Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Enable the pgvector extension for your database connection. This is a prerequisite for all integrations. ```go // Most libraries db.Exec("CREATE EXTENSION IF NOT EXISTS vector") ``` -------------------------------- ### Get Underlying Slice from Vector Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/vector.md The Slice method returns the original float32 slice used to create the Vector. This is useful for direct manipulation or inspection of the vector's data. ```go vec := pgvector.NewVector([]float32{1.0, 2.5, 3.5}) slice := vec.Slice() // []float32{1.0, 2.5, 3.5} ``` -------------------------------- ### Test Organization Structure Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md The test directory contains specific test files for different integrations (pg, pgx, Bun, GORM, Ent) and vector types, along with Ent schema generation files. ```go test/ ├── pg_test.go - go-pg library tests ├── pgx_test.go - pgx library tests ├── bun_test.go - Bun ORM tests ├── gorm_test.go - GORM tests ├── ent_test.go - Ent ORM tests ├── sparsevec_test.go - SparseVector specific tests ├── halfvec_test.go - HalfVector specific tests ├── ent/ │ ├── generate.go - Ent schema generation │ └── schema/ │ └── item.go - Ent schema definition └── go.mod - Test module dependencies ``` -------------------------------- ### Root Module Imports (Standard Library) Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md The root module relies solely on Go's standard library packages for its core functionality, including database/sql, encoding, and string manipulation. ```go github.com/pgvector/pgvector-go ↓ database/sql (standard library) database/sql/driver (standard library) encoding/binary (standard library) encoding/json (standard library) ``` -------------------------------- ### Import pgvector-go and Ent extension Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Import the necessary packages for pgvector-go and its Ent extension. ```go import ( "github.com/pgvector/pgvector-go" entvec "github.com/pgvector/pgvector-go/ent" ) ``` -------------------------------- ### Basic Operations with Ent Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Create an item with an embedding and query for nearest neighbors using L2 distance with the Ent framework. ```go // Create _, err := client.Item. Create(). SetEmbedding(pgvector.NewVector([]float32{1, 2, 3})). Save(ctx) // Query with L2 distance items, err := client.Item. Query(). Order(func(s *sql.Selector) { s.OrderExpr(entvec.L2Distance("embedding", pgvector.NewVector([]float32{1, 2, 3}))) }). Limit(5). All(ctx) ``` -------------------------------- ### Create HNSW Index with Bun Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Define a hook to create an HNSW index on the embedding column after the table is created using Bun. ```go func (Item) AfterCreateTable(ctx context.Context, query *bun.CreateTableQuery) error { _, err := query.DB().NewCreateIndex(). Model((*Item)(nil)). Index("items_embedding_idx"). ColumnExpr("embedding vector_l2_ops"). Using("hnsw"). Exec(ctx) return err } ``` -------------------------------- ### Batch Insert Vectors using pgx Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Efficiently insert multiple vectors in a batch using pgx. Prepares a batch of INSERT statements and executes them together. ```go import "github.com/jackc/pgx/v5/pgconn" items := []struct { text string embedding pgvector.Vector }{ {"text1", pgvector.NewVector([]float32{1, 1, 1})}, {"text2", pgvector.NewVector([]float32{2, 2, 2})}, {"text3", pgvector.NewVector([]float32{3, 3, 3})} } batch := &pgx.Batch{} for _, item := range items { batch.Queue( "INSERT INTO items (text, embedding) VALUES ($1, $2)", item.text, item.embedding, ) } results := conn.SendBatch(ctx, batch) defer results.Close() for _, item := range items { _, err := results.Exec() if err != nil { return err } } ``` -------------------------------- ### Enable vector extension with GORM Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Enable the vector extension in PostgreSQL using a GORM database instance. ```go db.Exec("CREATE EXTENSION IF NOT EXISTS vector") ``` -------------------------------- ### GORM Import Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Import the pgvector-go library for use with GORM. ```go import "github.com/pgvector/pgvector-go" ``` -------------------------------- ### Create IVFFlat approximate index with pgx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Add an Inverted File Index (IVFFlat) to the 'embedding' column using L2 distance, with 100 lists. This is another method for accelerating similarity searches. ```go _ , err := conn.Exec(ctx, "CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100)") ``` -------------------------------- ### Enable vector extension in Ent Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Enable the vector extension in PostgreSQL using the Ent client. Requires the sql/execquery feature. ```go _ , err := client.ExecContext(ctx, "CREATE EXTENSION IF NOT EXISTS vector") ``` -------------------------------- ### Import pgx and pgvector-go pgx packages Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Import the required packages for using pgvector with pgx. Alias the pgx-specific module for clarity. ```go import ( "github.com/pgvector/pgvector-go" pgxvec "github.com/pgvector/pgvector-go/pgx" ) ``` -------------------------------- ### Value for Database Operations (HalfVector) Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/halfvector.md Implement the `driver.Valuer` interface using the `Value` method to serialize HalfVector data for database INSERT or UPDATE operations. This allows direct use with `database/sql`. ```go // Used automatically by database/sql vec := pgvector.NewHalfVector([]float32{1, 2, 3}) _, err := db.Exec("INSERT INTO items (embedding) VALUES ($1)", vec) if err != nil { log.Fatal(err) } ``` -------------------------------- ### GORM Index Creation Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Create an HNSW index on the embedding column using GORM. ```go db.Exec("CREATE INDEX ON items USING hnsw (embedding vector_l2_ops)") ``` -------------------------------- ### IVFFlat Index Creation Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Create an IVFFlat index on the embedding column as an alternative to HNSW, specifying the number of lists. ```go // IVFFlat index (alternative) db.Exec("CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100)") ``` -------------------------------- ### Insert a vector using sqlx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Insert a new item with a vector embedding into the database using sqlx NamedExec. ```go item := Item{ Embedding: pgvector.NewVector([]float32{1, 2, 3}), } _, err := db.NamedExec(`INSERT INTO items (embedding) VALUES (:embedding)`, item) ``` -------------------------------- ### Create HNSW approximate index with Bun library Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Implement the AfterCreateTable hook to automatically create a Hierarchical Navigable Small World (HNSW) index on the 'embedding' column using L2 distance when the table is created. ```go var _ bun.AfterCreateTableHook = (*Item)(nil) func (*Item) AfterCreateTable(ctx context.Context, query *bun.CreateTableQuery) error { _, err := query.DB().NewCreateIndex(). Model((*Item)(nil)). Index("items_embedding_idx"). ColumnExpr("embedding vector_l2_ops"). Using("hnsw"). Exec(ctx) return err } ``` -------------------------------- ### Create a New Vector Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/vector.md Use NewVector to create a Vector from a slice of float32 values. This is the primary way to instantiate a Vector object. ```go package main import ( "github.com/pgvector/pgvector-go" ) func main() { vec := pgvector.NewVector([]float32{1.0, 2.0, 3.0}) } ``` -------------------------------- ### Go Module Specification Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md The root module is specified as github.com/pgvector/pgvector-go and requires Go version 1.21 or higher. The go.mod file defines its dependencies. ```go // No external dependencies ``` -------------------------------- ### HNSW Index Creation Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/database-libraries.md Create an HNSW index on the embedding column for optimized nearest neighbor searches. ```go // HNSW index (default) db.Exec("CREATE INDEX ON items USING hnsw (embedding vector_l2_ops)") ``` -------------------------------- ### Define a vector column with sqlx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Define a struct with a vector field for sqlx. The type will be inferred by the driver. ```go type Item struct { Embedding pgvector.Vector } ``` -------------------------------- ### Root Module Functions Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/DOCUMENTATION_SUMMARY.txt These functions are available at the root module level for creating new vector instances. ```APIDOC ## Root Module Functions ### NewVector - **Description**: Creates a new Vector from a slice of float32. - **Signature**: `NewVector(vec []float32) Vector` ### NewHalfVector - **Description**: Creates a new HalfVector from a slice of float32. - **Signature**: `NewHalfVector(vec []float32) HalfVector` ### NewSparseVector - **Description**: Creates a new SparseVector from a slice of float32. - **Signature**: `NewSparseVector(vec []float32) SparseVector` ### NewSparseVectorFromMap - **Description**: Creates a new SparseVector from a map of elements and dimension. - **Signature**: `NewSparseVectorFromMap(elements map[int32]float32, dim int32) SparseVector` ``` -------------------------------- ### Insert a vector using Bun library Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Insert a new 'Item' record with a vector embedding into the database using the Bun ORM. The vector is created using pgvector.NewVector. ```go item := Item{ Embedding: pgvector.NewVector([]float32{1, 2, 3}), } _, err := db.NewInsert().Model(&item).Exec(ctx) ``` -------------------------------- ### Query nearest neighbors using L2 distance with Bun library Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Retrieve items ordered by proximity to a given vector using the L2 distance (<-> operator) with the Bun ORM. Limits the results to 5. ```go var items []Item err := db.NewSelect(). Model(&items). OrderExpr("embedding <-> ?", pgvector.NewVector([]float32{1, 2, 3})). Limit(5). Scan(ctx) ``` -------------------------------- ### Create HalfVector from Float32 Slice Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/types.md Initializes a HalfVector from a slice of float32 values, which will be stored as float16 in the database. ```go // From float32 slice (stored as float16 in database) vec := pgvector.NewHalfVector([]float32{1.0, 2.0, 3.0}) ``` -------------------------------- ### Implement driver.Valuer for Vector Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/vector.md Serializes vector values for database operations using the driver.Valuer interface. This is used automatically by the database/sql package for INSERT statements. ```go func (v Vector) Value() (driver.Value, error) ``` ```go // Used automatically by database/sql vec := pgvector.NewVector([]float32{1, 2, 3}) _, err := db.Exec("INSERT INTO items (embedding) VALUES ($1)", vec) if err != nil { log.Fatal(err) } ``` -------------------------------- ### Insert Single Vector using pgx Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Insert a single vector into a PostgreSQL table using the pgx driver. Requires context and a database connection. ```go import ( "context" "github.com/jackc/pgx/v5" "github.com/pgvector/pgvector-go" ) ctx := context.Background() conn, _ := pgx.Connect(ctx, "postgres://...") var id int64 embedding := pgvector.NewVector([]float32{1, 2, 3}) err := conn.QueryRow(ctx, "INSERT INTO items (text, embedding) VALUES ($1, $2) RETURNING id", "Hello world", embedding).Scan(&id) ``` -------------------------------- ### Query nearest neighbors using L2 distance with pg library Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Retrieve items ordered by proximity to a given vector using the L2 distance (<-> operator) with the pg library. Limits the results to 5. ```go var items []Item err := db.Model(&items). OrderExpr("embedding <-> ?", pgvector.NewVector([]float32{1, 2, 3})). Limit(5). Select() ``` -------------------------------- ### Scan HalfVector from Database Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/halfvector.md Implement the `sql.Scanner` interface using the `Scan` method to automatically read HalfVector data from database query results. It supports `[]byte` or `string` source types. ```go // Used automatically by database/sql var vec pgvector.HalfVector err := db.QueryRow("SELECT embedding FROM items WHERE id = 1").Scan(&vec) if err != nil { log.Fatal(err) } ``` -------------------------------- ### Insert Sparse Vectors Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Insert a sparse vector into the database. Requires the pgvector extension to be enabled. ```go vec := pgvector.NewSparseVector([]float32{1, 0, 2, 0, 3, 0}) err := conn.QueryRow(ctx, "INSERT INTO items (sparse_embedding) VALUES ($1) RETURNING id", vec).Scan(&id) ``` -------------------------------- ### pgx Submodule Dependencies Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md The pgx submodule has direct dependencies on github.com/jackc/pgx/v5 and github.com/x448/float16. ```go github.com/jackc/pgx/v5 (direct) github.com/x448/float16 (direct) ``` -------------------------------- ### Add an approximate index with sqlx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Create an approximate nearest neighbor index (HNSW or IVFFlat) on a vector column using sqlx. ```go db.MustExec("CREATE INDEX ON items USING hnsw (embedding vector_l2_ops)") // or db.MustExec("CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100)") ``` -------------------------------- ### Root Module Imports Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md Import the main pgvector-go package to use Vector, HalfVector, and SparseVector types. This requires no external dependencies beyond the Go standard library. ```go import "github.com/pgvector/pgvector-go" // Use Vector, HalfVector, SparseVector vec := pgvector.NewVector([]float32{1, 2, 3}) ``` -------------------------------- ### Query nearest neighbors using L2 distance with pgx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Retrieve item IDs from the 'items' table, ordered by their proximity to a given vector using the L2 distance (<-> operator). Results are limited to 5. ```go rows, err := conn.Query(ctx, "SELECT id FROM items ORDER BY embedding <-> $1 LIMIT 5", pgvector.NewVector([]float32{1, 2, 3})) ``` -------------------------------- ### Enable pgvector extension with pg library Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Enable the pgvector extension in your PostgreSQL database using a pg library connection. This is a prerequisite for using vector types. ```go _ , err := db.Exec("CREATE EXTENSION IF NOT EXISTS vector") ``` -------------------------------- ### Create Vector from Float64 Slice Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/types.md Initializes a Vector type from a slice of float64 values, requiring conversion to float32. ```go // From float64 slice (requires conversion) float64vals := []float64{1.0, 2.0, 3.0} float32vals := make([]float32, len(float64vals)) for i, v := range float64vals { float32vals[i] = float32(v) } vec := pgvector.NewVector(float32vals) ``` -------------------------------- ### Handling Database Scan Errors including sql.ErrNoRows Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/errors.md Shows how to handle errors during a database query scan, specifically checking for `sql.ErrNoRows` and other general scan errors. ```Go var vec pgvector.Vector er := db.QueryRow("SELECT embedding FROM items WHERE id = 1").Scan(&vec) if err != nil { if err == sql.ErrNoRows { log.Println("Item not found") } else { log.Printf("Scan failed: %v", err) } return err } ``` -------------------------------- ### Create New HalfVector Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/api-reference/halfvector.md Use `NewHalfVector` to create a new HalfVector from a slice of float32 values. This is the primary way to initialize a HalfVector. ```go package main import ( "github.com/pgvector/pgvector-go" ) func main() { vec := pgvector.NewHalfVector([]float32{1.0, 2.0, 3.0}) } ``` -------------------------------- ### Validating Vector Format Before Parsing Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/errors.md Demonstrates a 'safe' parsing function that first validates the string format (prefix and suffix) before attempting to parse it as a vector, returning an error for invalid formats. ```Go // Check format before parsing func safeParseVector(s string) (pgvector.Vector, error) { if !strings.HasPrefix(s, "[") || !strings.HasSuffix(s, "]") { return pgvector.Vector{}, fmt.Errorf("invalid vector format") } var vec pgvector.Vector return vec, vec.Parse(s) } ``` -------------------------------- ### Ent Integration Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md Import the Ent submodule to utilize distance functions within Ent ORM queries. This requires the Ent library. ```go import ( "github.com/pgvector/pgvector-go" entvec "github.com/pgvector/pgvector-go/ent" ) // Use distance functions in Ent queries entvec.L2Distance("embedding", vec) ``` -------------------------------- ### pgx Submodule Integration Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/DOCUMENTATION_SUMMARY.txt Functions and types for integrating pgvector-go with the pgx driver. ```APIDOC ## pgx Submodule Integration ### RegisterTypes - **Description**: Registers pgvector types with a pgx connection. - **Signature**: `RegisterTypes(ctx context.Context, conn *pgx.Conn) error` ### VectorCodec, HalfVectorCodec, SparseVectorCodec - **Description**: Codecs for handling vector types with pgx. ``` -------------------------------- ### Update Multiple Vectors in Batch Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Efficiently update multiple vectors in the database using pgx's batch API. This reduces network overhead compared to individual updates. ```go batch := &pgx.Batch{} updates := []struct { id int64 embedding pgvector.Vector }{ {1, pgvector.NewVector([]float32{1, 2, 3})}, {2, pgvector.NewVector([]float32{4, 5, 6})}, } for _, u := range updates { batch.Queue("UPDATE items SET embedding = $1 WHERE id = $2", u.embedding, u.id) } results := conn.SendBatch(ctx, batch) for range updates { _, err := results.Exec() if err != nil { return err } } results.Close() ``` -------------------------------- ### Ent ORM Integration for Vector Queries Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/README.md Illustrates how to use the Ent ORM with pgvector-go to query for items ordered by L2 distance to a specified vector. ```go import entvec "github.com/pgvector/pgvector-go/ent" items, _ := client.Item. Query(). Order(func(s *sql.Selector) { s.OrderExpr(entvec.L2Distance("embedding", pgvector.NewVector([]float32{1, 2, 3}))) }). Limit(5). All(ctx) ``` -------------------------------- ### Half Vector Precision Trade-offs Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Illustrates the precision trade-off when using HalfVectors. While offering smaller storage, HalfVectors use float16 which results in some precision loss upon retrieval. ```go // Full precision vector vec := pgvector.NewVector([]float32{1.23456789, 2.87654321, 3.14159265}) // Half precision (reduced precision, smaller storage) hvec := pgvector.NewHalfVector([]float32{1.23456789, 2.87654321, 3.14159265}) // Note: stored as float16, some precision loss upon retrieval ``` -------------------------------- ### Query nearest neighbors with sqlx Source: https://github.com/pgvector/pgvector-go/blob/master/README.md Query for items nearest to a given vector using sqlx. The query specifies the distance metric and target vector. ```go var items []Item db.Select(&items, "SELECT * FROM items ORDER BY embedding <-> $1 LIMIT 5", pgvector.NewVector([]float32{1, 1, 1})) ``` -------------------------------- ### Create Sparse Vector from Map Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/README.md Create a sparse vector from a map of indices to values. Specify the total dimension of the vector. ```go elements := map[int32]float32{ 0: 1, 2: 2, 4: 3, } vec := pgvector.NewSparseVectorFromMap(elements, 6) ``` -------------------------------- ### Create Vector from Slice Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/README.md Create a new vector from a slice of float32 values. This is a fundamental operation for initializing vector data. ```go vec := pgvector.NewVector([]float32{1, 2, 3}) ``` -------------------------------- ### Updating pgx Submodule Dependencies Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/module-structure.md If you are using the pgx submodule, update its direct dependencies, github.com/jackc/pgx/v5 and github.com/x448/float16. ```bash go get -u github.com/jackc/pgx/v5 go get -u github.com/x448/float16 ``` -------------------------------- ### Batch Insert Operations Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Use the pgx batch API for inserting multiple items to improve performance by reducing network round trips. This is more efficient than executing individual INSERT statements. ```go // Instead of individual inserts // for _, item := range items { // conn.Exec(ctx, "INSERT INTO items (...) VALUES (...)", ...) // } // Use batch API batch := &pgx.Batch{} for _, item := range items { batch.Queue("INSERT INTO items (...) VALUES (...)", ...) } results := conn.SendBatch(ctx, batch) results.Close() ``` -------------------------------- ### Query with Sparse Vectors Source: https://github.com/pgvector/pgvector-go/blob/master/_autodocs/usage-patterns.md Query the database using a sparse vector to find similar items. The distance is calculated using the '<->' operator. ```go queryVec := pgvector.NewSparseVector([]float32{1, 0, 2, 0, 3, 0}) rows, _ := conn.Query(ctx, "SELECT id, text, sparse_embedding <-> $1 AS distance FROM items ORDER BY distance LIMIT 5", queryVec) ```