# Milvus Vector Database ## Introduction Milvus is a high-performance, open-source vector database built for scale, designed to organize and search through large amounts of unstructured data such as text, images, and multi-modal information. It is implemented in Go and C++ with CPU/GPU instruction-level optimizations for optimal vector search performance. Milvus powers AI applications requiring similarity search over billions of vectors, including Retrieval-Augmented Generation (RAG), recommendation systems, image search, and semantic search applications. The system features a fully-distributed microservices architecture that can scale horizontally on Kubernetes, handling tens of thousands of concurrent search queries while maintaining real-time data freshness through streaming updates. Milvus supports multiple deployment modes including distributed clusters for production scale, standalone mode for Docker-based deployments, and Milvus Lite for lightweight Python applications. It integrates seamlessly with popular AI frameworks like LangChain, LlamaIndex, and HuggingFace, and provides comprehensive SDKs in Python, Go, Java, and Node.js. ## APIs and Key Functions ### Connect to Milvus Establish connection to Milvus server using the Go client SDK. ```go package main import ( "context" "fmt" "github.com/milvus-io/milvus/client/v2/milvusclient" ) func main() { ctx := context.Background() // Connect to Milvus server client, err := milvusclient.New(ctx, &milvusclient.ClientConfig{ Address: "127.0.0.1:19530", APIKey: "root:Milvus", // username:password format }) if err != nil { panic(fmt.Sprintf("failed to connect: %v", err)) } defer client.Close(ctx) fmt.Println("Successfully connected to Milvus") } ``` ### Create Collection with Schema Define and create a collection with custom schema including vector and scalar fields. ```go import ( "github.com/milvus-io/milvus/client/v2/entity" "github.com/milvus-io/milvus/client/v2/index" "github.com/milvus-io/milvus/client/v2/milvusclient" ) func createCollection(ctx context.Context, client milvusclient.Client) error { // Define schema with multiple fields schema := entity.NewSchema(). WithName("product_search"). WithDescription("Product catalog with embeddings"). WithDynamicFieldEnabled(true). WithField(entity.NewField(). WithName("product_id"). WithDataType(entity.FieldTypeInt64). WithIsPrimaryKey(true). WithIsAutoID(true)). WithField(entity.NewField(). WithName("product_name"). WithDataType(entity.FieldTypeVarChar). WithMaxLength(256)). WithField(entity.NewField(). WithName("price"). WithDataType(entity.FieldTypeFloat)). WithField(entity.NewField(). WithName("embedding"). WithDataType(entity.FieldTypeFloatVector). WithDim(768)) // Create index configuration indexOption := milvusclient.NewCreateIndexOption( "product_search", "embedding", index.NewHNSWIndex(entity.COSINE, 32, 128), ) // Create collection with auto-index err := client.CreateCollection(ctx, milvusclient.NewCreateCollectionOption("product_search", schema). WithIndexOptions(indexOption)) if err != nil { return fmt.Errorf("failed to create collection: %w", err) } fmt.Println("Collection created successfully") return nil } ``` ### Insert Data into Collection Insert vector embeddings along with scalar data into a collection. ```go func insertData(ctx context.Context, client milvusclient.Client) error { // Prepare data columns productNames := []string{ "Laptop Pro 15", "Wireless Mouse", "USB-C Cable", "Monitor 27 inch", } prices := []float32{1299.99, 29.99, 12.99, 449.99} // Generate or load vector embeddings (768 dimensions) embeddings := make([][]float32, 4) for i := range embeddings { embeddings[i] = make([]float32, 768) // Fill with actual embedding values for j := range embeddings[i] { embeddings[i][j] = float32(i+j) * 0.001 } } // Insert data insertResult, err := client.Insert(ctx, milvusclient.NewColumnBasedInsertOption("product_search"). WithVarcharColumn("product_name", productNames). WithFloatColumn("price", prices). WithFloatVectorColumn("embedding", 768, embeddings)) if err != nil { return fmt.Errorf("failed to insert data: %w", err) } fmt.Printf("Inserted %d records\n", insertResult.InsertCount) fmt.Printf("Insert IDs: %v\n", insertResult.IDs) return nil } ``` ### Vector Search Perform similarity search to find the most similar vectors in the collection. ```go func searchVectors(ctx context.Context, client milvusclient.Client) error { // Load collection into memory first loadTask, err := client.LoadCollection(ctx, milvusclient.NewLoadCollectionOption("product_search")) if err != nil { return fmt.Errorf("failed to load collection: %w", err) } // Wait for loading to complete err = loadTask.Await(ctx) if err != nil { return fmt.Errorf("failed to wait for load: %w", err) } // Prepare query vector (768 dimensions) queryVector := make([]float32, 768) for i := range queryVector { queryVector[i] = float32(i) * 0.001 } // Perform search searchResults, err := client.Search(ctx, milvusclient.NewSearchOption( "product_search", 10, // top 10 results []entity.Vector{entity.FloatVector(queryVector)}, ). WithOutputFields("product_name", "price"). WithFilter("price < 1000"). // metadata filtering WithConsistencyLevel(entity.ClStrong)) if err != nil { return fmt.Errorf("failed to search: %w", err) } // Process results for _, resultSet := range searchResults { fmt.Printf("\nFound %d results:\n", resultSet.ResultCount) for i := 0; i < resultSet.ResultCount; i++ { id := resultSet.IDs.GetAsInt64(i) score := resultSet.Scores[i] // Access output fields nameCol, _ := resultSet.Fields.GetColumn("product_name") priceCol, _ := resultSet.Fields.GetColumn("price") productName := nameCol.Get(i) price := priceCol.Get(i) fmt.Printf(" Rank %d: ID=%d, Score=%.4f, Product=%v, Price=$%.2f\n", i+1, id, score, productName, price) } } return nil } ``` ### Query with Filtering Execute filtered queries to retrieve specific records based on scalar field conditions. ```go func queryWithFilter(ctx context.Context, client milvusclient.Client) error { // Query by filter expression queryResult, err := client.Query(ctx, milvusclient.NewQueryOption("product_search"). WithFilter("price >= 100 and price <= 500"). WithOutputFields("product_id", "product_name", "price"). WithLimit(100)) if err != nil { return fmt.Errorf("failed to query: %w", err) } fmt.Printf("Query returned %d results:\n", queryResult.ResultCount) // Access result columns idCol, _ := queryResult.Fields.GetColumn("product_id") nameCol, _ := queryResult.Fields.GetColumn("product_name") priceCol, _ := queryResult.Fields.GetColumn("price") for i := 0; i < queryResult.ResultCount; i++ { fmt.Printf(" Product: %v, Price: $%.2f (ID: %v)\n", nameCol.Get(i), priceCol.Get(i), idCol.Get(i)) } return nil } ``` ### Get Records by IDs Retrieve specific records using their primary key values. ```go func getByIDs(ctx context.Context, client milvusclient.Client) error { // Get specific records by primary key IDs ids := []int64{1, 2, 3, 5, 8} result, err := client.Get(ctx, milvusclient.NewQueryOption("product_search"). WithIDs(column.NewColumnInt64("product_id", ids)). WithOutputFields("product_id", "product_name", "price", "embedding")) if err != nil { return fmt.Errorf("failed to get records: %w", err) } fmt.Printf("Retrieved %d records:\n", result.ResultCount) nameCol, _ := result.Fields.GetColumn("product_name") priceCol, _ := result.Fields.GetColumn("price") for i := 0; i < result.ResultCount; i++ { fmt.Printf(" ID %d: %v - $%.2f\n", ids[i], nameCol.Get(i), priceCol.Get(i)) } return nil } ``` ### Hybrid Search with Multiple Vectors Perform hybrid search combining multiple vector fields with reranking. ```go func hybridSearch(ctx context.Context, client milvusclient.Client) error { // Prepare query vectors for different modalities denseVector := make([]float32, 768) for i := range denseVector { denseVector[i] = float32(i) * 0.001 } // Sparse vector for text search (BM25) sparseVector := entity.SparseEmbedding{ {Index: 100, Value: 0.8}, {Index: 250, Value: 0.6}, {Index: 500, Value: 0.4}, } // Perform hybrid search with RRF (Reciprocal Rank Fusion) reranking results, err := client.HybridSearch(ctx, milvusclient.NewHybridSearchOption( "multimodal_collection", 10, // top K results after reranking milvusclient.NewAnnRequest( "dense_vector", entity.FloatVector(denseVector), entity.COSINE, 20, // retrieve top 20 before reranking ), milvusclient.NewAnnRequest( "sparse_vector", sparseVector, entity.BM25, 20, ), ). WithReranker(milvusclient.NewRRFReranker()). WithOutputFields("title", "content")) if err != nil { return fmt.Errorf("failed to hybrid search: %w", err) } // Process hybrid search results for _, resultSet := range results { fmt.Printf("\nHybrid search found %d results:\n", resultSet.ResultCount) titleCol, _ := resultSet.Fields.GetColumn("title") for i := 0; i < resultSet.ResultCount; i++ { fmt.Printf(" Rank %d: Score=%.4f, Title=%v\n", i+1, resultSet.Scores[i], titleCol.Get(i)) } } return nil } ``` ### Create and Manage Indexes Build vector indexes to optimize search performance. ```go func manageIndexes(ctx context.Context, client milvusclient.Client) error { collectionName := "product_search" // Create HNSW index for approximate nearest neighbor search hnswIndex := index.NewHNSWIndex( entity.COSINE, // metric type 32, // M parameter (max connections) 128, // efConstruction parameter ) indexTask, err := client.CreateIndex(ctx, milvusclient.NewCreateIndexOption( collectionName, "embedding", hnswIndex, )) if err != nil { return fmt.Errorf("failed to create index: %w", err) } // Wait for index building to complete fmt.Println("Building index...") err = indexTask.Await(ctx) if err != nil { return fmt.Errorf("index build failed: %w", err) } fmt.Println("Index built successfully") // Describe index indexInfo, err := client.DescribeIndex(ctx, milvusclient.NewDescribeIndexOption(collectionName, "embedding")) if err != nil { return fmt.Errorf("failed to describe index: %w", err) } fmt.Printf("Index Info:\n") fmt.Printf(" Name: %s\n", indexInfo.Name) fmt.Printf(" Type: %s\n", indexInfo.IndexType) fmt.Printf(" Metric: %s\n", indexInfo.Params["metric_type"]) // List all indexes indexes, err := client.ListIndexes(ctx, milvusclient.NewListIndexOption(collectionName)) if err != nil { return fmt.Errorf("failed to list indexes: %w", err) } fmt.Printf("Collection has %d indexes\n", len(indexes)) return nil } ``` ### Upsert and Delete Operations Update existing records or delete records by ID or filter expression. ```go func upsertAndDelete(ctx context.Context, client milvusclient.Client) error { // Upsert: insert new or update existing records newPrices := []float32{999.99, 24.99} newNames := []string{"Laptop Pro 15 Updated", "Wireless Mouse V2"} newEmbeddings := make([][]float32, 2) for i := range newEmbeddings { newEmbeddings[i] = make([]float32, 768) // Fill with updated embeddings } upsertResult, err := client.Upsert(ctx, milvusclient.NewColumnBasedInsertOption("product_search"). WithVarcharColumn("product_name", newNames). WithFloatColumn("price", newPrices). WithFloatVectorColumn("embedding", 768, newEmbeddings)) if err != nil { return fmt.Errorf("failed to upsert: %w", err) } fmt.Printf("Upserted %d records\n", upsertResult.UpsertCount) // Delete by IDs deleteByIDs, err := client.Delete(ctx, milvusclient.NewDeleteOption("product_search"). WithInt64IDs("product_id", []int64{100, 101, 102})) if err != nil { return fmt.Errorf("failed to delete by IDs: %w", err) } fmt.Printf("Deleted %d records by ID\n", deleteByIDs.DeleteCount) // Delete by filter expression deleteByFilter, err := client.Delete(ctx, milvusclient.NewDeleteOption("product_search"). WithFilter("price > 2000")) if err != nil { return fmt.Errorf("failed to delete by filter: %w", err) } fmt.Printf("Deleted %d records by filter\n", deleteByFilter.DeleteCount) return nil } ``` ### Database Management Create and manage multiple databases for multi-tenancy. ```go func manageDatabases(ctx context.Context, client milvusclient.Client) error { // Create a new database err := client.CreateDatabase(ctx, milvusclient.NewCreateDatabaseOption("product_db")) if err != nil { return fmt.Errorf("failed to create database: %w", err) } fmt.Println("Database created") // List all databases databases, err := client.ListDatabases(ctx) if err != nil { return fmt.Errorf("failed to list databases: %w", err) } fmt.Println("Available databases:") for _, db := range databases { fmt.Printf(" - %s\n", db.Name) } // Switch to a specific database err = client.UsingDatabase(ctx, "product_db") if err != nil { return fmt.Errorf("failed to switch database: %w", err) } fmt.Println("Switched to product_db") // All subsequent operations will use the selected database return nil } ``` ### Partition Management Organize data into partitions for improved query performance and data management. ```go func managePartitions(ctx context.Context, client milvusclient.Client) error { collectionName := "product_search" // Create partitions partitions := []string{"electronics", "furniture", "clothing"} for _, partition := range partitions { err := client.CreatePartition(ctx, milvusclient.NewCreatePartitionOption(collectionName, partition)) if err != nil { return fmt.Errorf("failed to create partition %s: %w", partition, err) } } fmt.Printf("Created %d partitions\n", len(partitions)) // List all partitions partitionList, err := client.ListPartitions(ctx, milvusclient.NewListPartitionOption(collectionName)) if err != nil { return fmt.Errorf("failed to list partitions: %w", err) } fmt.Println("Partitions:") for _, p := range partitionList { fmt.Printf(" - %s\n", p) } // Load specific partitions loadTask, err := client.LoadPartitions(ctx, milvusclient.NewLoadPartitionsOption( collectionName, []string{"electronics", "furniture"})) if err != nil { return fmt.Errorf("failed to load partitions: %w", err) } loadTask.Await(ctx) // Search in specific partitions queryVector := make([]float32, 768) results, err := client.Search(ctx, milvusclient.NewSearchOption( collectionName, 10, []entity.Vector{entity.FloatVector(queryVector)}, ). WithPartitions([]string{"electronics"})) if err != nil { return fmt.Errorf("failed to search in partition: %w", err) } fmt.Printf("Search in electronics partition found %d results\n", results[0].ResultCount) return nil } ``` ### User and Role Management (RBAC) Implement role-based access control for secure multi-user environments. ```go func setupRBAC(ctx context.Context, client milvusclient.Client) error { // Create a new user err := client.CreateUser(ctx, milvusclient.NewCreateUserOption("analyst_user", "SecurePass123!")) if err != nil { return fmt.Errorf("failed to create user: %w", err) } fmt.Println("User created") // Create a custom role err = client.CreateRole(ctx, milvusclient.NewCreateRoleOption("analyst_role")) if err != nil { return fmt.Errorf("failed to create role: %w", err) } fmt.Println("Role created") // Grant privileges to role err = client.Grant(ctx, milvusclient.NewGrantOption( "analyst_role", "product_search", // collection name "Search", // privilege )) if err != nil { return fmt.Errorf("failed to grant privilege: %w", err) } err = client.Grant(ctx, milvusclient.NewGrantOption( "analyst_role", "product_search", "Query", )) if err != nil { return fmt.Errorf("failed to grant query privilege: %w", err) } // Assign role to user err = client.GrantRole(ctx, milvusclient.NewGrantRoleOption("analyst_user", "analyst_role")) if err != nil { return fmt.Errorf("failed to grant role to user: %w", err) } fmt.Println("Role assigned to user") // List user's roles userInfo, err := client.DescribeUser(ctx, milvusclient.NewDescribeUserOption("analyst_user")) if err != nil { return fmt.Errorf("failed to describe user: %w", err) } fmt.Printf("User '%s' has roles: %v\n", userInfo.Name, userInfo.Roles) return nil } ``` ### Collection Statistics and Metadata Retrieve collection metadata, statistics, and operational information. ```go func getCollectionInfo(ctx context.Context, client milvusclient.Client) error { collectionName := "product_search" // Check if collection exists exists, err := client.HasCollection(ctx, milvusclient.NewHasCollectionOption(collectionName)) if err != nil { return fmt.Errorf("failed to check collection: %w", err) } if !exists { fmt.Println("Collection does not exist") return nil } // Describe collection (get schema and details) collInfo, err := client.DescribeCollection(ctx, milvusclient.NewDescribeCollectionOption(collectionName)) if err != nil { return fmt.Errorf("failed to describe collection: %w", err) } fmt.Printf("Collection: %s\n", collInfo.Name) fmt.Printf(" Description: %s\n", collInfo.Schema.Description) fmt.Printf(" Shard Number: %d\n", collInfo.ShardNum) fmt.Printf(" Consistency Level: %v\n", collInfo.ConsistencyLevel) fmt.Println(" Fields:") for _, field := range collInfo.Schema.Fields { fmt.Printf(" - %s (%s)", field.Name, field.DataType) if field.IsPrimaryKey { fmt.Print(" [PRIMARY KEY]") } if field.DataType == entity.FieldTypeFloatVector { fmt.Printf(" [DIM=%d]", field.Dim) } fmt.Println() } // Get collection statistics stats, err := client.GetCollectionStats(ctx, milvusclient.NewGetCollectionStatisticsOption(collectionName)) if err != nil { return fmt.Errorf("failed to get stats: %w", err) } fmt.Printf("\nCollection Statistics:\n") fmt.Printf(" Row Count: %d\n", stats.RowCount) // List all collections collections, err := client.ListCollections(ctx) if err != nil { return fmt.Errorf("failed to list collections: %w", err) } fmt.Printf("\nTotal collections: %d\n", len(collections)) for _, coll := range collections { fmt.Printf(" - %s\n", coll.Name) } return nil } ``` ### Resource Group Management Manage compute resources and isolate workloads using resource groups. ```go func manageResourceGroups(ctx context.Context, client milvusclient.Client) error { // Create resource groups for workload isolation err := client.CreateResourceGroup(ctx, milvusclient.NewCreateResourceGroupOption("rg_high_priority")) if err != nil { return fmt.Errorf("failed to create resource group: %w", err) } err = client.CreateResourceGroup(ctx, milvusclient.NewCreateResourceGroupOption("rg_low_priority")) if err != nil { return fmt.Errorf("failed to create resource group: %w", err) } fmt.Println("Resource groups created") // List resource groups groups, err := client.ListResourceGroups(ctx) if err != nil { return fmt.Errorf("failed to list resource groups: %w", err) } fmt.Println("Available resource groups:") for _, group := range groups { fmt.Printf(" - %s\n", group) } // Describe resource group rgInfo, err := client.DescribeResourceGroup(ctx, milvusclient.NewDescribeResourceGroupOption("rg_high_priority")) if err != nil { return fmt.Errorf("failed to describe resource group: %w", err) } fmt.Printf("\nResource Group: %s\n", rgInfo.Name) fmt.Printf(" Capacity: %d\n", rgInfo.Capacity) fmt.Printf(" Available nodes: %d\n", rgInfo.NumAvailableNode) // Transfer query nodes between resource groups err = client.TransferNode(ctx, milvusclient.NewTransferNodeOption( "rg_low_priority", // source "rg_high_priority", // target 1, // number of nodes )) if err != nil { return fmt.Errorf("failed to transfer node: %w", err) } fmt.Println("Node transferred successfully") return nil } ``` ### Complete RAG Application Example End-to-end example of building a Retrieval-Augmented Generation application. ```go package main import ( "context" "fmt" "github.com/milvus-io/milvus/client/v2/entity" "github.com/milvus-io/milvus/client/v2/index" "github.com/milvus-io/milvus/client/v2/milvusclient" ) type Document struct { ID int64 Text string Source string Vector []float32 } func buildRAGApplication() error { ctx := context.Background() // 1. Connect to Milvus client, err := milvusclient.New(ctx, &milvusclient.ClientConfig{ Address: "localhost:19530", }) if err != nil { return fmt.Errorf("connection failed: %w", err) } defer client.Close(ctx) collectionName := "knowledge_base" // 2. Create collection for document embeddings schema := entity.NewSchema(). WithName(collectionName). WithDynamicFieldEnabled(true). WithField(entity.NewField(). WithName("doc_id"). WithDataType(entity.FieldTypeInt64). WithIsPrimaryKey(true). WithIsAutoID(true)). WithField(entity.NewField(). WithName("text"). WithDataType(entity.FieldTypeVarChar). WithMaxLength(65535)). WithField(entity.NewField(). WithName("source"). WithDataType(entity.FieldTypeVarChar). WithMaxLength(512)). WithField(entity.NewField(). WithName("embedding"). WithDataType(entity.FieldTypeFloatVector). WithDim(1536)) // OpenAI embedding dimension idx := index.NewAutoIndex(entity.COSINE) err = client.CreateCollection(ctx, milvusclient.NewCreateCollectionOption(collectionName, schema). WithIndexOptions( milvusclient.NewCreateIndexOption(collectionName, "embedding", idx))) if err != nil { return fmt.Errorf("failed to create collection: %w", err) } // 3. Load and insert documents documents := []Document{ { Text: "Milvus is a vector database for AI applications", Source: "docs/intro.md", Vector: generateEmbedding("Milvus is a vector database..."), }, { Text: "It supports billion-scale similarity search", Source: "docs/features.md", Vector: generateEmbedding("It supports billion-scale..."), }, // ... more documents } texts := make([]string, len(documents)) sources := make([]string, len(documents)) vectors := make([][]float32, len(documents)) for i, doc := range documents { texts[i] = doc.Text sources[i] = doc.Source vectors[i] = doc.Vector } _, err = client.Insert(ctx, milvusclient.NewColumnBasedInsertOption(collectionName). WithVarcharColumn("text", texts). WithVarcharColumn("source", sources). WithFloatVectorColumn("embedding", 1536, vectors)) if err != nil { return fmt.Errorf("failed to insert documents: %w", err) } // 4. Load collection for search task, _ := client.LoadCollection(ctx, milvusclient.NewLoadCollectionOption(collectionName)) task.Await(ctx) // 5. Perform RAG retrieval query := "What is Milvus used for?" queryVector := generateEmbedding(query) searchResults, err := client.Search(ctx, milvusclient.NewSearchOption( collectionName, 5, // retrieve top 5 most relevant documents []entity.Vector{entity.FloatVector(queryVector)}, ). WithOutputFields("text", "source"). WithConsistencyLevel(entity.ClStrong)) if err != nil { return fmt.Errorf("search failed: %w", err) } // 6. Build context from retrieved documents context := "" fmt.Println("\nRetrieved Documents:") for i := 0; i < searchResults[0].ResultCount; i++ { textCol, _ := searchResults[0].Fields.GetColumn("text") sourceCol, _ := searchResults[0].Fields.GetColumn("source") text := textCol.Get(i).(string) source := sourceCol.Get(i).(string) score := searchResults[0].Scores[i] fmt.Printf("%d. [Score: %.4f] %s\n Source: %s\n\n", i+1, score, text, source) context += text + "\n" } // 7. Generate answer using LLM with retrieved context prompt := fmt.Sprintf(`Answer the question based on the context below. Context: %s Question: %s Answer:`, context, query) // Send prompt to LLM (e.g., OpenAI, local model) // answer := callLLM(prompt) fmt.Printf("Prompt sent to LLM:\n%s\n", prompt) return nil } // Placeholder for embedding generation (use actual embedding model) func generateEmbedding(text string) []float32 { embedding := make([]float32, 1536) for i := range embedding { embedding[i] = float32(i) * 0.0001 } return embedding } func main() { if err := buildRAGApplication(); err != nil { panic(err) } } ``` ## Summary Milvus provides a comprehensive set of APIs for building production-scale AI applications that require semantic search, similarity matching, and vector storage capabilities. The primary use cases include Retrieval-Augmented Generation (RAG) systems that enhance Large Language Models with domain-specific knowledge, recommendation engines that match users with relevant items based on behavioral embeddings, semantic search applications for text, images, and multi-modal content, and similarity detection systems for fraud detection, duplicate removal, and content moderation. The Go client SDK offers a clean, type-safe interface with full support for all Milvus operations including collection management, CRUD operations, vector search with metadata filtering, hybrid search combining multiple vectors, index optimization, multi-tenancy through databases and partitions, and enterprise-grade security via RBAC. Integration patterns typically involve embedding generation using models like OpenAI, Sentence Transformers, or CLIP, followed by vector storage in Milvus collections with appropriate indexes (HNSW for high recall, IVF for large-scale deployment), and search operations with optional metadata filtering and reranking. Milvus seamlessly integrates with popular frameworks including LangChain, LlamaIndex, Haystack, and cloud platforms through S3/Azure/GCS object storage backends, making it an ideal foundation for AI applications requiring efficient vector similarity search at scale.