### Chroma Core Interface Example (Python) Source: https://github.com/philippgille/chromem-go/blob/main/README.md Demonstrates the core interface of the Chroma vector database, including client setup, collection creation, and adding documents with metadata and IDs. This serves as a reference for the API design of chromem-go. ```python import chromadb # setup Chroma in-memory, for easy prototyping. Can add persistence easily! client = chromadb.Client() # Create collection. get_collection, get_or_create_collection, delete_collection also available! collection = client.create_collection("all-my-documents") # Add docs to the collection. Can also update and delete. Row-based API coming soon! collection.add( documents=["This is document1", "This is document2"], # we handle tokenization, embedding, and indexing automatically. You can skip that and add your own embeddings as well metadatas=[{"source": "notion"}, {"source": "google-docs"}], # filter on these! ids=["doc1", "doc2"], # unique for each doc ) ``` -------------------------------- ### Compare Benchmark Results using benchstat Source: https://github.com/philippgille/chromem-go/blob/main/README.md Instructions for comparing benchmark results using the `benchstat` tool. This involves installing the tool and then running it with two benchmark output files. ```bash # Install benchstat: go install golang.org/x/perf/cmd/benchstat@latest # Compare two benchmark results: benchstat before.out after.out ``` -------------------------------- ### Install chromem-go Source: https://github.com/philippgille/chromem-go/blob/main/README.md Installs the latest version of the chromem-go library using the go get command. This is the primary method for integrating the library into your Go projects. ```bash go get github.com/philippgille/chromem-go@latest ``` -------------------------------- ### Quickstart: Create Collection, Add Documents, and Query in Go Source: https://github.com/philippgille/chromem-go/blob/main/README.md A minimal Go program demonstrating the basic usage of chromem-go. It shows how to create a new collection, add documents with IDs and content, and perform a similarity search query. It defaults to using OpenAI for embeddings if no embedding function is provided. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() // Passing nil as embedding function leads to OpenAI being used and requires // "OPENAI_API_KEY" env var to be set. Other providers are supported as well. // For example pass `chromem.NewEmbeddingFuncOllama(...)` to use Ollama. c, err := db.CreateCollection("knowledge-base", nil, nil) if err != nil { panic(err) } err = c.AddDocuments(ctx, []chromem.Document{ { ID: "1", Content: "The sky is blue because of Rayleigh scattering.", }, { ID: "2", Content: "Leaves are green because chlorophyll absorbs red and blue light.", }, }, runtime.NumCPU()) if err != nil { panic(err) } res, err := c.Query(ctx, "Why is the sky blue?", 1, nil, nil) if err != nil { panic(err) } fmt.Printf("ID: %v\nSimilarity: %v\nContent: %v\n", res[0].ID, res[0].Similarity, res[0].Content) } ``` -------------------------------- ### Get or Create Collection in chromem-go Source: https://github.com/philippgille/chromem-go/blob/main/examples/rag-wikipedia-ollama/README.md Demonstrates how to retrieve an existing collection or create a new one within the chromem-go database. It shows variations for specifying embedding functions, including Ollama and a default/nil option. ```go collection, err := db.GetOrCreateCollection("Wikipedia", nil, chromem.NewEmbeddingFuncOllama(embeddingModel)) if err != nil { panic(err) } ``` ```go collection, err := db.GetOrCreateCollection("Wikipedia", nil, nil) if err != nil { panic(err) } ``` -------------------------------- ### Get or Create Collection in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Demonstrates how to get an existing collection or create a new one if it doesn't exist using chromem-go. This is useful for ensuring idempotency in operations. ```go package main import ( "fmt" "github.com/philippgille/chromem-go" ) func main() { db := chromem.NewDB() // First call creates the collection col1, err := db.GetOrCreateCollection("my-collection", nil, nil) if err != nil { panic(err) } // Second call returns the existing collection col2, err := db.GetOrCreateCollection("my-collection", nil, nil) if err != nil { panic(err) } fmt.Printf("Same collection: %v\n", col1 == col2) // Output: Same collection: true } ``` -------------------------------- ### Serve WASM Application Locally Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/README.md This command starts a local web server to serve the compiled WASM application and its associated files. It uses the 'philippgille/serve' tool to host the content on localhost at port 8080, making it accessible via a web browser. ```bash cd ../examples/webassembly go run github.com/philippgille/serve@latest -b localhost -p 8080 ``` -------------------------------- ### Create Persistent Database - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Creates a persistent chromem-go database that automatically saves collections and documents to disk using gzip compression. Data is written synchronously and restored on reopen. Demonstrates creating or getting a collection and adding documents. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() // Create a persistent database with gzip compression // If path is empty, defaults to "./chromem-go" db, err := chromem.NewPersistentDB("./my-vector-db", true) if err != nil { panic(err) } // Create or get existing collection // EmbeddingFunc must be provided when loading a persisted collection collection, err := db.GetOrCreateCollection("persistent-docs", nil, nil) if err != nil { panic(err) } // Add documents - they are automatically persisted err = collection.AddDocuments(ctx, []chromem.Document{ { ID: "doc1", Content: "Machine learning is a subset of artificial intelligence.", Metadata: map[string]string{"category": "tech", "source": "wiki"}, }, }, runtime.NumCPU()) if err != nil { panic(err) } fmt.Printf("Document count: %d\n", collection.Count()) // Output: Document count: 1 } ``` -------------------------------- ### Export/Import Database via Streams - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Enables exporting and importing the database using io.Writer and io.ReadSeeker interfaces. Facilitates integration with cloud storage services like S3 or direct network transfers. Supports compression but not encryption in this example. ```go package main import ( "bytes" "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() // Create and populate a database db := chromem.NewDB() collection, _ := db.CreateCollection("streamable", nil, nil) collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "Streamable content."}, }, runtime.NumCPU()) // Export to a buffer (could be S3, HTTP response, etc.) var buffer bytes.Buffer err := db.ExportToWriter(&buffer, true, "") // compressed, no encryption if err != nil { panic(err) } fmt.Printf("Exported %d bytes\n", buffer.Len()) // Import from the buffer newDB := chromem.NewDB() reader := bytes.NewReader(buffer.Bytes()) err = newDB.ImportFromReader(reader, "") if err != nil { panic(err) } restored := newDB.GetCollection("streamable", nil) fmt.Printf("Restored collection with %d documents\n", restored.Count()) } ``` -------------------------------- ### Configure OpenAI for Embeddings and LLM in Go Source: https://github.com/philippgille/chromem-go/blob/main/examples/rag-wikipedia-ollama/README.md This Go code snippet shows how to fully integrate OpenAI for both embedding generation and LLM responses within the chromem-go framework. It replaces the Ollama-specific configurations with OpenAI client initialization using the OPENAI_API_KEY environment variable and specifies OpenAI's GPT-3.5 Turbo model for chat completions. This approach simplifies the setup by relying on a single external service. ```go diff --git a/examples/rag-wikipedia-ollama/llm.go b/examples/rag-wikipedia-ollama/llm.go index 1fde4ec..7cb81cc 100644 --- a/examples/rag-wikipedia-ollama/llm.go +++ b/examples/rag-wikipedia-ollama/llm.go @@ -2,23 +2,13 @@ package main import ( "context" - "net/http" + "os" "strings" "text/template" "github.com/sashabaranov/go-openai" ) -const ( - // We use a local LLM running in Ollama for asking the question: https://github.com/ollama/ollama - ollamaBaseURL = "http://localhost:11434/v1" - // We use Google's Gemma (2B), a very small model that doesn't need much resources - // and is fast, but doesn't have much knowledge: https://huggingface.co/google/gemma-2b - // We found Gemma 2B to be superior to TinyLlama (1.1B), Stable LM 2 (1.6B) - // and Phi-2 (2.7B) for the retrieval augmented generation (RAG) use case. - llmModel = "gemma:2b" -) - // There are many different ways to provide the context to the LLM. // You can pass each context as user message, or the list as one user message, // or pass it in the system prompt. The system prompt itself also has a big impact @@ -47,10 +37,7 @@ Don't mention the knowledge base, context or search results in your answer. func askLLM(ctx context.Context, contexts []string, question string) string { // We can use the OpenAI client because Ollama is compatible with OpenAI's API. - openAIClient := openai.NewClientWithConfig(openai.ClientConfig{ - BaseURL: ollamaBaseURL, - HTTPClient: http.DefaultClient, - }) + openAIClient := openai.NewClient(os.Getenv("OPENAI_API_KEY")) sb := &strings.Builder{} err := systemPromptTpl.Execute(sb, contexts) if err != nil { @@ -66,7 +53,7 @@ func askLLM(ctx context.Context, contexts []string, question string) string { }, } res, err := openAIClient.CreateChatCompletion(ctx, openai.ChatCompletionRequest{ - Model: llmModel, + Model: openai.GPT3Dot5Turbo, Messages: messages, }) if err != nil { ``` ```go diff --git a/examples/rag-wikipedia-ollama/main.go b/examples/rag-wikipedia-ollama/main.go index 55b3076..044a246 100644 --- a/examples/rag-wikipedia-ollama/main.go +++ b/examples/rag-wikipedia-ollama/main.go @@ -12,19 +12,11 @@ import ( "github.com/philippgille/chromem-go" ) -const ( - question = "When did the Monarch Company exist?" - // We use a local LLM running in Ollama for the embedding: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 - embeddingModel = "nomic-embed-text" -) + +const question = "When did the Monarch Company exist?" func main() { ctx := context.Background() - // Warm up Ollama, in case the model isn't loaded yet - log.Println("Warming up Ollama...") - _ = askLLM(ctx, nil, "Hello!") - // First we ask an LLM a fairly specific question that it likely won't know // the answer to. log.Println("Question: " + question) @@ -48,7 +40,7 @@ func main() { // variable to be set. // For this example we choose to use a locally running embedding model though. // It requires Ollama to serve its API at "http://localhost:11434/api". ``` -------------------------------- ### Create Collection with Metadata - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Creates a new collection within a chromem-go database with optional custom metadata. If no embedding function is specified, it defaults to OpenAI's text-embedding-3-small model. This example demonstrates creating a collection named 'tech-docs'. ```go package main import ( "context" "fmt" "github.com/philippgille/chromem-go" ) func main() { db := chromem.NewDB() // Create collection with custom metadata metadata := map[string]string{ "description": "Technical documentation", "version": "1.0", } // Using default OpenAI embeddings (requires OPENAI_API_KEY env var) collection, err := db.CreateCollection("tech-docs", metadata, nil) if err != nil { panic(err) } fmt.Printf("Collection created: %s\n", collection.Name) // Output: Collection created: tech-docs } ``` -------------------------------- ### Run Benchmarks in Go Source: https://github.com/philippgille/chromem-go/blob/main/README.md Commands to run benchmarks for the chromem-go project. It includes options for basic benchmarking, memory allocation analysis, and generating profile data for CPU, memory, blocking, and mutex contention. ```bash go test -benchmem -run=^$ -bench . # With profiling: go test -benchmem -run ^$ -cpuprofile cpu.out -bench . # (profiles: -cpuprofile, -memprofile, -blockprofile, -mutexprofile) ``` -------------------------------- ### Build and Test Go Project Source: https://github.com/philippgille/chromem-go/blob/main/README.md Standard commands for building and testing Go projects. The build command compiles the project, while the test command runs unit tests with race detection enabled. ```bash go build ./... go test -v -race -count 1 ./... ``` -------------------------------- ### Export/Import Database to File - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Demonstrates exporting the entire database to a single file and importing it back. Supports optional compression and encryption using a provided key. Essential for backups and data migration. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() // Create and populate a database db := chromem.NewDB() collection, _ := db.CreateCollection("exportable", nil, nil) collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "This will be exported."}, {ID: "2", Content: "This too."}, }, runtime.NumCPU()) // Export with compression and encryption encryptionKey := "32-byte-long-encryption-key!!" // Must be exactly 32 bytes err := db.ExportToFile("./backup.gob.gz.enc", true, encryptionKey) if err != nil { panic(err) } fmt.Println("Database exported") // Import into a new database newDB := chromem.NewDB() err = newDB.ImportFromFile("./backup.gob.gz.enc", encryptionKey) if err != nil { panic(err) } restored := newDB.GetCollection("exportable", nil) fmt.Printf("Restored %d documents\n", restored.Count()) // Output: // Database exported // Restored 2 documents } ``` -------------------------------- ### Initialize Database with OpenAI API Key Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/index.html Initializes the database using a provided OpenAI API key. It retrieves the API key from an HTML input element with the ID 'openai-api-key'. Error handling is included for the initialization process. ```javascript function initDBWithKey() { console.log("Initializing DB...") const openaiApiKey = document.getElementById("openai-api-key").value; const err = initDB(openaiApiKey) if (err) { console.error('Returned error:', err) } else { console.log("DB initialized.") } } ``` -------------------------------- ### List All Collections in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Shows how to retrieve a map of all collections currently present in the database using chromem-go. The returned map is a copy and can be safely modified. ```go package main import ( "fmt" "github.com/philippgille/chromem-go" ) func main() { db := chromem.NewDB() db.CreateCollection("collection-a", nil, nil) db.CreateCollection("collection-b", nil, nil) db.CreateCollection("collection-c", nil, nil) collections := db.ListCollections() for name, col := range collections { fmt.Printf("Collection: %s, Documents: %d\n", name, col.Count()) } // Output: // Collection: collection-a, Documents: 0 // Collection: collection-b, Documents: 0 // Collection: collection-c, Documents: 0 } ``` -------------------------------- ### Initialize and Query Collection in Go Source: https://github.com/philippgille/chromem-go/blob/main/README.md Initializes an in-memory chromem-go database, creates a collection, adds documents with embeddings and metadata, and then queries for similar documents. Supports optional filters. ```go package main import "github.com/philippgille/chromem-go" func main() { // Set up chromem-go in-memory, for easy prototyping. Can add persistence easily! // We call it DB instead of client because there's no client-server separation. The DB is embedded. db := chromem.NewDB() // Create collection. GetCollection, GetOrCreateCollection, DeleteCollection also available! collection, _ := db.CreateCollection("all-my-documents", nil, nil) // Add docs to the collection. Update and delete will be added in the future. // Can be multi-threaded with AddConcurrently()! // We're showing the Chroma-like method here, but more Go-idiomatic methods are also available! _ = collection.Add(ctx, []string{"doc1", "doc2"}, // unique ID for each doc nil, // We handle embedding automatically. You can skip that and add your own embeddings as well. []map[string]string{{"source": "notion"}, {"source": "google-docs"}}, // Filter on these! []string{"This is document1", "This is document2"}, ) // Query/search 2 most similar results. You can also get by ID. results, _ := collection.Query(ctx, "This is a query document", 2, map[string]string{"metadata_field": "is_equal_to_this"}, // optional filter map[string]string{"$contains": "search_string"}, // optional filter ) } ``` -------------------------------- ### Run Full Workflow Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/index.html Executes the complete workflow, including initializing the database with a key, adding documents, and then querying the database to print the results. This function orchestrates the other asynchronous operations. ```javascript async function runWorkflow() { initDBWithKey(); await addDocuments(); await queryAndPrint(); } ``` -------------------------------- ### Create Vertex AI Embeddings - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Demonstrates creating embedding functions using Google Cloud's Vertex AI. Supports basic usage and advanced options like auto-truncation and custom API endpoints. Requires GCP access token and project ID. ```go package main import ( "github.com/philippgille/chromem-go" ) func main() { apiKey := "your-gcp-access-token" project := "your-gcp-project-id" // Basic usage embedFunc := chromem.NewEmbeddingFuncVertex( apiKey, project, chromem.EmbeddingModelVertexEnglishV4, ) // With options embedFuncWithOpts := chromem.NewEmbeddingFuncVertex( apiKey, project, chromem.EmbeddingModelVertexGeminiV1, chromem.WithVertexAutoTruncate(true), chromem.WithVertexAPIEndpoint("https://custom-endpoint.googleapis.com/v1"), ) db := chromem.NewDB() collection, _ := db.CreateCollection("vertex-docs", nil, embedFunc) _ = collection _ = embedFuncWithOpts } ``` -------------------------------- ### Add Multiple Documents Concurrently in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Shows how to add multiple documents to a collection efficiently using concurrent operations in chromem-go. Embeddings are generated automatically if not provided. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() collection, _ := db.CreateCollection("articles", nil, nil) documents := []chromem.Document{ { ID: "article-1", Content: "Go is a statically typed, compiled programming language.", Metadata: map[string]string{"topic": "programming", "lang": "go"}, }, { ID: "article-2", Content: "Python is known for its simple syntax and readability.", Metadata: map[string]string{"topic": "programming", "lang": "python"}, }, { ID: "article-3", Content: "Rust provides memory safety without garbage collection.", Metadata: map[string]string{"topic": "programming", "lang": "rust"}, }, } // Add documents with concurrency matching CPU count err := collection.AddDocuments(ctx, documents, runtime.NumCPU()) if err != nil { panic(err) } fmt.Printf("Added %d documents\n", collection.Count()) // Output: Added 3 documents } ``` -------------------------------- ### Create Local Ollama Embeddings in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Demonstrates creating embeddings using a locally running Ollama instance, ideal for privacy-sensitive applications. It allows specifying the Ollama model and base URL. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() // Use Ollama with nomic-embed-text model // Second parameter is base URL, empty uses default "http://localhost:11434/api" embedFunc := chromem.NewEmbeddingFuncOllama("nomic-embed-text", "") db := chromem.NewDB() collection, _ := db.CreateCollection("local-docs", nil, embedFunc) err := collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "Local embeddings keep your data private."}, }, runtime.NumCPU()) if err != nil { panic(err) } fmt.Printf("Documents with local embeddings: %d\n", collection.Count()) } ``` -------------------------------- ### Initialize WASM and Run Go Program Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/index.html Initializes the WebAssembly module from 'chromem-go.wasm' and runs the Go program. It requires the Go WASM runtime and the 'Go' object to be available in the global scope. ```javascript const go = new Go(); console.log("Initializing WASM..."); WebAssembly.instantiateStreaming(fetch("chromem-go.wasm"), go.importObject).then((result) => { console.log("WASM initialized."); go.run(result.instance); }); ``` -------------------------------- ### Create Document with Embeddings - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Creates a new document and generates embeddings immediately using default OpenAI embeddings. This is useful for controlling document creation before adding it to a collection. Requires context, document ID, metadata, content, and optionally embeddings and an embedding function. ```go package main import ( "context" "fmt" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() // Create document with default OpenAI embeddings doc, err := chromem.NewDocument( ctx, "doc-id-123", // ID map[string]string{"source": "manual"}, // Metadata nil, // Embeddings (nil = generate) "Vector databases enable semantic search.", // Content nil, // EmbeddingFunc (nil = default) ) if err != nil { panic(err) } fmt.Printf("Document ID: %s\n", doc.ID) fmt.Printf("Embedding dimensions: %d\n", len(doc.Embedding)) // Output: // Document ID: doc-id-123 // Embedding dimensions: 1536 } ``` -------------------------------- ### Configure OpenAI Embeddings with Gemma LLM in Go Source: https://github.com/philippgille/chromem-go/blob/main/examples/rag-wikipedia-ollama/README.md This Go code snippet demonstrates how to configure the chromem-go library to use OpenAI for generating embeddings while still utilizing a local Gemma LLM for question answering. It involves modifying the database collection creation to use OpenAI's embedding capabilities and updating log messages to reflect the change. Ensure the OPENAI_API_KEY environment variable is set. ```go diff --git a/examples/rag-wikipedia-ollama/main.go b/examples/rag-wikipedia-ollama/main.go index 55b3076..cee9561 100644 --- a/examples/rag-wikipedia-ollama/main.go +++ b/examples/rag-wikipedia-ollama/main.go @@ -14,8 +14,6 @@ import ( const ( question = "When did the Monarch Company exist?" - // We use a local LLM running in Ollama for the embedding: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 - embeddingModel = "nomic-embed-text" ) func main() { @@ -48,7 +46,7 @@ func main() { // variable to be set. // For this example we choose to use a locally running embedding model though. // It requires Ollama to serve its API at "http://localhost:11434/api". - collection, err := db.GetOrCreateCollection("Wikipedia", nil, chromem.NewEmbeddingFuncOllama(embeddingModel)) + collection, err := db.GetOrCreateCollection("Wikipedia", nil, nil) if err != nil { panic(err) } @@ -82,7 +80,7 @@ func main() { Content: article.Text, }) } - log.Println("Adding documents to chromem-go, including creating their embeddings via Ollama API...") + log.Println("Adding documents to chromem-go, including creating their embeddings via OpenAI API...") err = collection.AddDocuments(ctx, docs, runtime.NumCPU()) if err != nil { panic(err) ``` -------------------------------- ### Create OpenAI Embeddings in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Shows how to generate embeddings using OpenAI's API. It requires the OPENAI_API_KEY environment variable and allows specifying different OpenAI embedding models. ```go package main import ( "os" "github.com/philippgille/chromem-go" ) func main() { // Using default model (text-embedding-3-small) defaultFunc := chromem.NewEmbeddingFuncDefault() // Using specific OpenAI model apiKey := os.Getenv("OPENAI_API_KEY") largeFunc := chromem.NewEmbeddingFuncOpenAI(apiKey, chromem.EmbeddingModelOpenAI3Large) // Create collection with custom embedding function db := chromem.NewDB() collection, _ := db.CreateCollection("docs", nil, largeFunc) _ = collection // Use collection... _ = defaultFunc } ``` -------------------------------- ### Add Documents to chromem-go with Embeddings (OpenAI) Source: https://github.com/philippgille/chromem-go/blob/main/examples/rag-wikipedia-ollama/README.md Illustrates the process of adding documents to a collection in chromem-go. This snippet specifically highlights the logging message indicating that embeddings are being created via the OpenAI API. ```go log.Println("Adding documents to chromem-go, including creating their embeddings via OpenAI API...") err = collection.AddDocuments(ctx, docs, runtime.NumCPU()) if err != nil { panic(err) } ``` -------------------------------- ### Create Cohere Embeddings in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Explains how to generate embeddings using Cohere's API, emphasizing the need to prefix text with input types for accurate embedding. It supports specifying Cohere embedding models. ```go package main import ( "context" "fmt" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() apiKey := "your-cohere-api-key" embedFunc := chromem.NewEmbeddingFuncCohere(apiKey, chromem.EmbeddingModelCohereEnglishV3) db := chromem.NewDB() collection, _ := db.CreateCollection("cohere-docs", nil, embedFunc) // Cohere requires input type prefix content := "Vector databases enable semantic search." contentWithPrefix := chromem.InputTypeCohereSearchDocumentPrefix + content doc, _ := chromem.NewDocument(ctx, "doc1", nil, nil, contentWithPrefix, embedFunc) doc.Content = content // Remove prefix for storage err := collection.AddDocument(ctx, doc) if err != nil { panic(err) } // Query also needs prefix results, _ := collection.Query(ctx, chromem.InputTypeCohereSearchQueryPrefix+"semantic search", 1, nil, nil) fmt.Printf("Found: %s\n", results[0].Content) } ``` -------------------------------- ### Copy Go WASM JavaScript Wrapper Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/README.md This command copies the necessary JavaScript wrapper file provided by the Go distribution. This file is essential for interacting with the compiled WebAssembly module from JavaScript. ```bash cp $(go env GOROOT)/misc/wasm/wasm_exec.js ../examples/webassembly/wasm_exec.js ``` -------------------------------- ### Create In-Memory Database - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Creates a new in-memory chromem-go database. Data is stored only in memory but can be exported/imported. It defaults to OpenAI embeddings and demonstrates adding and querying documents. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() // Create a new in-memory database db := chromem.NewDB() // Create a collection (uses OpenAI embeddings by default) collection, err := db.CreateCollection("my-documents", nil, nil) if err != nil { panic(err) } // Add documents err = collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "The sky is blue because of Rayleigh scattering."}, {ID: "2", Content: "Leaves are green because chlorophyll absorbs red and blue light."}, }, runtime.NumCPU()) if err != nil { panic(err) } // Query the collection results, err := collection.Query(ctx, "Why is the sky blue?", 1, nil, nil) if err != nil { panic(err) } fmt.Printf("ID: %s\nSimilarity: %f\nContent: %s\n", results[0].ID, results[0].Similarity, results[0].Content) // Output: // ID: 1 // Similarity: 0.683337 // Content: The sky is blue because of Rayleigh scattering. } ``` -------------------------------- ### Compile Go to WASM for Browser Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/README.md This command compiles the Go WASM binding for use in JavaScript environments. It sets the target operating system to 'js' and architecture to 'wasm', outputting a .wasm file. Ensure you are in the correct directory before executing. ```bash cd /path/to/chromem-go/wasm GOOS=js GOARCH=wasm go build -o ../examples/webassembly/chromem-go.wasm ``` -------------------------------- ### Selective Export/Import of Collections in Go Source: https://context7.com/philippgille/chromem-go/llms.txt This Go code snippet demonstrates how to export specific collections ('collection-a', 'collection-b') to a file and then import only 'collection-a' into a new database. It utilizes the chromem-go library for database operations. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() // Create multiple collections col1, _ := db.CreateCollection("collection-a", nil, nil) col2, _ := db.CreateCollection("collection-b", nil, nil) col3, _ := db.CreateCollection("collection-c", nil, nil) col1.AddDocuments(ctx, []chromem.Document{{ID: "1", Content: "A"}}, runtime.NumCPU()) col2.AddDocuments(ctx, []chromem.Document{{ID: "1", Content: "B"}}, runtime.NumCPU()) col3.AddDocuments(ctx, []chromem.Document{{ID: "1", Content: "C"}}, runtime.NumCPU()) // Export only specific collections err := db.ExportToFile("./partial-backup.gob", false, "", "collection-a", "collection-b") if err != nil { panic(err) } // Import only specific collections newDB := chromem.NewDB() err = newDB.ImportFromFile("./partial-backup.gob", "", "collection-a") if err != nil { panic(err) } fmt.Printf("Imported collections: %d\n", len(newDB.ListCollections())) // Output: Imported collections: 1 } ``` -------------------------------- ### Query Similar Documents in Python Source: https://github.com/philippgille/chromem-go/blob/main/README.md Queries a collection to find the 'n_results' most similar documents to the provided 'query_texts'. Supports optional metadata and document content filtering. ```python results = collection.query( query_texts=["This is a query document"], n_results=2, # where={"metadata_field": "is_equal_to_this"}, # optional filter # where_document={"$ contains":"search_string"} # optional filter ) ``` -------------------------------- ### Perform Semantic Search by Text in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Demonstrates how to perform a semantic similarity search using a text query. The query text is converted to embeddings using the collection's embedding function. This requires the chromem-go library. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() collection, _ := db.CreateCollection("knowledge", nil, nil) collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "The mitochondria is the powerhouse of the cell."}, {ID: "2", Content: "Photosynthesis converts sunlight into chemical energy."}, {ID: "3", Content: "DNA contains genetic instructions for organisms."}, {ID: "4", Content: "Neurons transmit electrical signals in the brain."}, }, runtime.NumCPU()) // Query for similar documents results, err := collection.Query(ctx, "How do cells generate energy?", 2, nil, nil) if err != nil { panic(err) } for _, result := range results { fmt.Printf("ID: %s (similarity: %.4f)\n %s\n", result.ID, result.Similarity, result.Content) } // Output: // ID: 1 (similarity: 0.8234) // The mitochondria is the powerhouse of the cell. // ID: 2 (similarity: 0.7156) // Photosynthesis converts sunlight into chemical energy. } ``` -------------------------------- ### Perform Advanced Queries with Negative Filtering in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Demonstrates how to use QueryWithOptions to perform searches with negative filtering, excluding specific unwanted results. This function takes a QueryOptions struct which includes QueryText, NResults, and NegativeQueryOptions. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() collection, _ := db.CreateCollection("articles", nil, nil) collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "Python is great for machine learning and data science."}, {ID: "2", Content: "Go excels at building concurrent web services."}, {ID: "3", Content: "Python web frameworks like Django are popular."}, {ID: "4", Content: "Machine learning with TensorFlow and PyTorch."}, }, runtime.NumCPU()) // Query for ML content but exclude web-related results options := chromem.QueryOptions{ QueryText: "machine learning programming", NResults: 2, Negative: chromem.NegativeQueryOptions{ Mode: chromem.NEGATIVE_MODE_FILTER, Text: "web services frameworks", FilterThreshold: 0.4, }, } results, err := collection.QueryWithOptions(ctx, options) if err != nil { panic(err) } for _, r := range results { fmt.Printf("%s: %s\n", r.ID, r.Content) } // Results exclude web-related articles } ``` -------------------------------- ### Query Database and Display Results Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/index.html Asynchronously queries the database for a specific question ('Why is the sky blue?') and displays the results in an HTML element with the ID 'output'. It handles potential errors during the query. ```javascript async function queryAndPrint() { console.log("Querying DB...") try { const res = await query("Why is the sky blue?"); console.log("DB queried."); const outputElement = document.getElementById("output"); outputElement.textContent = `ID: ${res.ID}\nSimilarity: ${res.Similarity}\nContent: ${res.Content}\n`; } catch (err) { console.error('Caught exception', err) } } ``` -------------------------------- ### Query with Filters Source: https://context7.com/philippgille/chromem-go/llms.txt Filters query results by metadata and document content using `where` and `whereDocument` parameters. This allows for more specific searches based on document attributes and content. ```APIDOC ## POST /query (with filters) ### Description Filter query results by metadata and document content using where and whereDocument parameters. ### Method POST ### Endpoint `/query` ### Parameters #### Query Parameters - **query** (string) - Required - The text to search for. - **limit** (integer) - Optional - The maximum number of results to return. - **where** (object) - Required - Metadata filters for the search. Example: `{"price": "high"}` - **where_document** (object) - Required - Document content filters for the search. Example: `{"$contains": "headphones"}` ### Request Body ```json { "query": "audio equipment", "limit": 2, "where": { "category": "electronics", "price": "high" }, "where_document": { "$contains": "headphones" } } ``` ### Response #### Success Response (200) - **results** (array) - An array of filtered search results. - **ID** (string) - The unique identifier of the document. - **Content** (string) - The content of the document. - **Metadata** (object) - The metadata associated with the document. #### Response Example ```json { "results": [ { "ID": "1", "Content": "Premium wireless headphones with noise cancellation", "Metadata": {"category": "electronics", "price": "high"} }, { "ID": "3", "Content": "Professional studio monitor headphones", "Metadata": {"category": "electronics", "price": "high"} } ] } ``` ``` -------------------------------- ### Search by Pre-computed Embedding in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Explains how to perform a similarity search using a pre-computed embedding vector instead of raw text. This method is useful when embeddings are generated separately or from external sources. It utilizes the `chromem-go` library and an embedding function like Ollama. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() // Use Ollama for local embeddings embedFunc := chromem.NewEmbeddingFuncOllama("nomic-embed-text", "") collection, _ := db.CreateCollection("docs", nil, embedFunc) collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "Database indexing improves query performance."}, {ID: "2", Content: "Caching reduces latency in web applications."}, }, runtime.NumCPU()) // Create embedding separately (e.g., from another source) queryEmbedding, err := embedFunc(ctx, "How to make databases faster?") if err != nil { panic(err) } // Query using the pre-computed embedding results, err := collection.QueryEmbedding(ctx, queryEmbedding, 1, nil, nil) if err != nil { panic(err) } fmt.Printf("Most similar: %s\n", results[0].Content) // Output: Most similar: Database indexing improves query performance. } ``` -------------------------------- ### Reset Database Collections in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Demonstrates how to clear all collections from the database using chromem-go. This operation also purges the persistence directory for persistent databases. ```go package main import ( "fmt" "github.com/philippgille/chromem-go" ) func main() { db := chromem.NewDB() db.CreateCollection("col1", nil, nil) db.CreateCollection("col2", nil, nil) fmt.Printf("Before reset: %d collections\n", len(db.ListCollections())) err := db.Reset() if err != nil { panic(err) } fmt.Printf("After reset: %d collections\n", len(db.ListCollections())) // Output: // Before reset: 2 collections // After reset: 0 collections } ``` -------------------------------- ### Add Documents to the Database Source: https://github.com/philippgille/chromem-go/blob/main/examples/webassembly/index.html Asynchronously adds two documents to the database with predefined IDs and content. It uses the 'addDocument' function and includes error handling for exceptions during the process. ```javascript async function addDocuments() { console.log("Adding documents...") try { await addDocument("1", "The sky is blue because of Rayleigh scattering."); console.log("Document 1 added.") await addDocument("2", "Leaves are green because chlorophyll absorbs red and blue light."); console.log("Document 2 added.") console.log("Documents added.") } catch (err) { console.error('Caught exception', err) } } ``` -------------------------------- ### List All Document IDs - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Retrieves a list of all document IDs present in a collection. This function is useful for inventorying documents or preparing for batch operations. Requires context and a collection. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() collection, _ := db.CreateCollection("docs", nil, nil) collection.AddDocuments(ctx, []chromem.Document{ {ID: "alpha", Content: "First"}, {ID: "beta", Content: "Second"}, {ID: "gamma", Content: "Third"}, }, runtime.NumCPU()) ids := collection.ListIDs(ctx) fmt.Printf("Document IDs: %v\n", ids) // Output: Document IDs: [alpha beta gamma] (order may vary) } ``` -------------------------------- ### Filter Semantic Search Results by Metadata and Content in Go Source: https://context7.com/philippgille/chromem-go/llms.txt Shows how to filter semantic search results based on document metadata and content. It utilizes the `where` parameter for metadata filtering and `whereDocument` for content filtering. Dependencies include the chromem-go library. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() collection, _ := db.CreateCollection("products", nil, nil) collection.AddDocuments(ctx, []chromem.Document{ {ID: "1", Content: "Premium wireless headphones with noise cancellation", Metadata: map[string]string{"category": "electronics", "price": "high"}}, {ID: "2", Content: "Budget wireless earbuds for everyday use", Metadata: map[string]string{"category": "electronics", "price": "low"}}, {ID: "3", Content: "Professional studio monitor headphones", Metadata: map[string]string{"category": "electronics", "price": "high"}}, }, runtime.NumCPU()) // Query with metadata filter whereFilter := map[string]string{"price": "high"} // Query with document content filter whereDocFilter := map[string]string{"$contains": "headphones"} results, err := collection.Query(ctx, "audio equipment", 2, whereFilter, whereDocFilter) if err != nil { panic(err) } for _, r := range results { fmt.Printf("%s: %s\n", r.ID, r.Content) } // Output shows only high-price headphones (not earbuds) } ``` -------------------------------- ### Retrieve Document by ID - Go Source: https://context7.com/philippgille/chromem-go/llms.txt Retrieves a specific document from a collection using its unique ID. The returned document is a deep copy, allowing for safe modification. Requires context, a collection, and the document ID. ```go package main import ( "context" "fmt" "runtime" "github.com/philippgille/chromem-go" ) func main() { ctx := context.Background() db := chromem.NewDB() collection, _ := db.CreateCollection("docs", nil, nil) collection.AddDocuments(ctx, []chromem.Document{ {ID: "doc-1", Content: "First document content"}, {ID: "doc-2", Content: "Second document content"}, }, runtime.NumCPU()) // Retrieve specific document doc, err := collection.GetByID(ctx, "doc-1") if err != nil { panic(err) } fmt.Printf("Found: %s - %s\n", doc.ID, doc.Content) // Output: Found: doc-1 - First document content } ```