### Quick Start: Add and Search Documents with VecturaKit Source: https://github.com/rryam/vecturakit/blob/main/README.md Demonstrates a basic workflow for initializing VecturaKit, adding text documents, and performing a similarity search. It utilizes `SwiftEmbedder` for embeddings and handles potential errors. ```swift import VecturaKit Task { do { let config = VecturaConfig(name: "my-db") let embedder = SwiftEmbedder(modelSource: .default) let vectorDB = try await VecturaKit(config: config, embedder: embedder) // Add documents let ids = try await vectorDB.addDocuments(texts: [ "The quick brown fox jumps over the lazy dog", "Swift is a powerful programming language" ]) // Search documents let results = try await vectorDB.search(query: "programming language", numResults: 5) print("Found \(results.count) results!") } catch { print("Error: \(error)") } } ``` -------------------------------- ### Implement Mock Indexed Storage (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md An example implementation of an actor conforming to the `IndexedVecturaStorage` protocol. This mock storage is used for testing accuracy and provides methods for loading documents, retrieving document counts, searching candidates, and loading documents by ID. ```swift actor MyIndexedStorage: IndexedVecturaStorage { func loadDocuments(offset: Int, limit: Int) async throws -> [VecturaDocument] { // Your implementation } func getTotalDocumentCount() async throws -> Int { // Your implementation } func searchCandidates(queryEmbedding: [Float], topK: Int, prefilterSize: Int) async throws -> [UUID] { // Your implementation } func loadDocuments(ids: [UUID]) async throws -> [UUID: VecturaDocument] { // Your implementation } } ``` -------------------------------- ### Backward Compatible VecturaKit Initialization (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Shows the unchanged code for initializing VecturaKit that remains compatible with existing implementations. This configuration uses the default memory strategy and does not require a custom storage provider, allowing searches to function as before. ```swift // No changes needed - automatic backward compatibility let config = VecturaConfig(name: "my-db") let vectura = try await VecturaKit(config: config) // Searches work exactly as before let results = try await vectura.search( query: "machine learning", numResults: 10 ) ``` -------------------------------- ### Default FileStorageProvider Usage (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Demonstrates the default initialization of VecturaKit, which implicitly uses the FileStorageProvider. Note that this provider does not support indexed operations and will fall back to full memory mode. ```swift // Uses FileStorageProvider by default let vectura = try await VecturaKit(config: config) ``` -------------------------------- ### Implement SQLite Indexed Storage Provider in Swift Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Provides a conceptual Swift implementation of an `SQLiteIndexedStorageProvider` that conforms to the `IndexedVecturaStorage` protocol. It includes methods for initializing the database, loading and saving documents, and searching for candidate documents using embeddings. Dependencies include the `SQLite3` library. ```swift import SQLite3 public final class SQLiteIndexedStorageProvider: IndexedVecturaStorage { private var db: OpaquePointer? public init(dbPath: String) throws { // Open database sqlite3_open(dbPath, &db) try createTables() } // MARK: - VecturaStorage public func loadDocuments() async throws -> [VecturaDocument] { // Full load (discouraged for large datasets) return try await loadDocuments(offset: 0, limit: Int.max) } public func saveDocument(_ document: VecturaDocument) async throws { // INSERT INTO documents (id, text, embedding) VALUES (?, ?, ?) // Update vector index } // MARK: - IndexedVecturaStorage public func getTotalDocumentCount() async throws -> Int { // SELECT COUNT(*) FROM documents return 0 // placeholder } public func searchCandidates( queryEmbedding: [Float], topK: Int, prefilterSize: Int ) async throws -> [UUID] { // Option 1: Use sqlite-vss extension for vector search // Option 2: Implement IVF (Inverted File) indexing // Option 3: Use Product Quantization (PQ) // Returns candidate document IDs return [] } public func loadDocuments(ids: [UUID]) async throws -> [UUID: VecturaDocument] { // SELECT * FROM documents WHERE id IN (?, ?, ...) return [:] } } ``` -------------------------------- ### Migrate to Indexed Storage Mode in VecturaKit (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Demonstrates how to opt into the indexed storage mode in VecturaKit. This involves configuring the `VecturaConfig` with `.indexed()` memory strategy and providing a custom storage implementation, such as `SQLiteIndexedStorageProvider`, during initialization. The API for searching remains unchanged. ```swift // 1. Configure indexed strategy let config = VecturaConfig( name: "my-db", memoryStrategy: .indexed() ) // 2. Provide custom storage (when available) let sqliteProvider = try SQLiteIndexedStorageProvider(dbPath: "/path/to/db") let vectura = try await VecturaKit(config: config, storageProvider: sqliteProvider) // 3. Use as normal - API unchanged let results = try await vectura.search( query: "machine learning", numResults: 10 ) ``` -------------------------------- ### Search Documents in VecturaKit Source: https://github.com/rryam/vecturakit/blob/main/README.md Provides examples for searching documents within the VecturaKit database. Supports searching by text using hybrid search (combining text and vector similarity) and by providing a vector embedding directly. ```swift let results = try await vectorDB.search( query: "search query", numResults: 5, // Optional threshold: 0.8 // Optional ) for result in results { print("Document ID: (result.id)") print("Text: (result.text)") print("Similarity Score: (result.score)") print("Created At: (result.createdAt)") } ``` ```swift // Using array literal let results = try await vectorDB.search( query: [0.1, 0.2, 0.3, ...], // Array literal matching config.dimension numResults: 5, // Optional threshold: 0.8 // Optional ) // Or explicitly use SearchQuery enum let embedding: [Float] = getEmbedding() let results = try await vectorDB.search( query: .vector(embedding), numResults: 5, threshold: 0.8 ) ``` -------------------------------- ### Configure Full Memory Strategy (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Explicitly configures VecturaKit to use the full memory strategy, ensuring maximum search speed by loading all data into RAM. Best for smaller datasets or when latency is critical. ```swift let config = VecturaConfig( name: "my-database", memoryStrategy: .fullMemory ) let vectura = try await VecturaKit(config: config) ``` -------------------------------- ### Get Document Count in VecturaKit Source: https://github.com/rryam/vecturakit/blob/main/README.md Shows how to retrieve the total number of documents currently stored in the VecturaKit database. ```swift let count = await vectorDB.documentCount print("Database contains (count) documents") ``` -------------------------------- ### ParameterTuningSuite: Optimal Configuration for Indexed Mode Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Focuses on finding optimal configurations for the indexed mode. It includes sweeps for 'candidateMultiplier', 'batchSize', and 'maxConcurrentBatches', as well as tests for multi-dimensional optimization and workload-specific recommendations. Dataset sizes and configurations are reduced to save memory. ```go func (s *ParameterTuningSuite) TestCandidateMultiplierSweep(t *testing.T) { // Test logic for candidate multipliers 5, 10, 15, 20 (1.5K docs) } func (s *ParameterTuningSuite) TestBatchSizeSweep(t *testing.T) { // Test logic for batch sizes 50, 100, 150 (1.5K docs) } func (s *ParameterTuningSuite) TestMaxConcurrentBatchesSweep(t *testing.T) { // Test logic for concurrency 2, 4, 6 (1.5K docs) } func (s *ParameterTuningSuite) TestOptimalParameterCombination(t *testing.T) { // Test logic for multi-dimensional optimization (3 configs, 1.5K docs) } func (s *ParameterTuningSuite) TestWorkloadSpecificRecommendations(t *testing.T) { // Test logic for preset configurations (3 workloads, 1.5K docs) } ``` -------------------------------- ### Get NLContextualEmbedder Model Information in Swift Source: https://github.com/rryam/vecturakit/blob/main/README.md Retrieves model information, including language and dimension, from an NLContextualEmbedder instance. This can be used to verify embedder configuration or obtain details before use. ```swift // Initialize with specific language let embedder = try await NLContextualEmbedder( language: .spanish ) // Get model information let modelInfo = await embedder.modelInfo print("Language: \(modelInfo.language)") if let dimension = modelInfo.dimension { print("Dimension: \(dimension)") } else { print("Dimension: Not yet determined") } ``` -------------------------------- ### Use Custom Storage Provider with VecturaKit Source: https://github.com/rryam/vecturakit/blob/main/README.md Demonstrates how to initialize VecturaKit using a custom storage provider. Any subsequent storage operations on the `vectorDB` instance will be handled by the implemented custom provider. ```swift let config = VecturaConfig(name: "my-db") let customStorage = MyCustomStorageProvider() let vectorDB = try await VecturaKit( config: config, storageProvider: customStorage ) // Use vectorDB normally - all storage operations will use your custom provider let documentId = try await vectorDB.addDocument(text: "Sample text") ``` -------------------------------- ### Configure Automatic Memory Strategy (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Sets up VecturaKit to automatically select between full memory and indexed storage modes based on dataset size. This is the default configuration. ```swift let config = VecturaConfig(name: "my-database") // config.memoryStrategy defaults to .automatic() let vectura = try await VecturaKit(config: config) ``` -------------------------------- ### Implement Custom Search Engine with Swift Source: https://github.com/rryam/vecturakit/blob/main/README.md Demonstrates how to create a custom search engine by conforming to the VecturaSearchEngine protocol in Swift. This allows for specialized search algorithms. It requires the VecturaKit framework. ```swift import Foundation import VecturaKit struct MyCustomSearchEngine: VecturaSearchEngine { func search( query: SearchQuery, storage: VecturaStorage, options: SearchOptions ) async throws -> [VecturaSearchResult] { // Load documents from storage let documents = try await storage.loadDocuments() // Implement your custom search logic // This example does a simple exact text match guard case .text(let queryText) = query else { return [] } let results = documents.filter { $0.text.lowercased().contains(queryText.lowercased()) }.map { VecturaSearchResult( id: $0.id, text: $0.text, score: 1.0, createdAt: $0.createdAt ) } return Array(results.prefix(options.numResults)) } func indexDocument(_ document: VecturaDocument) async throws { // Optional: Update your search engine's internal index } func removeDocument(id: UUID) async throws { // Optional: Remove from your search engine's internal index } } let config = VecturaConfig(name: "my-db") let embedder = SwiftEmbedder(modelSource: .default) let customEngine = MyCustomSearchEngine() let vectorDB = try await VecturaKit( config: config, embedder: embedder, searchEngine: customEngine ) // All searches will use your custom search engine let results = try await vectorDB.search(query: "search query") ``` -------------------------------- ### Configure Indexed Memory Strategy (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Configures VecturaKit to use indexed mode for memory efficiency, suitable for large datasets. Allows tuning of candidateMultiplier for accuracy-performance trade-offs. ```swift let config = VecturaConfig( name: "my-database", memoryStrategy: .indexed( candidateMultiplier: 10 // Search 10× topK candidates ) ) let vectura = try await VecturaKit(config: config) ``` -------------------------------- ### ScalabilitySuite: Performance Scaling Analysis Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Analyzes how different strategies scale from small to large datasets. It includes tests for growth curves of 'fullMemory' and 'indexed' strategies, cross-strategy comparisons, throughput analysis, and memory usage patterns. Dataset sizes have been reduced to fit within memory limits. ```go func (s *ScalabilitySuite) TestFullMemoryScaling(t *testing.T) { // Test logic for fullMemory growth curves (500, 1K, 1.5K docs) } func (s *ScalabilitySuite) TestIndexedStrategyScaling(t *testing.T) { // Test logic for indexed strategy growth curves (500, 1K, 1.5K docs) } func (s *ScalabilitySuite) TestCrossStrategyScaling(t *testing.T) { // Test logic for side-by-side comparison of strategies } func (s *ScalabilitySuite) TestAddDocumentThroughputScaling(t *testing.T) { // Test logic for throughput analysis (500, 1K docs) } func (s *ScalabilitySuite) TestMemoryGrowthPatterns(t *testing.T) { // Test logic for memory usage patterns (500, 1K, 1.5K docs) } ``` -------------------------------- ### Configure and Initialize VecturaKit Database Source: https://github.com/rryam/vecturakit/blob/main/README.md Initializes VecturaKit with a configuration, setting up the database name, directory, search options, and an embedder. It supports default settings or custom options for large-scale datasets using indexed memory strategies. ```swift import Foundation import VecturaKit let config = VecturaConfig( name: "my-vector-db", directoryURL: nil, // Optional custom storage location dimension: nil, // Auto-detect dimension from embedder (recommended) searchOptions: VecturaConfig.SearchOptions( defaultNumResults: 10, minThreshold: 0.7, hybridWeight: 0.5, // Balance between vector and text search k1: 1.2, // BM25 parameters b: 0.75, bm25NormalizationFactor: 10.0 ) ) // Create an embedder (SwiftEmbedder uses swift-embeddings library) let embedder = SwiftEmbedder(modelSource: .default) let vectorDB = try await VecturaKit(config: config, embedder: embedder) ``` ```swift let config = VecturaConfig( name: "my-vector-db", memoryStrategy: .indexed(candidateMultiplier: 10) ) let vectorDB = try await VecturaKit(config: config, embedder: embedder) // Reduced memory footprint with on-demand document loading ``` -------------------------------- ### Run Production Tuning Tests (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Execute benchmark suites focused on production tuning and validation. This includes finding optimal parameters for performance and validating the trade-offs between accuracy and performance. ```bash # Find optimal parameters swift test --filter ParameterTuningSuite.optimalParameterCombination # Validate accuracy trade-offs swift test --filter AccuracyTests.accuracyPerformanceTradeoff ``` -------------------------------- ### Use Release Builds for Tests (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Run Swift tests using a release build configuration, which is generally more memory-efficient than debug builds. This command includes a filter to target a specific benchmark suite. ```bash swift test -c release --filter BenchmarkSuite.smallDatasetFullMemory ``` -------------------------------- ### Run Core Benchmarks and Save Results (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Execute core benchmarks and redirect both standard output and standard error to a file. This is useful for saving benchmark results for comparison and later analysis, especially before submitting a pull request. ```bash # Run core benchmarks (~5 min) swift test --filter BenchmarkSuite # Save results for comparison swift test --filter BenchmarkSuite \ > ArchivedResults/pr_$(date +%Y%m%d).txt 2>&1 ``` -------------------------------- ### Configure Vectura with Full Memory Strategy for Small Datasets (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/TEST_RESULTS_SUMMARY.md This code configures Vectura to use the fullMemory strategy, which is suitable for known small datasets (under 10,000 documents). This strategy loads all documents into RAM at initialization, resulting in zero I/O during search operations and consistent sub-15ms search latency. However, memory usage scales linearly with the document count. ```swift VecturaConfig(name: "my-db", memoryStrategy: .fullMemory) ``` -------------------------------- ### IndexedVecturaStorage Protocol Definition (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/INDEXED_STORAGE_GUIDE.md Defines the `IndexedVecturaStorage` protocol required for custom storage providers to support indexed mode operations in VecturaKit. Includes methods for pagination, document count, candidate searching, and loading documents by ID. ```swift public protocol IndexedVecturaStorage: VecturaStorage { // Pagination func loadDocuments(offset: Int, limit: Int) async throws -> [VecturaDocument] func getTotalDocumentCount() async throws -> Int // Vector indexing func searchCandidates( queryEmbedding: [Float], topK: Int, prefilterSize: Int ) async throws -> [UUID] func loadDocuments(ids: [UUID]) async throws -> [UUID: VecturaDocument] } ``` -------------------------------- ### Configure Vectura with Indexed Memory Strategy (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/TEST_RESULTS_SUMMARY.md This snippet demonstrates how to configure Vectura with the indexed memory strategy, recommended for datasets exceeding 10,000 documents. This configuration includes parameters like candidateMultiplier for recall, batchSize for throughput, and maxConcurrentBatches for parallelism. True indexed behavior requires a custom storage provider like IndexedVecturaStorage, often implemented with SQLite and a vector index like FAISS. ```swift VecturaConfig( name: "large-db", memoryStrategy: . indexed( candidateMultiplier: 10, // 80-90% recall batchSize: 100, // Balanced throughput maxConcurrentBatches: 4 // Good parallelism ) ) ``` -------------------------------- ### Initialize VecturaKit with MLX Embedder in Swift Source: https://context7.com/rryam/vecturakit/llms.txt This Swift code demonstrates how to initialize VecturaKit using Apple's MLX framework for hardware-accelerated embedding generation. It configures the database, creates an `MLXEmbedder` with a specified model, and then initializes `VecturaKit`. Documents are added using the MLX embedder, and searches leverage this hardware acceleration for improved performance. It requires VecturaKit and MLXEmbedders frameworks. ```swift import VecturaKit import VecturaMLXKit import MLXEmbedders Task { do { // Configure database let config = VecturaConfig( name: "mlx-vector-db", dimension: nil // Auto-detect from MLX model ) // Create MLX embedder with Nomic model let embedder = try await MLXEmbedder( configuration: .nomic_text_v1_5 ) // Initialize VecturaKit with MLX embedder let vectorDB = try await VecturaKit( config: config, embedder: embedder ) // Add documents (uses MLX for embedding) let texts = [ "MLX provides GPU-accelerated machine learning on Apple Silicon", "Vector embeddings capture semantic meaning of text", "On-device inference protects user privacy" ] let ids = try await vectorDB.addDocuments(texts: texts) print("Added (ids.count) documents using MLX embedder") // Search with hardware acceleration let results = try await vectorDB.search( query: "machine learning privacy", numResults: 3, threshold: 0.7 ) print("\nMLX-powered search results:") for (index, result) in results.enumerated() { print("[(index + 1)] Score: (result.score)") print(" (result.text)\n") } // Get embedding dimension let dimension = try await embedder.dimension print("MLX embedding dimension: (dimension)") } catch { print("Error with MLX embedder: (error)") } } ``` -------------------------------- ### Swift: Implement Custom In-Memory Storage for VecturaKit Source: https://context7.com/rryam/vecturakit/llms.txt This Swift code defines a custom in-memory storage provider by conforming to the VecturaStorage protocol. It manages documents in dictionaries and arrays for quick access and demonstrates basic CRUD operations. This example shows how to initialize VecturaKit with a custom storage provider for flexible data persistence. ```swift import Foundation import VecturaKit // Custom in-memory storage provider example final class CustomMemoryStorage: VecturaStorage { private var documents: [UUID: VecturaDocument] = [: ] private var documentOrder: [UUID] = [] func createStorageDirectoryIfNeeded() async throws { // No setup needed for in-memory storage print("Memory storage initialized") } func loadDocuments() async throws -> [VecturaDocument] { return documentOrder.compactMap { documents[$0] } } func saveDocument(_ document: VecturaDocument) async throws { if documents[document.id] == nil { documentOrder.append(document.id) } documents[document.id] = document print("Saved document (document.id)") } func deleteDocument(withID id: UUID) async throws { documents.removeValue(forKey: id) documentOrder.removeAll { $0 == id } print("Deleted document (id)") } func updateDocument(_ document: VecturaDocument) async throws { guard documents[document.id] != nil else { throw VecturaError.documentNotFound(document.id) } documents[document.id] = document print("Updated document (document.id)") } func getTotalDocumentCount() async throws -> Int { return documents.count } func saveDocuments(_ documents: [VecturaDocument]) async throws { // Optimized batch save for doc in documents { if self.documents[doc.id] == nil { documentOrder.append(doc.id) } self.documents[doc.id] = doc } print("Batch saved (documents.count) documents") } } // Usage Task { do { let config = VecturaConfig(name: "custom-storage-db") let embedder = SwiftEmbedder(modelSource: .default) let customStorage = CustomMemoryStorage() let vectorDB = try await VecturaKit( config: config, embedder: embedder, storageProvider: customStorage ) // Use database normally - all operations use custom storage let docId = try await vectorDB.addDocument( text: "Test document with custom storage" ) print("Added document: (docId)") let results = try await vectorDB.search(query: "test") print("Search found (results.count) results") let count = try await vectorDB.documentCount print("Total documents: (count)") } catch { print("Error: (error)") } } ``` -------------------------------- ### BenchmarkSuite: Performance Baseline Tests Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Establishes performance baselines for different strategies and dataset sizes. It includes tests for various document counts and optimization strategies like fullMemory, automatic, and indexed modes. Note that dataset sizes have been reduced to accommodate memory constraints. ```go func (s *BenchmarkSuite) BenchmarkSmallDatasetFullMemory(b *testing.B) { b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { // Test logic for 1K docs, fullMemory baseline } } func (s *BenchmarkSuite) BenchmarkSmallDatasetAutomatic(b *testing.B) { b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { // Test logic for 1K docs, automatic strategy } } func (s *BenchmarkSuite) BenchmarkMediumDatasetFullMemory(b *testing.B) { b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { // Test logic for 3K docs, fullMemory baseline } } func (s *BenchmarkSuite) BenchmarkMediumDatasetIndexed(b *testing.B) { b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { // Test logic for 3K docs, indexed mode } } func (s *BenchmarkSuite) BenchmarkCompareStrategies10K(b *testing.B) { b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { // Test logic for direct comparison at 2K docs } } func (s *BenchmarkSuite) BenchmarkCandidateMultiplierComparison(b *testing.B) { b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { // Test logic for candidate multipliers 5, 10, 15, 20 } } func (s *BenchmarkSuite) BenchmarkBatchSizeComparison(b *testing.B) { b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { // Test logic for batch sizes 50, 100, 150, 200 } } ``` -------------------------------- ### Initialize VecturaKit with MLX Embedder in Swift Source: https://github.com/rryam/vecturakit/blob/main/README.md Configures and initializes VecturaKit using Apple's MLX framework via the MLXEmbedder for accelerated on-device ML. Requires VecturaKit, VecturaMLXKit, and MLXEmbedders. ```swift import VecturaKit import VecturaMLXKit import MLXEmbedders let config = VecturaConfig( name: "my-mlx-vector-db", dimension: nil // Auto-detect dimension from MLX embedder ) // Create MLX embedder let embedder = try await MLXEmbedder(configuration: .nomic_text_v1_5) let vectorDB = try await VecturaKit(config: config, embedder: embedder) ``` -------------------------------- ### Import VecturaKit Module Source: https://github.com/rryam/vecturakit/blob/main/README.md Shows the basic Swift import statement required to use the VecturaKit library in your project. ```swift import VecturaKit ``` -------------------------------- ### Run Full Release Validation Suite (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Execute the complete suite of performance tests, typically used for release validation. This comprehensive test run takes approximately one hour and covers all aspects of performance. ```bash # Full suite (~1 hour) swift test --filter PerformanceTests ``` -------------------------------- ### Run Smaller Dataset Tests (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Execute tests using smaller datasets to mitigate CoreML memory exhaustion. This is useful for debugging and running tests on systems with limited RAM. The command filters for specific benchmark suites designed for smaller datasets. ```bash swift test --filter BenchmarkSuite.smallDatasetFullMemory swift test --filter BenchmarkSuite.smallDatasetAutomatic ``` -------------------------------- ### Configure Vectura with Automatic Memory Strategy (Swift) Source: https://github.com/rryam/vecturakit/blob/main/Docs/TEST_RESULTS_SUMMARY.md This snippet shows how to configure Vectura with the default automatic memory strategy. The automatic strategy intelligently selects the fullMemory strategy for smaller datasets, ensuring no performance penalty for decision logic and providing seamless fallback behavior. ```swift VecturaConfig(name: "my-db") // Uses automatic ``` -------------------------------- ### Vectura CLI for Document Management Source: https://github.com/rryam/vecturakit/blob/main/README.md Command-line interface for interacting with the vector database using the default embedding model. Supports adding, searching, updating, deleting, resetting, and running mock data simulations. Dimensions are auto-detected by default. ```bash # Add documents (dimension auto-detected from model) vectura add "First document" "Second document" "Third document" \ --db-name "my-vector-db" # Search documents vectura search "search query" \ --db-name "my-vector-db" \ --threshold 0.7 \ --num-results 5 # Update document vectura update "Updated text content" \ --db-name "my-vector-db" # Delete documents vectura delete \ --db-name "my-vector-db" # Reset database vectura reset \ --db-name "my-vector-db" # Run demo with sample data vectura mock \ --db-name "my-vector-db" \ --threshold 0.7 \ --num-results 10 ``` -------------------------------- ### MLX CLI for Document Management Source: https://github.com/rryam/vecturakit/blob/main/README.md Command-line interface for interacting with the vector database using the MLX embedding backend. Supports adding, searching, updating, deleting, and resetting documents. Threshold is optional. ```bash # Add documents vectura-mlx add "First document" "Second document" "Third document" --db-name "my-mlx-vector-db" # Search documents vectura-mlx search "search query" --db-name "my-mlx-vector-db" --threshold 0.7 --num-results 5 # Update document vectura-mlx update "Updated text content" --db-name "my-mlx-vector-db" # Delete documents vectura-mlx delete --db-name "my-mlx-vector-db" # Reset database vectura-mlx reset --db-name "my-mlx-vector-db" # Run demo with sample data vectura-mlx mock --db-name "my-mlx-vector-db" ``` -------------------------------- ### Run Performance Investigation Tests (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Execute specific benchmark suites for performance investigation. This includes memory profiling, scalability testing, and parameter optimization sweeps to diagnose performance bottlenecks. ```bash # Memory analysis swift test --filter MemoryProfilerSuite.fullMemoryLifecycle # Scaling issues swift test --filter ScalabilitySuite.fullMemoryScaling # Parameter optimization swift test --filter ParameterTuningSuite.candidateMultiplierSweep ``` -------------------------------- ### Implement Custom Storage Provider for VecturaKit Source: https://github.com/rryam/vecturakit/blob/main/README.md Defines a custom storage provider by conforming to the `VecturaStorage` protocol, enabling integration with external storage systems. Includes methods for creating storage, loading, saving, deleting, and updating documents. ```swift import Foundation import VecturaKit final class MyCustomStorageProvider: VecturaStorage { private var documents: [UUID: VecturaDocument] = [: ] func createStorageDirectoryIfNeeded() async throws { // Initialize your storage system } func loadDocuments() async throws -> [VecturaDocument] { // Load documents from your storage return Array(documents.values) } func saveDocument(_ document: VecturaDocument) async throws { // Save document to your storage documents[document.id] = document } func deleteDocument(withID id: UUID) async throws { // Delete document from your storage documents.removeValue(forKey: id) } func updateDocument(_ document: VecturaDocument) async throws { // Update document in your storage documents[document.id] = document } func getTotalDocumentCount() async throws -> Int { // Return total count (optional - protocol provides default implementation) return documents.count } } ``` -------------------------------- ### Run Specific Performance Benchmark Suite Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md This command executes a specific test case within the `BenchmarkSuite` for initial verification and debugging. It filters tests to run only the `smallDatasetFullMemory` benchmark. ```bash swift test --filter BenchmarkSuite.smallDatasetFullMemory ``` -------------------------------- ### Run Swift Tests with Custom Data (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/TestData/README.md This bash command demonstrates how to filter and run specific Swift tests, particularly custom benchmarks, from the command line. It assumes your custom tests are named appropriately. ```bash swift test --filter MyCustomBenchmark ``` -------------------------------- ### Monitor System Memory Usage (Bash) Source: https://github.com/rryam/vecturakit/blob/main/Tests/PerformanceTests/README.md Monitor system memory usage in real-time using the 'watch' and 'top' commands. This is crucial for diagnosing memory-related test failures by observing physical memory consumption during test execution. ```bash # In Terminal 1: watch -n 1 'top -l 1 | grep -A 5 "PhysMem"' # In Terminal 2: swift test --filter BenchmarkSuite.smallDatasetFullMemory ``` -------------------------------- ### Initialize VecturaKit with NaturalLanguage Embedder in Swift Source: https://github.com/rryam/vecturakit/blob/main/README.md Configures and initializes VecturaKit using Apple's NaturalLanguage framework via the NLContextualEmbedder for contextual embeddings without external dependencies. Requires VecturaKit and VecturaNLKit. ```swift import VecturaKit import VecturaNLKit let config = VecturaConfig( name: "my-nl-vector-db", dimension: nil // Auto-detect dimension from NL embedder ) // Create NLContextualEmbedder let embedder = try await NLContextualEmbedder( language: .english ) let vectorDB = try await VecturaKit(config: config, embedder: embedder) ```