SimilaritySearchKit

https://github.com/zachnagengast/similarity-search-kit
Admin
SimilaritySearchKit is a Swift package enabling on-device text embeddings and semantic search...
Tokens:7,430
Snippets:79
Trust Score:9.7
License:Apache-2.0
Update:5 months ago
Show doc for...
Tokens:
Raw
### Save and Load Index with SimilaritySearchKit

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Demonstrates how to create, add items to, save, and load a Similarity Index using JSON or compressed binary storage. Includes getting the index path and estimating memory usage.

```swift
import SimilaritySearchKit

// Create index with JSON storage (default)
let jsonIndex = await SimilarityIndex(
    name: "MySearchIndex",
    vectorStore: JsonStore()
)

// Or use compressed binary storage for smaller file sizes
let binaryIndex = await SimilarityIndex(
    name: "CompressedIndex",
    vectorStore: BinaryStore()
)

// Add items
await jsonIndex.addItem(id: "1", text: "Sample document", metadata: ["type": "demo"])

// Save index to custom directory
let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let savedURL = try jsonIndex.saveIndex(toDirectory: documentsURL, name: "MySearchIndex")
print("Index saved to: \(savedURL.path)")

// Load index from disk
let loadedItems = try jsonIndex.loadIndex(fromDirectory: documentsURL, name: "MySearchIndex")
print("Loaded \(loadedItems?.count ?? 0) items")

// Get index file path
if let indexPath = try jsonIndex.getIndexPath(fromDirectory: documentsURL, name: "MySearchIndex") {
    print("Index path: \(indexPath.path)")
}

// Estimate memory usage
let sizeInBytes = jsonIndex.estimatedSizeInBytes()
print("Estimated size: \(sizeInBytes) bytes")
```

--------------------------------

### Implement Custom Distance Metric Protocol

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Implement the DistanceMetricProtocol to define custom similarity measures. This example shows ManhattanDistance.

```swift
import SimilaritySearchKit

// Custom distance metric implementation
public struct ManhattanDistance: DistanceMetricProtocol {
    public init() {}

    public func distance(between firstEmbedding: [Float], and secondEmbedding: [Float]) -> Float {
        guard firstEmbedding.count == secondEmbedding.count else {
            return Float.greatestFiniteMagnitude
        }

        var distance: Float = 0
        for i in 0..<firstEmbedding.count {
            distance += abs(firstEmbedding[i] - secondEmbedding[i])
        }
        return distance
    }

    public func findNearest(for queryEmbedding: [Float], in neighborEmbeddings: [[Float]], resultsCount: Int) -> [(Float, Int)] {
        let distances = neighborEmbeddings.map { distance(between: queryEmbedding, and: $0) }
        return sortedDistances(distances: distances, topK: resultsCount)
    }
}

// Use custom metric
let index = await SimilarityIndex(
    model: NativeEmbeddings(),
    metric: ManhattanDistance()
)

// Can also override metric at query time
let results = await index.search("query text", metric: CosineSimilarity())
```

--------------------------------

### Implement Custom Vector Store - Swift

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Implement the `VectorStoreProtocol` to create a custom storage backend. This example shows saving and loading index items using JSON encoding and decoding to a custom file extension. Ensure your custom store handles serialization and deserialization correctly.

```swift
import SimilaritySearchKit

// Custom SQLite-based storage example
public class CustomStore: VectorStoreProtocol {
    public init() {}

    public func saveIndex(items: [IndexItem], to url: URL, as name: String) throws -> URL {
        // Implement custom save logic
        // Could save to SQLite, CoreData, CloudKit, etc.
        let fileURL = url.appendingPathComponent("\(name).custom")

        let encoder = JSONEncoder()
        let data = try encoder.encode(items)
        try data.write(to: fileURL)

        return fileURL
    }

    public func loadIndex(from url: URL) throws -> [IndexItem] {
        // Implement custom load logic
        let data = try Data(contentsOf: url)
        let decoder = JSONDecoder()
        return try decoder.decode([IndexItem].self, from: url)
    }

    public func listIndexes(at url: URL) -> [URL] {
        let fileManager = FileManager.default
        do {
            let files = try fileManager.contentsOfDirectory(at: url, includingPropertiesForKeys: nil)
            return files.filter { $0.pathExtension == "custom" }
        } catch {
            return []
        }
    }
}
```

```swift
// Use custom store
let index = await SimilarityIndex(
    name: "CustomIndex",
    vectorStore: CustomStore()
)
```

--------------------------------

### Use MiniLMEmbeddings for Document Similarity

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Shows how to initialize the all-MiniLM-L6-v2 model and create a Similarity Index for general document similarity. Includes adding documents and searching for similar content.

```swift
import SimilaritySearchKit
import SimilaritySearchKitMiniLMAll

// Initialize MiniLM model for general text similarity
let miniLMModel = MiniLMEmbeddings()

// Create index for document similarity
let docIndex = await SimilarityIndex(
    name: "DocumentIndex",
    model: miniLMModel,
    metric: CosineSimilarity()
)

// Add documents
await docIndex.addItem(
    id: "doc1",
    text: "Climate change is affecting global weather patterns.",
    metadata: ["category": "environment"]
)

await docIndex.addItem(
    id: "doc2",
    text: "Rising temperatures cause more extreme weather events.",
    metadata: ["category": "environment"]
)

await docIndex.addItem(
    id: "doc3",
    text: "The stock market showed gains in tech sector.",
    metadata: ["category": "finance"]
)

// Find similar documents
let similar = await docIndex.search("global warming effects", top: 2)
for result in similar {
    print("\(result.id): \(result.score) - \(result.text)")
}
```

--------------------------------

### Utilize Distance Metrics in SimilaritySearchKit

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Demonstrates the usage of CosineSimilarity, DotProduct, and EuclideanDistance metrics. Shows how to calculate distances between vectors and find nearest neighbors.

```swift
import SimilaritySearchKit

// Cosine Similarity - best for sparse embeddings and when magnitude matters
let cosineMetric = CosineSimilarity()

// Dot Product - best for dense embeddings when magnitude doesn't matter
let dotProductMetric = DotProduct()

// Euclidean Distance - best for well-distributed embeddings in Euclidean space
let euclideanMetric = EuclideanDistance()

// Use with index
let index = await SimilarityIndex(
    model: NativeEmbeddings(),
    metric: cosineMetric
)

// Calculate distance between two vectors directly
let embedding1: [Float] = [0.1, 0.2, 0.3, 0.4]
let embedding2: [Float] = [0.15, 0.25, 0.35, 0.45]

let cosineScore = cosineMetric.distance(between: embedding1, and: embedding2)
let dotScore = dotProductMetric.distance(between: embedding1, and: embedding2)
let euclideanDist = euclideanMetric.distance(between: embedding1, and: embedding2)

print("Cosine similarity: \(cosineScore)")
print("Dot product: \(dotScore)")
print("Euclidean distance: \(euclideanDist)")

// Find nearest neighbors in a set of embeddings
let queryEmbedding: [Float] = [0.1, 0.2, 0.3, 0.4]
let neighborEmbeddings: [[Float]] = [
    [0.1, 0.2, 0.3, 0.4],
    [0.5, 0.6, 0.7, 0.8],
    [0.15, 0.25, 0.35, 0.45]
]

let nearest = cosineMetric.findNearest(
    for: queryEmbedding,
    in: neighborEmbeddings,
    resultsCount: 2
)

for (score, index) in nearest {
    print("Index \(index): score \(score)")
}
```

--------------------------------

### Use DistilbertEmbeddings for Q&A Search

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Demonstrates initializing the DistilBERT model and creating a Similarity Index optimized for question-answering tasks. Includes adding knowledge base content and querying.

```swift
import SimilaritySearchKit
import SimilaritySearchKitDistilbert

// Initialize DistilBERT model (requires macOS 13.0+ / iOS 16.0+)
let distilbertModel = DistilbertEmbeddings()

// Create index optimized for Q&A
let qaIndex = await SimilarityIndex(
    name: "QAIndex",
    model: distilbertModel,
    metric: CosineSimilarity()
)

// Add knowledge base content
await qaIndex.addItem(
    id: "faq-1",
    text: "To reset your password, click on 'Forgot Password' on the login page.",
    metadata: ["topic": "authentication"]
)

await qaIndex.addItem(
    id: "faq-2",
    text: "Billing cycles run from the 1st to the last day of each month.",
    metadata: ["topic": "billing"]
)

// Query with natural questions
let results = await qaIndex.search("How do I change my password?", top: 1)
print("Best match: \(results.first?.text ?? "No results")")
// Output: Best match: To reset your password, click on 'Forgot Password' on the login page.
```

--------------------------------

### Import SimilaritySearchKit Framework

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Import the SimilaritySearchKit framework at the beginning of your Swift file to use its functionalities.

```swift
import SimilaritySearchKit
```

--------------------------------

### Initialize and Use SimilarityIndex

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Initializes a SimilarityIndex with a specific model and metric, adds items, and performs a semantic search. Ensure the necessary model and metric classes are imported.

```swift
import SimilaritySearchKit
import SimilaritySearchKitMiniLMAll

// Initialize with custom model and metric
let similarityIndex = await SimilarityIndex(
    name: "MyDocumentIndex",
    model: MiniLMEmbeddings(),
    metric: CosineSimilarity()
)

// Add items to the index
await similarityIndex.addItem(
    id: "doc1",
    text: "Metal was released in June 2014.",
    metadata: ["source": "apple-docs.pdf", "page": "42"]
)

await similarityIndex.addItem(
    id: "doc2",
    text: "Swift is a powerful programming language for iOS development.",
    metadata: ["source": "swift-guide.pdf", "page": "1"]
)

await similarityIndex.addItem(
    id: "doc3",
    text: "CoreML enables on-device machine learning inference.",
    metadata: ["source": "ml-docs.pdf", "page": "15"]
)

// Search the index
let results = await similarityIndex.search("When was metal released?", top: 3)

// Process results
for result in results {
    print("ID: \(result.id)")
    print("Score: \(result.score)")
    print("Text: \(result.text)")
    print("Metadata: \(result.metadata)")
}
// Output:
// ID: doc1
// Score: 0.86216
// Text: Metal was released in June 2014.
// Metadata: ["source": "apple-docs.pdf", "page": "42"]

```

--------------------------------

### Initialize MultiQAMiniLMEmbeddings Model

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Initializes the MultiQAMiniLMEmbeddings model for question-answering tasks. Use DotProduct for the metric with this model. Adds items to the index and performs a search.

```swift
import SimilaritySearchKit
import SimilaritySearchKitMiniLMMultiQA

// Initialize Multi-QA MiniLM model
let multiQAModel = MultiQAMiniLMEmbeddings()

let index = await SimilarityIndex(
    model: multiQAModel,
    metric: DotProduct()  // Dot product works well with this model
)

// Add FAQ entries
await index.addItem(
    id: "q1",
    text: "Returns are accepted within 30 days of purchase with original receipt.",
    metadata: ["department": "customer-service"]
)

await index.addItem(
    id: "q2",
    text: "Free shipping is available on orders over $50.",
    metadata: ["department": "shipping"]
)

// Search with questions
let answers = await index.search("What is your return policy?")
print("Answer: \(answers.first?.text ?? "Not found")")
```

--------------------------------

### Save and Load Index Items

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Implement this protocol to manage the persistence of index items. The default uses JSON files, but custom implementations can use any storage mechanism.

```swift
func saveIndex(items: [IndexItem], to url: URL, as name: String) throws -> URL
```

```swift
func loadIndex(from url: URL) throws -> [IndexItem]
```

```swift
func listIndexes(at url: URL) -> [URL]
```

--------------------------------

### Manage Index Items (CRUD)

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Demonstrates basic Create, Read, Update, and Delete operations for items within a SimilarityIndex. Includes methods for retrieving single items, sampling, updating, removing, and clearing the entire index.

```swift
import SimilaritySearchKit

let index = await SimilarityIndex()

// Create
await index.addItem(
    id: "article-1",
    text: "Original content here",
    metadata: ["version": "1.0"]
)

// Read
if let item = index.getItem(id: "article-1") {
    print("Found: \(item.text)")
    print("Metadata: \(item.metadata)")
}

// Sample first N items
if let sample = index.sample(5) {
    print("Sample count: \(sample.count)")
}

// Update
index.updateItem(
    id: "article-1",
    text: "Updated content here",
    metadata: ["version": "2.0"]
)

// Delete single item
index.removeItem(id: "article-1")

// Delete all items
index.removeAll()

```

--------------------------------

### Create SimilarityIndex Instance

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Instantiate SimilarityIndex with a chosen distance metric and embedding model. Ensure the model is available and imported.

```swift
let similarityIndex = await SimilarityIndex(
    model: NativeEmbeddings(),
    metric: CosineSimilarity()
)
```

--------------------------------

### Generate LLM Prompts from Search Results

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Use this to generate prompts for LLMs from search results, suitable for RAG applications. Ensure the SimilaritySearchKit is imported.

```swift
import SimilaritySearchKit

let index = await SimilarityIndex(
    model: NativeEmbeddings(),
    metric: CosineSimilarity()
)

// Add knowledge base
await index.addItem(
    id: "1",
    text: "The Eiffel Tower was completed in 1889.",
    metadata: ["source": "history.txt"]
)

await index.addItem(
    id: "2",
    text: "It stands 330 meters tall including antennas.",
    metadata: ["source": "facts.txt"]
)

// Search and get results
let query = "How tall is the Eiffel Tower?"
let results = await index.search(query, top: 2)

// Generate combined results string
let combinedText = SimilarityIndex.combinedResultsString(results)
print("Combined results:\n\(combinedText)")

// Generate LLM prompt for question answering
let llmPrompt = SimilarityIndex.exportLLMPrompt(query: query, results: results)
print("LLM Prompt:\n\(llmPrompt)")
```

--------------------------------

### Add Items in Batch with Progress

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Adds multiple items to the index concurrently using provided arrays for IDs, texts, and metadata. Includes a progress callback to track indexing status.

```swift
import SimilaritySearchKit

let similarityIndex = await SimilarityIndex(
    model: NativeEmbeddings(),
    metric: CosineSimilarity()
)

let ids = ["id1", "id2", "id3"]
let texts = [
    "The quick brown fox jumps over the lazy dog.",
    "Machine learning models can process natural language.",
    "Swift provides memory safety without garbage collection."
]
let metadata: [[String: String]] = [
    ["category": "animals"],
    ["category": "technology"],
    ["category": "programming"]
]

// Add items with progress callback
await similarityIndex.addItems(
    ids: ids,
    texts: texts,
    metadata: metadata,
    onProgress: { completedId in
        print("Indexed item: \(completedId)")
    }
)

print("Total items indexed: \(similarityIndex.indexItems.count)")
// Output: Total items indexed: 3

```

--------------------------------

### Tokenize and Detokenize Text

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Implement this protocol for custom text tokenization and detokenization. Use this for models with unique tokenization requirements.

```swift
func tokenize(text: String) -> [String]
```

```swift
func detokenize(tokens: [String]) -> String
```

--------------------------------

### Add SimilaritySearchKit Dependency to Package.swift

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Add this line to your Package.swift dependencies array to include the SimilaritySearchKit package.

```swift
.package(url: "https://github.com/ZachNagengast/similarity-search-kit.git", from: "0.0.1")
```

--------------------------------

### Tokenize Text with BertTokenizer

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Tokenizes text using BERT's WordPiece algorithm. Includes methods for tokenizing to string tokens, building model tokens with special tokens, creating model inputs for CoreML, and detokenizing back to text.

```swift
import SimilaritySearchKit

let tokenizer = BertTokenizer()

// Tokenize text to string tokens
let text = "Hello, how are you doing today?"
let tokens = tokenizer.tokenize(text: text)
print("Tokens: \(tokens)")
// Output: ["hello", ",", "how", "are", "you", "doing", "today", "?"]

// Build model tokens with [CLS], [SEP], and padding
let modelTokens = tokenizer.buildModelTokens(sentence: text)
print("Model token count: \(modelTokens.count)")  // Always 512

// Build MLMultiArray inputs for CoreML models
let (inputIds, attentionMask) = tokenizer.buildModelInputs(from: modelTokens)

// Detokenize back to text (may lose some information)
let reconstructed = tokenizer.detokenize(tokens: tokens)
print("Reconstructed: \(reconstructed)")
```

--------------------------------

### Use NativeEmbeddings with SimilaritySearchKit

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Shows how to utilize Apple's NaturalLanguage embedding models for sentence and word embeddings. Includes direct encoding and sparse BM25 embeddings.

```swift
import SimilaritySearchKit
import NaturalLanguage

// Sentence embeddings (default)
let sentenceModel = NativeEmbeddings(
    language: .english,
    type: .sentenceEmbedding
)

// Word embeddings for different use cases
let wordModel = NativeEmbeddings(
    language: .english,
    type: .wordEmbedding
)

// Use with SimilarityIndex
let index = await SimilarityIndex(
    model: sentenceModel,
    metric: CosineSimilarity()
)

// Direct encoding
let embedding = await sentenceModel.encode(sentence: "Hello world")
print("Embedding dimension: \(embedding?.count ?? 0)")

// Sparse BM25 embeddings for keyword-based search
let sentences = ["The cat sat on the mat.", "Dogs are loyal pets."]
if let sparseEmbeddings = NativeEmbeddings.encodeSparse(sentences: sentences) {
    for (index, sparse) in sparseEmbeddings.enumerated() {
        print("Sentence \(index): \(sparse.count) non-zero values")
    }
}
```

--------------------------------

### Add Target Dependency for SimilaritySearchKit

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Specify the SimilaritySearchKit targets your project depends on in your Package.swift file. Omit model-specific targets if not needed to reduce binary size.

```swift
.target(name: "YourTarget", dependencies: [
    "SimilaritySearchKit", 
    "SimilaritySearchKitDistilbert", 
    "SimilaritySearchKitMiniLMMultiQA", 
    "SimilaritySearchKitMiniLMAll"
])
```

--------------------------------

### Implement Custom Embeddings Protocol

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Implement the EmbeddingsProtocol to integrate your own embedding models. Requires importing SimilaritySearchKit and CoreML. Ensure your model is compatible with MLModel.

```swift
import SimilaritySearchKit
import CoreML

// Custom embedding model implementation
@available(macOS 11.0, iOS 15.0, *)
public class CustomEmbeddings: EmbeddingsProtocol {
    public typealias TokenizerType = BertTokenizer
    public typealias ModelType = MLModel  // Your CoreML model type

    public let tokenizer: BertTokenizer
    public let model: MLModel

    public init(modelURL: URL) throws {
        self.tokenizer = BertTokenizer()
        self.model = try MLModel(contentsOf: modelURL)
    }

    public func encode(sentence: String) async -> [Float]? {
        // Tokenize input
        let tokens = tokenizer.buildModelTokens(sentence: sentence)
        let (inputIds, attentionMask) = tokenizer.buildModelInputs(from: tokens)

        // Run inference with your model
        // Return embedding as [Float] array

        // Placeholder implementation
        return Array(repeating: 0.0, count: 384)
    }
}

// Use custom model with SimilarityIndex
// let customModel = try CustomEmbeddings(modelURL: modelURL)
// let index = await SimilarityIndex(model: customModel, metric: CosineSimilarity())
```

--------------------------------

### Implement Text Splitting with TokenSplitter

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Splits long documents into smaller chunks using a token-based strategy. Configurable with chunk size and overlap size. May lose punctuation.

```swift
import SimilaritySearchKit

// Token-based splitter - fast but may lose some punctuation
let tokenizer = BertTokenizer()
let tokenSplitter = TokenSplitter(withTokenizer: tokenizer)

let longText = """
Machine learning is a subset of artificial intelligence.
It enables computers to learn from data without explicit programming.
Neural networks are a key component of modern ML systems.
"""

let (chunks, tokens) = tokenSplitter.split(
    text: longText,
    chunkSize: 50,  // Max 510 tokens per chunk
    overlapSize: 0
)

print("Number of chunks: \(chunks.count)")
for (index, chunk) in chunks.enumerated() {
    print("Chunk \(index): \(chunk)")
}
```

--------------------------------

### Customizable Protocols

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

SimilaritySearchKit allows for customization of its core components by implementing specific protocols. This section details the available protocols for embeddings, distance metrics, text splitting, tokenization, and vector storage.

```APIDOC
## EmbeddingsProtocol

### Description
Accepts a string and returns an array of floats representing the embedding of the input text.

### Method Signature
```swift
func encode(sentence: String) async -> [Float]?
```

## DistanceMetricProtocol

### Description
Accepts a query embedding vector and a list of embeddings vectors and returns a tuple of the distance metric score and index of the nearest neighbor.

### Method Signature
```swift
func findNearest(for queryEmbedding: [Float], in neighborEmbeddings: [[Float]], resultsCount: Int) -> [(Float, Int)]
```

## TextSplitterProtocol

### Description
Splits a string into chunks of a given size, with a given overlap. This is useful for splitting long documents into smaller chunks for embedding. It returns the list of chunks and an optional list of tokensIds for each chunk.

### Method Signature
```swift
func split(text: String, chunkSize: Int, overlapSize: Int) -> ([String], [[String]]?)
```

## TokenizerProtocol

### Description
Tokenizes and detokenizes text. Use this for custom models that use different tokenizers than are available in the current list.

### Method Signatures
```swift
func tokenize(text: String) -> [String]
func detokenize(tokens: [String]) -> String
```

## VectorStoreProtocol

### Description
Save and load index items. The default implementation uses JSON files, but this can be overriden to use any storage mechanism.

### Method Signatures
```swift
func saveIndex(items: [IndexItem], to url: URL, as name: String) throws -> URL
func loadIndex(from url: URL) throws -> [IndexItem]
func listIndexes(at url: URL) -> [URL]
```
```

--------------------------------

### Split Text into Chunks

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Implement this protocol for custom text splitting. It divides a string into specified chunk sizes with overlap, optionally returning token IDs for each chunk.

```swift
func split(text: String, chunkSize: Int, overlapSize: Int) -> ([String], [[String]]?)
```

--------------------------------

### Find Nearest Neighbors

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Implement this protocol to define custom logic for finding the nearest neighbors. It takes a query embedding and a list of neighbor embeddings, returning scores and indices.

```swift
func findNearest(for queryEmbedding: [Float], in neighborEmbeddings: [[Float]], resultsCount: Int) -> [(Float, Int)]
```

--------------------------------

### Search Similarity Index

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Query the SimilarityIndex with a given text string to find the most similar items. The results include item IDs, similarity scores, and associated metadata.

```swift
let results = await similarityIndex.search("When was metal released?")
print(results)
```

--------------------------------

### Add Item to Similarity Index

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Add text content to the SimilarityIndex. Each item requires a unique ID, the text itself, and optional metadata.

```swift
await similarityIndex.addItem(
    id: "id1", 
    text: "Metal was released in June 2014.", 
    metadata: ["source": "example.pdf"]
)
```

--------------------------------

### Implement Text Splitting with RecursiveTokenSplitter

Source: https://context7.com/zachnagengast/similarity-search-kit/llms.txt

Splits long documents into smaller chunks using a recursive token-based strategy, preserving punctuation and structure better than the basic TokenSplitter. Configurable with chunk size.

```swift
import SimilaritySearchKit

// Recursive splitter - preserves punctuation and structure better
let tokenizer = BertTokenizer()
let recursiveSplitter = RecursiveTokenSplitter(withTokenizer: tokenizer)

let longText = """
Machine learning is a subset of artificial intelligence.
It enables computers to learn from data without explicit programming.
Neural networks are a key component of modern ML systems.
"""

let (recursiveChunks, recursiveTokens) = recursiveSplitter.split(
    text: longText,
    chunkSize: 50
)

print("Recursive chunks: \(recursiveChunks.count)")
```

--------------------------------

### Encode Text to Embedding

Source: https://github.com/zachnagengast/similarity-search-kit/blob/main/README.md

Implement this protocol to provide custom text embedding functionality. It accepts a string and returns an optional array of floats representing the embedding.

```swift
func encode(sentence: String) async -> [Float]?
```
Tokens:
Raw