Blaze

https://github.com/wizenheimer/blaze
Admin
Blaze is a high-performance full-text search engine in Go, offering fast search capabilities with...
Tokens:26,673
Snippets:140
Trust Score:9.2
License:MIT
Update:1 year ago
Show doc for...
Tokens:
Raw
### Query Builder Quick Start Examples

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates common query patterns using the Blaze Query Builder.

```APIDOC
## Query Builder Quick Start Examples

### Single Term Query

Find all documents containing a specific term.

```go
// Find all documents containing "machine"
results := blaze.NewQueryBuilder(idx).
    Term("machine").
    Execute()

fmt.Printf("Found %d documents\n", results.GetCardinality())
```

### AND Query

Find documents that contain all specified terms.

```go
// Find documents with BOTH "machine" AND "learning"
results := blaze.NewQueryBuilder(idx).
    Term("machine").
    And().
    Term("learning").
    Execute()
```

### OR Query

Find documents that contain at least one of the specified terms.

```go
// Find documents with "python" OR "javascript"
results := blaze.NewQueryBuilder(idx).
    Term("python").
    Or().
    Term("javascript").
    Execute()
```

### NOT Query

Find documents that contain a term but exclude another.

```go
// Find documents with "python" but NOT "snake"
results := blaze.NewQueryBuilder(idx).
    Term("python").
    And().Not().
    Term("snake").
    Execute()
```

### Complex Grouped Query

Construct queries with precedence using grouping.

```go
// (machine OR deep) AND learning
results := blaze.NewQueryBuilder(idx).
    Group(func(q *blaze.QueryBuilder) {
        q.Term("machine").Or().Term("deep")
    }).
    And().
    Term("learning").
    Execute()
```

### Phrase Query

Find exact phrase matches.

```go
// Find exact phrase "machine learning"
results := blaze.NewQueryBuilder(idx).
    Phrase("machine learning").
    Execute()
```

### BM25 Ranked Results

Execute queries and retrieve results ranked by BM25 relevance.

```go
// Get top 10 results ranked by relevance
matches := blaze.NewQueryBuilder(idx).
    Term("machine").
    And().
    Term("learning").
    ExecuteWithBM25(10)

for _, match := range matches {
    fmt.Printf("Doc %d: score=%.2f\n", match.DocID, match.Score)
}
```
```

--------------------------------

### Install Blaze Go Package

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This command installs the Blaze Go package using the go get command. Ensure you have Go installed and configured in your environment.

```bash
go get github.com/wizenheimer/blaze
```

--------------------------------

### Convenience Functions for Simple Searches in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This Go example demonstrates the use of convenience functions for simpler search scenarios. It shows a 'good' example using `blaze.AllOf` for a concise and readable query, contrasting it with a more verbose manual query construction.

```go
// Good: Clean and readable
results := blaze.AllOf(idx, "python", "django", "web")

// Bad: Verbose for simple case
results := blaze.NewQueryBuilder(idx).
    Term("python").And().Term("django").And().Term("web").
    Execute()
```

--------------------------------

### Go Example: BM25 Parameter Tuning for Different Domains

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Provides examples of setting BM25 parameters K1 and B for specific document types like academic papers and short messages. Default values are also shown.

```go
// Academic papers (long documents, repeated terms important)
idx.BM25Params.K1 = 2.0
idx.BM25Params.B = 0.5

// Short messages (length less important)
idx.BM25Params.K1 = 1.2
idx.BM25Params.B = 0.3

// Default (works well for most cases)
idx.BM25Params.K1 = 1.5
idx.BM25Params.B = 0.75
```

--------------------------------

### Clone Repository and Install Dependencies (Bash)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Provides essential bash commands for setting up the Blaze project development environment. It includes cloning the repository, navigating into the project directory, and installing necessary dependencies using `make deps`.

```bash
# Clone repository
git clone https://github.com/wizenheimer/blaze.git
cd blaze

# Install dependencies
make deps
```

--------------------------------

### Query Builder Execution Trace Example

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Details the step-by-step execution of a sample query using the Query Builder, including sub-query execution and bitmap operations (Union, Intersection).

```text
Query: NewQueryBuilder(idx).
         Group(func(q) { q.Term("machine").Or().Term("deep") }).
         And().
         Term("learning").
         Execute()

Step 1: Group(func(q) { ... })
    Result: {1,2,3,4,5,6,7,8,9,10}

Step 2: And()
    → Push operator: AND

Step 3: Term("learning")
    → Lookup: DocBitmaps["learning"]
    → Push: {1,2,4,5,6,7,8,9,10}

Step 4: Execute()
    → Intersect: {1,2,4,5,6,7,8,9,10}
    → Return final bitmap
```

--------------------------------

### Combining Phrases and Terms Strategically in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This Go example shows how to effectively combine exact phrases with individual terms for more precise searches. It presents a 'good' approach using `Phrase` and `Term` for flexibility, and a 'bad' example that is overly restrictive by requiring an exact phrase match.

```go
// Good: Exact phrase + related term
qb.Phrase("machine learning").And().Term("python")

// Bad: Overly restrictive
qb.Phrase("machine learning python")  // Requires exact phrase
```

--------------------------------

### Go Quick Start: Indexing and Searching Documents

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates the basic usage of Blaze by creating an inverted index, indexing several documents, and then performing a proximity search for "quick brown". The output shows the document IDs and their relevance scores.

```go
package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Create a new inverted index
    idx := blaze.NewInvertedIndex()

    // Index some documents
    idx.Index(1, "The quick brown fox jumps over the lazy dog")
    idx.Index(2, "A quick brown dog runs fast")
    idx.Index(3, "The lazy cat sleeps all day")

    // Search for documents containing "quick" and "brown"
    matches := idx.RankProximity("quick brown", 10)

    // Print results
    for _, match := range matches {
        fmt.Printf("Document %d (score: %.2f)\n",
            int(match.Offsets[0].DocumentID),
            match.Score)
    }
}
```

--------------------------------

### Create Analyzer Configurations for Different Matching Strategies in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Provides examples of creating AnalyzerConfig instances for various matching strategies: exact, fuzzy, and code search. These configurations demonstrate how to adjust parameters like MinTokenLength, EnableStemming, and EnableStopwords.

```go
// Exact matching (no stemming, keep all words)
exactConfig := blaze.AnalyzerConfig{
    MinTokenLength:  1,
    EnableStemming:  false,
    EnableStopwords: false,
}

// Fuzzy matching (aggressive stemming)
fuzzyConfig := blaze.AnalyzerConfig{
    MinTokenLength:  2,
    EnableStemming:  true,
    EnableStopwords: true,
}

// Code search (no stemming, no stopwords, longer tokens)
codeConfig := blaze.AnalyzerConfig{
    MinTokenLength:  3,
    EnableStemming:  false,
    EnableStopwords: false,
}
```

--------------------------------

### Persistence with Serialization using Go Blaze

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates how to index documents, serialize the index to binary data, save it to a file, and then load it back to use for searching. This example showcases the persistence capabilities of the Blaze library.

```go
package main

import (
    "fmt"
    "os"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Build and save index
    idx := blaze.NewInvertedIndex()
    idx.Index(1, "machine learning algorithms")
    idx.Index(2, "deep learning neural networks")
    idx.Index(3, "natural language processing")

    // Serialize to binary
    data, err := idx.Encode()
    if err != nil {
        panic(err)
    }

    // Save to file
    err = os.WriteFile("search_index.bin", data, 0644)
    if err != nil {
        panic(err)
    }
    fmt.Println("Index saved to search_index.bin")

    // Load index from file
    loadedData, err := os.ReadFile("search_index.bin")
    if err != nil {
        panic(err)
    }

    loadedIdx := blaze.NewInvertedIndex()
    err = loadedIdx.Decode(loadedData)
    if err != nil {
        panic(err)
    }

    // Use loaded index
    results := loadedIdx.RankProximity("learning", 5)
    fmt.Printf("Found %d documents\n", len(results))
}
```

--------------------------------

### Go Example: Tuning BM25 Parameters in Blaze

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates how to adjust BM25 parameters (K1 and B) before indexing documents in Blaze. This allows customization of ranking behavior for different domain characteristics.

```go
idx := blaze.NewInvertedIndex()

// Adjust BM25 parameters before indexing
idx.BM25Params.K1 = 2.0  // Higher = less saturation (more weight to TF)
idx.BM25Params.B = 0.5   // Lower = less length penalty

// Now index and search
idx.Index(1, "document content")
results := idx.RankBM25("query", 10)
```

--------------------------------

### Limit Result Set Size (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Illustrates the importance of limiting the number of search results returned. The good example shows setting a reasonable maximum, while the bad example demonstrates an inappropriate use of `math.MaxInt32`, which could lead to excessive resource consumption.

```go
// Good: Limit results
results := idx.RankBM25("search query", 100)

// Bad: Could return millions of results
results := idx.RankBM25("search query", math.MaxInt32)
```

--------------------------------

### Go: Blaze Query Builder Core Methods - Execute

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates the `Execute` method, which runs the constructed query and returns a `roaring.Bitmap` containing the IDs of the matching documents. It also shows how to get the cardinality (count) of the results.

```go
results := qb.Execute()
docCount := results.GetCardinality()
```

--------------------------------

### Go Example Test Function

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This Go code snippet demonstrates how to write a unit test for search functionality using the blaze library. It shows the process of initializing an inverted index, indexing test documents, and then performing a phrase search to verify the results against expected outcomes.

```go
import (
    "testing"

    "github.com/wizenheimer/blaze"
)

func TestSearchFunctionality(t *testing.T) {
    idx := blaze.NewInvertedIndex()

    // Index test documents
    idx.Index(1, "the quick brown fox")
    idx.Index(2, "the lazy brown dog")

    // Test phrase search
    matches := idx.FindAllPhrases("brown fox", blaze.BOFDocument)

    if len(matches) != 1 {
        t.Errorf("Expected 1 match, got %d", len(matches))
    }

    if int(matches[0][0].DocumentID) != 1 {
        t.Errorf("Expected document 1, got %d", int(matches[0][0].DocumentID))
    }
}
```

--------------------------------

### Commit Message Conventions (Text)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Outlines the recommended format for commit messages in the Blaze project, emphasizing descriptive messages for better code history tracking. It provides examples of 'Good' and 'Bad' commit messages to guide developers.

```text
Good:
- "feat: Add proximity ranking algorithm"
- "feat: Handle empty query in RankProximity"
- "fix: Reduce allocations in skip list insert"

Bad:
- "Update code"
- "Fix bug uwu"
- "WIP"
```

--------------------------------

### Building a Search Engine with Go Blaze

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Provides a complete example of building a simple, interactive search engine using the Blaze library. It indexes a collection of documents and allows users to enter queries, with results ranked using the BM25 algorithm.

```go
package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Create index
    idx := blaze.NewInvertedIndex()

    // Index some documents
    docs := map[int]string{
        1: "Go is an open source programming language that makes it easy to build simple, reliable, and efficient software",
        2: "Python is a programming language that lets you work quickly and integrate systems more effectively",
        3: "JavaScript is a programming language that conforms to the ECMAScript specification",
        4: "Rust is a multi-paradigm programming language focused on performance and safety",
        5: "Java is a class-based, object-oriented programming language designed for portability",
    }

    for id, doc := range docs {
        idx.Index(id, doc)
    }

    // Interactive search
    scanner := bufio.NewScanner(os.Stdin)

    for {
        fmt.Print("\nSearch query (or 'quit' to exit): ")
        if !scanner.Scan() {
            break
        }

        query := strings.TrimSpace(scanner.Text())
        if query == "quit" {
            break
        }

        if query == "" {
            continue
        }

        // Perform search using BM25
        results := idx.RankBM25(query, 5)

        if len(results) == 0 {
            fmt.Println("No results found")
            continue
        }

        // Display results
        fmt.Printf("\nFound %d result(s):\n", len(results))
        for i, match := range results {
            fmt.Printf("\n%d. Document %d (Score: %.3f)\n", i+1, match.DocID, match.Score)
            fmt.Printf("   %s\n", docs[match.DocID])
        }
    }
}
```

--------------------------------

### Index Source Code Files and Search in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Illustrates how to index Go source code files within a directory. It walks through a directory, reads `.go` files, and indexes their content using a custom analyzer configuration that disables stemming and stopwords. Search examples using BM25 and proximity are also shown.

```go
import (
    "os"
    "path/filepath"
    "strings"
    // Assuming blaze is imported
)

func IndexCodebase(rootDir string) (*blaze.InvertedIndex, error) {
    idx := blaze.NewInvertedIndex()
    fileID := 1

    // Custom config for code (no stemming, keep all words)
    config := blaze.AnalyzerConfig{
        MinTokenLength:  2,
        EnableStemming:  false,
        EnableStopwords: false,
    }

    err := filepath.Walk(rootDir, func(path string, info os.FileInfo, err error) error {
        if err != nil || info.IsDir() {
            return err
        }

        // Only index Go files
        if !strings.HasSuffix(path, ".go") {
            return nil
        }

        content, err := os.ReadFile(path)
        if err != nil {
            return err
        }

        // Use custom analyzer
        tokens := blaze.AnalyzeWithConfig(string(content), config)
        // ... index tokens ...

        fileID++
        return nil
    })

    return idx, err
}

// BM25: Find files with frequent mentions of a function/variable
// bm25Results := idx.RankBM25("http.Handler", 20)

// Proximity: Find exact API patterns (e.g., function calls with parameters)
// Better for finding "http.HandleFunc" as a specific pattern
// proximityResults := idx.RankProximity("http HandleFunc", 20)

```

--------------------------------

### Using Groups for Clear Precedence in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This Go snippet highlights the best practice of using query groups to ensure clear logical precedence in complex boolean queries. It shows a 'good' example with explicit grouping for readability and a 'bad' example that could be ambiguous.

```go
// Good: Clear precedence with groups
qb.Group(func(q *blaze.QueryBuilder) {
    q.Term("a").Or().Term("b")
}).And().Term("c")

// Bad: Ambiguous without groups
qb.Term("a").Or().Term("b").And().Term("c")  // Is this (a OR b) AND c or a OR (b AND c)?
```

--------------------------------

### Go: Convenience Functions - AllOf, AnyOf, TermExcluding

Source: https://github.com/wizenheimer/blaze/blob/main/public/index.html

Provides examples of using Blaze's convenience functions in Go for common search patterns. `AllOf` performs an AND operation, `AnyOf` performs an OR operation, and `TermExcluding` combines a term search with an exclusion.

```go
// AllOf: Documents containing ALL terms (AND operation)
results := blaze.AllOf(idx, "machine", "learning", "python")

// AnyOf: Documents containing ANY term (OR operation)
results := blaze.AnyOf(idx, "cat", "dog", "bird")

// TermExcluding: Term with exclusion (AND NOT operation)
results := blaze.TermExcluding(idx, "python", "snake")
```

--------------------------------

### Go: Query Builder - BM25 Ranked Results

Source: https://github.com/wizenheimer/blaze/blob/main/public/index.html

This Go example showcases how to execute a query using the Blaze Query Builder and retrieve the results ranked by BM25 relevance score. It allows specifying the number of top results to return.

```go
// Get top 10 results ranked by relevance
matches := blaze.NewQueryBuilder(idx).
	Term("machine").
	And().
	Term("learning").
	ExecuteWithBM25(10)

for _, match := range matches {
	fmt.Printf("Doc %d: score=%.2f\n", match.DocID, match.Score)
}
```

--------------------------------

### Broad to Narrow Query in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This Go code snippet demonstrates how to build a query that starts broad and then narrows down the search results using specific criteria. It finds programming content related to Python or JavaScript while excluding beginner material.

```go
results := blaze.NewQueryBuilder(idx).
    Term("programming").
    And().
    Group(func(q *blaze.QueryBuilder) {
        q.Term("python").Or().Term("javascript")
    }).
    And().Not().
    Term("beginner").
    ExecuteWithBM25(10)
```

--------------------------------

### Using BM25 for User-Facing Searches in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This Go snippet illustrates the best practice of using the BM25 scoring algorithm for user-facing searches to provide ranked results. It contrasts a 'good' example using `ExecuteWithBM25` with a 'bad' example that returns an unranked bitmap.

```go
// Good: Ranked results for users
matches := qb.ExecuteWithBM25(10)

// Bad: Unranked - harder for users to find relevant docs
bitmap := qb.Execute()
```

--------------------------------

### Phrase Search using Inverted Index (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Illustrates how to perform an exact phrase search within documents indexed by the InvertedIndex. The example shows indexing documents and then using the FindAllPhrases function to locate all occurrences of a specific two-word phrase, returning the document ID and the start and end offsets of the phrase.

```go
package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    idx := blaze.NewInvertedIndex()

    idx.Index(1, "the quick brown fox jumps over the lazy dog")
    idx.Index(2, "a quick brown dog runs fast")
    idx.Index(3, "the lazy brown fox sleeps")

    // Find exact phrase "brown fox"
    matches := idx.FindAllPhrases("brown fox", blaze.BOFDocument)

    fmt.Println("Documents containing 'brown fox' as a phrase:")
    for _, match := range matches {
        docID := int(match[0].DocumentID)
        start := int(match[0].Offset)
        end := int(match[1].Offset)
        fmt.Printf("Document %d: positions %d-%d\n", docID, start, end)
    }
}
```

--------------------------------

### Go: Blaze Query Builder Core Methods - NewQueryBuilder

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Shows the instantiation of a new query builder using the `NewQueryBuilder` function, which requires an `InvertedIndex` instance. This is the starting point for constructing any query.

```go
qb := blaze.NewQueryBuilder(idx)
```

--------------------------------

### Analyze Document Statistics and Metrics in Go

Source: https://context7.com/wizenheimer/blaze/llms.txt

Provides a Go code example for accessing and displaying various statistics of an inverted index built with the blaze library. It shows how to retrieve the total number of documents and terms, calculate the average document length, and iterate through per-document statistics including term frequencies and document bitmaps for term distribution analysis.

```go
package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    idx := blaze.NewInvertedIndex()

    idx.Index(1, "machine learning is a subset of artificial intelligence")
    idx.Index(2, "deep learning neural networks")
    idx.Index(3, "machine learning algorithms for data science")
    idx.Index(4, "natural language processing")

    // Access document statistics
    fmt.Printf("Total documents: %d\n", idx.TotalDocs)
    fmt.Printf("Total terms indexed: %d\n", idx.TotalTerms)

    // Average document length
    if idx.TotalDocs > 0 {
        avgLen := float64(idx.TotalTerms) / float64(idx.TotalDocs)
        fmt.Printf("Average document length: %.2f terms\n", avgLen)
    }

    // Per-document statistics
    for docID, stats := range idx.DocStats {
        fmt.Printf("\nDocument %d:\n", docID)
        fmt.Printf("  Length: %d terms\n", stats.Length)
        fmt.Printf("  Unique terms: %d\n", len(stats.TermFreqs))
        fmt.Printf("  Term frequencies:\n")

        for term, freq := range stats.TermFreqs {
            fmt.Printf("    '%s': %d occurrences\n", term, freq)
        }
    }

    // Term distribution across documents
    fmt.Println("\nTerm document frequencies:")
    for term, bitmap := range idx.DocBitmaps {
        docCount := bitmap.GetCardinality()
        fmt.Printf("  '%s': appears in %d documents\n", term, docCount)
    }
}
```

--------------------------------

### Phrase Search with Blaze

Source: https://github.com/wizenheimer/blaze/blob/main/public/index.html

Finds exact sequences of words within a document using Blaze's phrase search capabilities. It returns the start and end positions of each phrase match.

```go
// Find exact phrase "quick brown fox"
matches := idx.FindAllPhrases("quick brown fox", blaze.BOFDocument)

for _, match := range matches {
    start, end := match[0], match[1]
    fmt.Printf("Found in Doc %d from Pos %d to %d\n",
        int(start.DocumentID), int(start.Offset), int(end.Offset))
}
```

--------------------------------

### Exclusion Filtering Query in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This Go example illustrates how to perform searches while actively filtering out unwanted categories. It finds content about 'apple' but excludes terms related to fruit, food, or cooking, effectively targeting non-food related 'apple' mentions.

```go
results := blaze.NewQueryBuilder(idx).
    Term("apple").
    And().Not().
    Group(func(q *blaze.QueryBuilder) {
        q.Term("fruit").Or().Term("food").Or().Term("cooking")
    }).
    Execute()
```

--------------------------------

### Index Log Files and Search for Errors in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Shows how to index log files line by line and search for specific patterns or errors. This function reads a log file, indexes each line with its line number, and provides examples of using RankBM25 and RankProximity for searching.

```go
import (
    "bufio"
    "os"
    // Assuming blaze is imported
)

func IndexLogs(logFile string) (*blaze.InvertedIndex, error) {
    idx := blaze.NewInvertedIndex()

    file, err := os.Open(logFile)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    lineNumber := 1

    for scanner.Scan() {
        idx.Index(lineNumber, scanner.Text())
        lineNumber++
    }

    return idx, scanner.Err()
}

// Find all ERROR log lines using BM25 (considers frequency and rarity)
// errorLogs := idx.RankBM25("ERROR", 100)

// Alternative: Use proximity for finding error patterns
// e.g., "connection timeout" appearing close together
// patternMatches := idx.RankProximity("connection timeout", 50)

```

--------------------------------

### Ranking Phase: BM25 Scoring Formula Example

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This snippet outlines the steps involved in the ranking phase using the BM25 scoring algorithm. It details the calculation of Inverse Document Frequency (IDF) and Term Frequency (TF), and how these are combined with document length normalization to produce a final score for each candidate document.

```plaintext
1. Calculate IDF (Inverse Document Frequency):
       - Uses bitmap cardinality for instant document counts
       - IDF("machine") = log((N - df + 0.5) / (df + 0.5))
       - df = DocBitmaps["machine"].GetCardinality()

    2. Calculate TF (Term Frequency):
       - Retrieves from pre-computed DocStats
       - TF("machine", Doc1) = termFreqs["machine"]

    3. Apply BM25 formula:
       - Combines IDF, TF, and length normalization
       - Score = IDF × (TF × (k1 + 1)) / (TF + k1 × length_norm)

    4. Sum scores for all query terms
```

--------------------------------

### Rank Documents using BM25 and Proximity (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates the usage of two ranking algorithms, BM25 and Proximity, for search queries. BM25 is suitable for general relevance ranking, while Proximity is effective for finding terms that appear close together, useful for phrase matching and snippet generation. The examples show how to use these methods for different e-commerce and document search scenarios, and how to combine them for comprehensive results.

```go
// E-commerce: General product search
// BM25 considers term frequency and rarity
bm25Results := idx.RankBM25("wireless bluetooth headphones", 20)
// Returns products with any/all terms, ranked by relevance

// E-commerce: Exact product name
// Proximity finds terms appearing together
proxResults := idx.RankProximity("Sony WH-1000XM4", 20)
// Returns products where terms appear close together

// Document search: Research papers
// BM25 for broad topic search
papers := idx.RankBM25("neural networks deep learning", 50)

// Document search: Finding specific phrase mentions
// Proximity for finding "neural networks" as a concept
mentions := idx.RankProximity("neural networks", 50)

// Best practice: Use both for different purposes!
generalResults := idx.RankBM25(query, 100)    // Cast wide net
preciseResults := idx.RankProximity(query, 20) // Refine results
```

--------------------------------

### Find Next Token Cover in Inverted Index (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Finds the next 'cover' - a range within the index that contains all specified tokens. It takes a slice of tokens and a starting position, returning the start and end positions of the cover.

```go
func (idx *InvertedIndex) NextCover(tokens []string, startPos Position) []Position

// Example:
// cover := idx.NextCover([]string{"quick", "fox", "brown"}, blaze.BOFDocument)
// fmt.Printf("Cover: Doc %d, Pos %d-%d\n",
//     int(cover[0].DocumentID),
//     int(cover[0].Offset),
//     int(cover[1].Offset))
```

--------------------------------

### Find Next Phrase Occurrence in Inverted Index (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Finds the next occurrence of an exact word sequence (phrase) within the inverted index, starting from a specified position. It returns the start and end positions of the phrase if found, or EOFDocument markers if not.

```go
func (idx *InvertedIndex) NextPhrase(query string, startPos Position) []Position

// Example:
// matches := idx.NextPhrase("quick brown fox", blaze.BOFDocument)
// if !matches[0].IsEnd() {
//     fmt.Printf("Phrase found in Doc %d from Pos %d to %d\n",
//         int(matches[0].DocumentID),
//         int(matches[0].Offset),
//         int(matches[1].Offset))
// }
```

--------------------------------

### Query Builder Core Methods

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Detailed documentation for the core methods available in the Query Builder.

```APIDOC
## Query Builder Core Methods

### `NewQueryBuilder(index *InvertedIndex) *QueryBuilder`

Creates a new query builder instance associated with a specific inverted index.

**Example:**
```go
qb := blaze.NewQueryBuilder(idx)
```

### `Term(term string) *QueryBuilder`

Adds a single term to the query. This operation is efficient, utilizing roaring bitmaps for O(1) document lookups.

**Example:**
```go
qb.Term("machine")
```

### `Phrase(phrase string) *QueryBuilder`

Adds an exact phrase match to the query. This method combines the efficiency of bitmaps with position checking using skip lists.

**Example:**
```go
qb.Phrase("machine learning")
```

### `And() *QueryBuilder`

Combines the current query results with the next term using an intersection (logical AND). This operation uses the bitmap AND operation.

**Example:**
```go
qb.Term("machine").And().Term("learning")
```

### `Or() *QueryBuilder`

Combines the current query results with the next term using a union (logical OR). This operation uses the bitmap OR operation.

**Example:**
```go
qb.Term("cat").Or().Term("dog")
```

### `Not() *QueryBuilder`

Negates the results of the next term (logical NOT), effectively excluding documents that match the subsequent term. This uses the bitmap difference operation.

**Example:**
```go
qb.Term("python").And().Not().Term("snake")
```

### `Group(fn func(*QueryBuilder)) *QueryBuilder`

Creates a sub-query with its own scope, allowing for controlled precedence in complex boolean expressions.

**Example:**
```go
qb.Group(func(q *blaze.QueryBuilder) {
    q.Term("machine").Or().Term("deep")
}).And().Term("learning")
```

### `Execute() *roaring.Bitmap`

Executes the constructed query and returns a `roaring.Bitmap` containing the IDs of all matching documents.

**Example:**
```go
results := qb.Execute()
docCount := results.GetCardinality()
```

### `ExecuteWithBM25(maxResults int) []Match`

Executes the query and returns a slice of `Match` structs, containing document IDs and their BM25 relevance scores, limited by `maxResults`.

**Example:**
```go
matches := qb.ExecuteWithBM25(10)  // Top 10 results
```
```

--------------------------------

### Go: Query Builder - AND Query

Source: https://github.com/wizenheimer/blaze/blob/main/public/index.html

Illustrates how to build an AND query using the Blaze Query Builder in Go. This allows you to find documents that contain all specified terms, ensuring both conditions are met.

```go
// Find documents with BOTH "machine" AND "learning"
results := blaze.NewQueryBuilder(idx).
	Term("machine").
	And().
	Term("learning").
	Execute()
```

--------------------------------

### Blaze Project Directory Structure

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This overview outlines the directory structure of the blaze project. It details the purpose of key Go files, including implementations for inverted indexes, query building, skip lists, search algorithms, text analysis, and serialization. It also points to the Makefile for development commands and the public directory for documentation.

```plaintext
blaze/
├── index.go          # Inverted index implementation with hybrid storage
├── query.go          # Query builder with roaring bitmaps
├── skiplist.go       # Skip list data structure for positions
├── search.go         # Search algorithms (phrase, proximity, BM25)
├── analyzer.go       # Text analysis pipeline
├── serialization.go  # Binary encoding/decoding (skip lists + bitmaps)
├── *_test.go         # Comprehensive test suite
├── Makefile          # Development commands
└── public/
    └── index.html
```

--------------------------------

### Run Development Commands with Make

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

This section provides Bash commands for managing development tasks within the blaze project using the Makefile. It includes commands for running all tests, generating test coverage reports, executing benchmarks, and performing comprehensive checks like formatting, vetting, linting, and testing.

```bash
# Run all tests
make test

# Run tests with coverage
make test-coverage

# Run benchmarks
make bench

# Run all checks (format, vet, lint, test)
make check
```

--------------------------------

### Find All Phrase Occurrences in Inverted Index (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Finds all occurrences of a given phrase within the entire inverted index. It returns a slice of position pairs, where each pair represents the start and end positions of a phrase match.

```go
func (idx *InvertedIndex) FindAllPhrases(query string, startPos Position) [][]Position

// Example:
// allMatches := idx.FindAllPhrases("brown fox", blaze.BOFDocument)
// for _, match := range allMatches {
//     fmt.Printf("Doc %d: Pos %d-%d\n",
//         int(match[0].DocumentID),
//         int(match[0].Offset),
//         int(match[1].Offset))
// }
```

--------------------------------

### Serialize and Persist Inverted Index in Go

Source: https://context7.com/wizenheimer/blaze/llms.txt

Demonstrates how to build an inverted index, serialize it to a binary format, save it to a file, load it back, and deserialize it. It uses the blaze library for index operations and Go's os package for file I/O. Handles potential errors during encoding, decoding, and file operations.

```go
package main

import (
    "fmt"
    "os"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Build and save index
    idx := blaze.NewInvertedIndex()
    idx.Index(1, "machine learning algorithms")
    idx.Index(2, "deep learning neural networks")
    idx.Index(3, "natural language processing")

    // Serialize to binary format
    data, err := idx.Encode()
    if err != nil {
        panic(err)
    }

    fmt.Printf("Index serialized: %d bytes\n", len(data))

    // Save to file
    err = os.WriteFile("search_index.bin", data, 0644)
    if err != nil {
        panic(err)
    }

    // Load from file
    loadedData, err := os.ReadFile("search_index.bin")
    if err != nil {
        panic(err)
    }

    // Deserialize into new index
    newIdx := blaze.NewInvertedIndex()
    err = newIdx.Decode(loadedData)
    if err != nil {
        panic(err)
    }

    // Verify loaded index works
    results := newIdx.RankBM25("machine learning", 10)
    fmt.Printf("Loaded index search results: %d documents\n", len(results))

    for _, match := range results {
        fmt.Printf("  Doc %d: Score %.4f\n", match.DocID, match.Score)
    }
}
```

--------------------------------

### Run Tests and Linter (Bash)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Includes bash commands to run automated tests (`make test`) and the code linter (`make lint`) for the Blaze project. These commands are crucial for maintaining code quality and ensuring that new changes adhere to project standards.

```bash
# Run tests
make test

# Run linter
make lint
```

--------------------------------

### Roaring Bitmap Storage Adaptation

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Illustrates how Roaring Bitmaps adapt storage based on cardinality. It uses Array Containers for low cardinality, Bitmap Containers for medium, and Run Containers for full cardinality within a chunk.

```text
┌────────────────────────────────────────────────────┐
│ If cardinality < 4096:                             │
│   → Use Array Container                            │
│   → Store sorted uint16 values directly            │
│   → Size: 2 bytes × cardinality                    │
│                                                     │
│ If cardinality > 4096:                             │
│   → Use Bitmap Container                           │
│   → Store 65536-bit bitmap (8 KB)                 │
│   → Size: 8 KB fixed                               │
│                                                     │
│ If cardinality = 65536 (all docs):                │
│   → Use Run Container                              │
│   → Store: [0-65535]                               │
│   → Size: 4 bytes                                  │
└────────────────────────────────────────────────────┘
```

--------------------------------

### Multi-Category Product Search (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/public/index.html

Searches for products across multiple specified categories. It uses boolean OR logic within a group to match products in any of the given categories. Results are ranked using BM25.

```Go
func SearchInCategories(idx *blaze.InvertedIndex, 
                            query string, categories []string) []blaze.Match {
        qb := blaze.NewQueryBuilder(idx).Term(query)
        
        if len(categories) > 0 {
            qb.And().Group(func(q *blaze.QueryBuilder) {
                q.Term(categories[0])
                for i := 1; i < len(categories); i++ {
                    q.Or().Term(categories[i])
                }
            })
        }
        
        return qb.ExecuteWithBM25(50)
    }
```

--------------------------------

### Go: Builder Pattern vs. String Parsing for Queries

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates the advantage of using the Blaze Query Builder pattern over a direct string parsing approach for constructing search queries. The builder pattern offers type safety and compile-time checks, preventing runtime errors.

```go
// String Parsing Approach:
// Error-prone, runtime failures
results, err := index.ExecuteQuery("(machine AND learning) OR python")
if err != nil {
    // Handle parsing errors
}

// Builder Pattern Approach:
// Type-safe, compile-time checks, IDE autocomplete!
results := blaze.NewQueryBuilder(index).
    Group(func(q *blaze.QueryBuilder) {
        q.Term("machine").And().Term("learning")
    }).
    Or().
    Term("python").
    Execute()
```

--------------------------------

### Basic Document Search with BM25 Ranking (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates how to use the InvertedIndex to index documents and perform a search using the BM25 ranking algorithm. It shows the process of creating an index, adding documents with their content, and then retrieving ranked search results for a given query. BM25 scores indicate relevance based on term frequency, document length, and rarity.

```go
package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Create index
    idx := blaze.NewInvertedIndex()

    // Index documents
    idx.Index(1, "Go is a programming language designed at Google")
    idx.Index(2, "Python is a high-level programming language")
    idx.Index(3, "Go is fast and efficient for system programming")

    // Search for "programming language" using BM25
    results := idx.RankBM25("programming language", 10)

    fmt.Println("Search results for 'programming language':")
    for i, match := range results {
        fmt.Printf("%d. Document %d (score: %.3f)\n", i+1, match.DocID, match.Score)
    }
}
```

--------------------------------

### Configuring Blaze Analyzers for Different Use Cases in Go

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Illustrates how to configure the Blaze analyzer for various text types, including natural language, technical documentation, and product names. Each configuration demonstrates different settings for minimum token length, stemming, and stopword removal to optimize search results.

```go
// Natural language text (books, articles)
naturalLanguageConfig := blaze.AnalyzerConfig{
    MinTokenLength:  2,
    EnableStemming:  true,   // Find related words
    EnableStopwords: true,   // Remove common words
}

// Technical documentation (code, APIs)
technicalConfig := blaze.AnalyzerConfig{
    MinTokenLength:  2,
    EnableStemming:  false,  // Keep exact terms
    EnableStopwords: false,  // Keep all words
}

// Product names (e-commerce)
productConfig := blaze.AnalyzerConfig{
    MinTokenLength:  1,      // Include single chars (e.g., "X")
    EnableStemming:  false,  // Exact product names
    EnableStopwords: false,  // Keep all words
}
```

--------------------------------

### Iterating Through Term Occurrences (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates how to iterate through all occurrences of a specific term within indexed documents using the InvertedIndex. The Next function is used in a loop, starting from the beginning of a document, to find each subsequent position (document ID and offset) where the term appears. The loop continues until the end of the document set is reached.

```go
package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    idx := blaze.NewInvertedIndex()

    idx.Index(1, "quick test quick test quick")
    idx.Index(2, "another quick test here")

    // Find all occurrences of "quick"
    fmt.Println("All occurrences of 'quick':")

    pos := blaze.BOFDocument
    for {
        pos, err := idx.Next("quick", pos)
        if err != nil || pos.IsEnd() {
            break
        }
        fmt.Printf("  Doc %d, Pos %d\n",
            int(pos.DocumentID),
            int(pos.Offset))
    }
}
```

--------------------------------

### Blaze Text Analysis Pipeline Configuration (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates how to use the Blaze library to analyze text, either with default settings or custom configurations. Custom configurations allow control over token length, stemming, and stopword filtering.

```go
// Use default configuration
tokens := blaze.Analyze("The quick brown fox")

// Custom configuration
config := blaze.AnalyzerConfig{
    MinTokenLength:  3,      // Only keep tokens >= 3 chars
    EnableStemming:  false,  // Disable stemming
    EnableStopwords: true,   // Keep stopword filtering
}
tokens := blaze.AnalyzeWithConfig("The quick brown fox", config)
```

--------------------------------

### Text Analysis Configuration with Custom Options

Source: https://context7.com/wizenheimer/blaze/llms.txt

Shows how to configure and use the Blaze text analyzer. It demonstrates default analysis (lowercasing, stemming, stopword removal) and custom configurations to disable stemming, control stopword inclusion, and set minimum token length.

```go
package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Default analysis pipeline
    tokens1 := blaze.Analyze("The Quick Brown Fox Jumps!")
    fmt.Printf("Default: %v\n", tokens1)
    // Output: [quick brown fox jump]
    // (lowercased, stopwords removed, stemmed)

    // Custom configuration: disable stemming
    config2 := blaze.AnalyzerConfig{
        MinTokenLength:  2,
        EnableStemming:  false,
        EnableStopwords: true,
    }
    tokens2 := blaze.AnalyzeWithConfig("Running quickly jumps", config2)
    fmt.Printf("No stemming: %v\n", tokens2)
    // Output: [running quickly jumps]

    // Keep stopwords, higher minimum length
    config3 := blaze.AnalyzerConfig{
        MinTokenLength:  3,
        EnableStemming:  true,
        EnableStopwords: false,
    }
    tokens3 := blaze.AnalyzeWithConfig("The cat is running", config3)
    fmt.Printf("Keep stopwords, min=3: %v\n", tokens3)
    // Output: [the cat run]

    // Minimal processing
    config4 := blaze.AnalyzerConfig{
        MinTokenLength:  1,
        EnableStemming:  false,
        EnableStopwords: false,
    }
    tokens4 := blaze.AnalyzeWithConfig("I am a developer", config4)
    fmt.Printf("Minimal: %v\n", tokens4)
    // Output: [i am a developer]
}
```

--------------------------------

### Go: Phrase Queries with Blaze

Source: https://github.com/wizenheimer/blaze/blob/main/README.md

Demonstrates how to perform exact phrase searches using the Blaze Query Builder. This method combines bitmap efficiency with skip list position checking for accurate phrase matching.

```go
// Find exact phrase "machine learning"
results := blaze.NewQueryBuilder(idx).
    Phrase("machine learning").
    Execute()
```

--------------------------------

### E-commerce Product Search with Filters (Go)

Source: https://github.com/wizenheimer/blaze/blob/main/public/index.html

Searches for products in an e-commerce context using an inverted index. Supports term search, category filtering, and excluding out-of-stock items. Returns a list of matching products ranked by BM25.

```Go
func SearchProducts(idx *blaze.InvertedIndex, query string, 
                        category string, excludeOutOfStock bool) []blaze.Match {
        qb := blaze.NewQueryBuilder(idx).Term(query)
        
        // Add category filter
        if category != "" {
            qb.And().Term(category)
        }
        
        // Exclude out of stock items
        if excludeOutOfStock {
            qb.And().Not().Term("outofstock")
        }
        
        return qb.ExecuteWithBM25(20)
    }
```
Tokens:
Raw