Wink BM25 Text Search

https://github.com/winkjs/wink-bm25-text-search
Admin
Wink BM25 Text Search is a fast, configurable full text search engine for Node.js and browser...
Tokens:2,678
Snippets:37
Trust Score:8.9
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install wink-bm25-text-search

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Use npm to install the package as a dependency.

```sh
npm install wink-bm25-text-search --save
```

--------------------------------

### Define document structure for addDoc

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Example JSON structure for documents added to the BM25 model.

```json
{
  title: 'Barack Obama',
  body: 'Barack Hussein Obama II born August 4, 1961 is an American politician...'
  tags: 'democratic nobel peace prize columbia michelle...'
}
```

--------------------------------

### Initialize and Use BM25 Search Engine

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Demonstrates configuring the engine, defining preprocessing tasks, adding documents, and performing a search query.

```javascript
// Load wink-bm25-text-search
var bm25 = require( 'wink-bm25-text-search' );
// Create search engine's instance
var engine = bm25();
// Load sample data (load any other JSON data instead of sample)
var docs = require( 'wink-bm25-text-search/sample-data/demo-data-for-wink-bm25.json' );
// Load wink nlp and its model
const winkNLP = require( 'wink-nlp' );
// Use web model
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );
const its = nlp.its;

const prepTask = function ( text ) {
  const tokens = [];
  nlp.readDoc(text)
      .tokens()
      // Use only words ignoring punctuations etc and from them remove stop words
      .filter( (t) => ( t.out(its.type) === 'word' && !t.out(its.stopWordFlag) ) )
      // Handle negation and extract stem of the word
      .each( (t) => tokens.push( (t.out(its.negationFlag)) ? '!' + t.out(its.stem) : t.out(its.stem) ) );

  return tokens;
};

// Contains search query.
var query;

// Step I: Define config
// Only field weights are required in this example.
engine.defineConfig( { fldWeights: { title: 1, body: 2 } } );
// Step II: Define PrepTasks pipe.
// Set up 'default' preparatory tasks i.e. for everything else
engine.definePrepTasks( [ prepTask ] );

// Step III: Add Docs
// Add documents now...
docs.forEach( function ( doc, i ) {
  // Note, 'i' becomes the unique id for 'doc'
  engine.addDoc( doc, i );
} );

// Step IV: Consolidate
// Consolidate before searching
engine.consolidate();

// All set, start searching!
query = 'not studied law';
// `results` is an array of [ doc-id, score ], sorted by score
var results = engine.search( query );
// Print number of results.
console.log( '%d entries found.', results.length );
// -> 1 entries found.
// results[ 0 ][ 0 ] i.e. the top result is:
console.log( docs[ results[ 0 ][ 0 ] ].body );
// -> George Walker Bush (born July 6, 1946) is an...
// -> ... He never studied Law...

// Whereas if you search for `law` then multiple entries will be
// found except the above entry!
```

--------------------------------

### Accessor Methods

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

A collection of read-only methods to retrieve internal state and configuration from the BM25 search instance.

```APIDOC
## Accessor Methods

### getDocs()
Returns the Term Frequencies and length of each document.

### getTokens()
Returns the `token: index` mapping.

### getIDF()
Returns IDF for each token. Tokens are referenced via their numerical index, which is accessed via `getTokens()`.

### getConfig()
Returns the BM25F Configuration as set up by `defineConfig()`.

### getTotalCorpusLength()
Returns the total number of tokens across all documents added.

### getTotalDocs()
Returns total documents added.
```

--------------------------------

### definePrepTasks( tasks [, field ] )

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Defines the text preparation tasks to transform raw text into tokens.

```APIDOC
## definePrepTasks( tasks [, field ] )

### Description
Defines the text preparation tasks to transform raw incoming text into an array of tokens required during `addDoc()` and `search()` operations.

### Parameters
- **tasks** (array) - Required - An array of functions where the first accepts a string and the last returns an array of tokens.
- **field** (string) - Optional - The document field for which the tasks are defined.
```

--------------------------------

### defineConfig( config )

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Defines the configuration for the BM25 model, including field weights, BM25 parameters, and fields to retain.

```APIDOC
## defineConfig( config )

### Description
Defines the configuration from the `config` object. This object defines field weights, BM25 parameters (k1, b, k), and fields whose original values must be retained.

### Parameters
- **config** (object) - Required - Object containing `fldWeights` (mandatory), `bm25Params` (optional), and `ovFldNames` (optional).
```

--------------------------------

### exportJSON() / importJSON( json )

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Methods for exporting and importing the BM25 model as JSON.

```APIDOC
## exportJSON()
Exports the BM25 model as JSON text.

## importJSON( json )
Imports an existing JSON BM25 model for searching.
- **json** (string) - Required - The JSON string representing the model.
```

--------------------------------

### consolidate( fp )

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Consolidates the BM25 model for all added documents.

```APIDOC
## consolidate( fp )

### Description
Consolidates the BM25 model. This is a prerequisite for `search()` and prevents further document additions.

### Parameters
- **fp** (number) - Optional - The precision at which term frequency values are stored (default: 4).
```

--------------------------------

### reset()

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Resets the BM25 model.

```APIDOC
## reset()
Completely resets the BM25 model by re-initializing all variables, except the preparatory tasks.
```

--------------------------------

### search( text [, limit, filter, params ] )

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Searches for text within the BM25 model.

```APIDOC
## search( text [, limit, filter, params ] )

### Description
Searches for the `text` and returns results sorted by relevance score.

### Parameters
- **text** (string) - Required - The search query.
- **limit** (number) - Optional - Maximum number of results (default: 10).
- **filter** (function) - Optional - A function to filter results.
- **params** (any) - Optional - Parameters passed to the filter function.
```

--------------------------------

### addDoc( doc, uniqueId )

Source: https://github.com/winkjs/wink-bm25-text-search/blob/master/README.md

Adds a document to the BM25 model.

```APIDOC
## addDoc( doc, uniqueId )

### Description
Adds the `doc` with the `uniqueId` to the BM25 model. Requires `defineConfig()` and `definePrepTasks()` to be called first.

### Parameters
- **doc** (object) - Required - The structured JSON document to add.
- **uniqueId** (string/number) - Required - A unique identifier for the document.
```
Tokens:
Raw