### BM25s Quickstart Example Source: https://github.com/xhluca/bm25s/blob/main/README.md A basic example demonstrating how to index a corpus, perform a search query, and retrieve results using the bm25s library. Stemming is optionally used for better results. ```python import bm25s import Stemmer # optional: for stemming # Create your corpus here corpus = [ "a cat is a feline and likes to purr", "a dog is the human's best friend and loves to play", "a bird is a beautiful animal that can fly", "a fish is a creature that lives in water and swims", ] # optional: create a stemmer stemmer = Stemmer.Stemmer("english") # Tokenize the corpus and only keep the ids (faster and saves memory) corpus_tokens = bm25s.tokenize(corpus, stopwords="en", stemmer=stemmer) # Create the BM25 model and index the corpus retriever = bm25s.BM25() retriever.index(corpus_tokens) # Query the corpus query = "does the fish purr like a cat?" query_tokens = bm25s.tokenize(query, stemmer=stemmer) # Get top-k results as a tuple of (doc ids, scores). Both are arrays of shape (n_queries, k). # To return docs instead of IDs, set the `corpus=corpus` parameter. results, scores = retriever.retrieve(query_tokens, k=2) for i in range(results.shape[1]): doc, score = results[0, i], scores[0, i] print(f"Rank {i+1} (score: {score:.2f}): {doc}") ``` -------------------------------- ### Install huggingface_hub library Source: https://github.com/xhluca/bm25s/blob/main/README.md Installs the necessary library for interacting with the Hugging Face Hub. ```bash pip install huggingface_hub ``` -------------------------------- ### Install BM25s with Core Optional Dependencies Source: https://github.com/xhluca/bm25s/blob/main/_autodocs/INDEX.md Installs bm25s with core optional features like stemming and Numba for faster retrieval. This is the recommended installation for most users. ```bash pip install "bm25s[core]" ``` -------------------------------- ### Install bm25s with MCP Extra Source: https://github.com/xhluca/bm25s/blob/main/README.md Install the bm25s library with the 'mcp' extra using uv. This is necessary to use the MCP server functionality. ```bash uv venv source .venv/bin/activate uv pip install "bm25s[mcp]" # or locally: uv pip install -e ".[mcp]" ``` -------------------------------- ### Install PyStemmer for stemming Source: https://github.com/xhluca/bm25s/blob/main/README.md Install the PyStemmer library if you want to use stemming for improved retrieval results. This is an optional dependency. ```bash pip install PyStemmer ``` -------------------------------- ### Create Index and Launch Server Example Source: https://github.com/xhluca/bm25s/blob/main/README.md Example demonstrating how to create a test index using examples/index_nq.py and then launch the MCP server pointing to the created index. ```bash source .venv/bin/activate python examples/index_nq.py # creates bm25s_indices/nq bm25 mcp launch --port 8000 --index-dir ./bm25s_indices/nq ``` -------------------------------- ### Catch ImportError for Scipy Backend Source: https://github.com/xhluca/bm25s/blob/main/_autodocs/errors.md This example demonstrates how to handle an ImportError when the 'scipy' csc_backend is requested but Scipy is not installed. It includes a fallback to the 'numpy' backend. ```python try: retriever = bm25s.BM25(csc_backend="scipy") except ImportError as e: print(f"Error: {e}") retriever = bm25s.BM25(csc_backend="numpy") # Fallback ``` -------------------------------- ### Example Workflow: Basic Indexing and Searching Source: https://github.com/xhluca/bm25s/blob/main/README.md A step-by-step example demonstrating the basic workflow of creating a text file, indexing its content using the bm25s CLI, and then searching the created index. ```bash # 1. Create a simple text file with documents echo -e "Machine learning is a subset of AI\nDeep learning uses neural networks\nNatural language processing handles text" > docs.txt # 2. Index the documents bm25 index docs.txt -o my_index # 3. Search the index bm25 search -i my_index "what is AI?" ``` -------------------------------- ### Catch ImportError for JAX Backend Selection Source: https://github.com/xhluca/bm25s/blob/main/_autodocs/errors.md Handle an ImportError when attempting to use the 'jax' backend selection in the retrieve() method without JAX installed. This example includes a fallback to the 'numpy' backend. ```python try: results = retriever.retrieve(query_tokens, backend_selection="jax", k=5) except ImportError as e: print(f"Error: {e}") results = retriever.retrieve(query_tokens, backend_selection="numpy", k=5) ```