Minish AI Packages

https://minish.ai/packages/
Admin
Minish AI Packages provides an overview of their current development efforts, offering tools for...
Tokens:20,317
Snippets:69
Trust Score:5.5
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Base Vicinity Package

Source: https://minish.ai/packages/vicinity/installation

Installs the core Vicinity package, which includes a lightweight flat index for exact nearest neighbor search.

```Python
pip install vicinity
```

--------------------------------

### Install Vicinity with Optional Dependencies

Source: https://minish.ai/packages/vicinity/installation

Installs Vicinity along with optional dependencies for various ANN backends and integrations. This allows for enhanced functionality and compatibility with different libraries.

```Python
pip install vicinity[all]
```

```Python
pip install vicinity[backends]
```

```Python
pip install vicinity[annoy]
```

```Python
pip install vicinity[faiss]
```

```Python
pip install vicinity[hnsw]
```

```Python
pip install vicinity[pynndescent]
```

```Python
pip install vicinity[usearch]
```

```Python
pip install vicinity[voyager]
```

```Python
pip install vicinity[integrations]
```

```Python
pip install vicinity[huggingface]
```

--------------------------------

### Install Base Model2Vec Package

Source: https://minish.ai/packages/model2vec/installation

Installs the core Model2Vec package using pip. This is the fundamental step for using the library.

```bash
pip install model2vec
```

--------------------------------

### Vicinity Quick Start Example

Source: https://minish.ai/packages/vicinity/introduction

Demonstrates how to use the Vicinity library in Python. It covers initializing Vicinity with dummy data, performing nearest neighbor searches using top-k and threshold methods, and querying with multiple vectors. Requires numpy.

```python
import numpy as np
from vicinity import Vicinity, Backend, Metric
# Create some dummy data as strings or other serializable objects
items = ["triforce", "master sword", "hylian shield", "boomerang", "hookshot"]
vectors = np.random.rand(len(items), 128)
# Initialize the Vicinity instance (using the basic backend and cosine metric)
vicinity = Vicinity.from_vectors_and_items(
    vectors=vectors,
    items=items,
    backend_type=Backend.BASIC,
    metric=Metric.COSINE
)
# Create a query vector
query_vector = np.random.rand(128)
# Query for nearest neighbors with a top-k search
results = vicinity.query(query_vector, k=3)
# Query for nearest neighbors with a threshold search
results = vicinity.query_threshold(query_vector, threshold=0.9)
# Query with a list of query vectors
query_vectors = np.random.rand(5, 128)
results = vicinity.query(query_vectors, k=3)

```

--------------------------------

### Install Model2Vec with Optional Dependencies

Source: https://minish.ai/packages/model2vec/installation

Installs Model2Vec with specific optional dependencies for enhanced functionality. These include packages for distillation, classifier training, inference, and ONNX conversion.

```bash
# For distillation
pip install model2vec[distill]
```

```bash
# For classifier training
pip install model2vec[train]
```

```bash
# For classifier inference
pip install model2vec[inference]
```

```bash
# For onnx conversion
pip install model2vec[onnx]
```

--------------------------------

### Install Tokenlearn

Source: https://minish.ai/packages/tokenlearn/usage

Installs the Tokenlearn package using pip. This is the first step to using Tokenlearn for pre-training Model2Vec models.

```shell
pip install tokenlearn
```

--------------------------------

### Install Model2Vec with Training Extras

Source: https://minish.ai/packages/model2vec/training

Installs the Model2Vec library with the necessary extras for training classification models. This is the first step before proceeding with any training operations.

```bash
pip install model2vec[training]
```

--------------------------------

### Install Tokenlearn

Source: https://minish.ai/packages/tokenlearn

Installs the Tokenlearn package using pip. This is the first step to using Tokenlearn for pre-training Model2Vec models.

```shell
pip install tokenlearn
```

--------------------------------

### Install Model2Vec

Source: https://minish.ai/packages/model2vec/introduction

Installs the Model2Vec library using pip. This is the first step to using Model2Vec for generating static embeddings.

```Shell
pip install model2vec
```

--------------------------------

### Install Vicinity

Source: https://minish.ai/packages/vicinity/introduction

Installs the Vicinity library using pip. This is the first step to using Vicinity for nearest neighbor search.

```bash
pip install vicinity

```

--------------------------------

### Install Model2Vec

Source: https://minish.ai/packages/model2vec

Installs the Model2Vec library using pip. This is the first step to using Model2Vec for generating static embeddings.

```Shell
pip install model2vec
```

--------------------------------

### Install Vicinity using pip

Source: https://minish.ai/packages/vicinity

This command installs the Vicinity package using pip, making it available for use in your Python projects. Ensure you have pip installed and configured.

```shell
pip install vicinity

```

--------------------------------

### Install SemHash

Source: https://minish.ai/packages/semhash/introduction

This command installs the SemHash library using pip, making it available for use in Python projects.

```bash
pip install semhash
```

--------------------------------

### Install Model2Vec with Distill Extra

Source: https://minish.ai/packages/model2vec/distillation

To use the distillation features of Model2Vec, you need to install the package with the 'distill' extra. This ensures all necessary dependencies for distillation are included.

```bash
pip install model2vec[distill]
```

--------------------------------

### Install SemHash with pip

Source: https://minish.ai/packages/semhash

Install SemHash using pip. This command will download and install the SemHash package and its dependencies.

```bash
pip install semhash
```

--------------------------------

### Install SemHash Package

Source: https://minish.ai/packages/semhash/installation

This snippet shows the command to install the SemHash package using pip. SemHash has minimal dependencies, relying only on Model2Vec and Vicinity.

```python
pip install semhash
```

--------------------------------

### Vicinity Nearest Neighbor Search Example

Source: https://minish.ai/packages/vicinity

This Python code demonstrates how to use the Vicinity library for nearest neighbor search. It covers initializing Vicinity with data, performing top-k queries, threshold-based queries, and batch queries with multiple query vectors.

```python
import numpy as np
from vicinity import Vicinity, Backend, Metric
# Create some dummy data as strings or other serializable objects
items = ["triforce", "master sword", "hylian shield", "boomerang", "hookshot"]
vectors = np.random.rand(len(items), 128)
# Initialize the Vicinity instance (using the basic backend and cosine metric)
vicinity = Vicinity.from_vectors_and_items(
    vectors=vectors,
    items=items,
    backend_type=Backend.BASIC,
    metric=Metric.COSINE
)
# Create a query vector
query_vector = np.random.rand(128)
# Query for nearest neighbors with a top-k search
results = vicinity.query(query_vector, k=3)
# Query for nearest neighbors with a threshold search
results = vicinity.query_threshold(query_vector, threshold=0.9)
# Query with a list of query vectors
query_vectors = np.random.rand(5, 128)
results = vicinity.query(query_vectors, k=3)

```

--------------------------------

### Deduplicate, Filter, and Sample with SemHash in Python

Source: https://minish.ai/packages/semhash

Demonstrates how to use SemHash to deduplicate texts, filter outliers, and select representative samples from a dataset. Requires the `datasets` library.  The example loads the 'ag_news' dataset, initializes a SemHash instance, and performs deduplication, outlier filtering, and representative sampling.

```Python
from datasets import load_dataset
from semhash import SemHash
# Load a dataset to deduplicate
texts = load_dataset("ag_news", split="train")["text"]
# Initialize a SemHash instance
semhash = SemHash.from_records(records=texts)
# Deduplicate the texts
deduplicated_texts = semhash.self_deduplicate().selected
# Filter outliers
filtered_texts = semhash.self_filter_outliers().selected
# Find representative texts
representative_texts = semhash.self_find_representative().selected
```

--------------------------------

### Install model2vec-rs CLI (Shell)

Source: https://minish.ai/packages/model2vec-rs

This command installs the model2vec-rs Command-Line Interface (CLI) tool globally using Cargo, making it available for terminal use.

```Shell
cargo install model2vec-rs
```

--------------------------------

### Install model2vec-rs CLI (Shell)

Source: https://minish.ai/packages/model2vec-rs/usage

This command installs the model2vec-rs Command-Line Interface (CLI) tool globally using Cargo, making it available for terminal use.

```Shell
cargo install model2vec-rs
```

--------------------------------

### Deduplicate, Filter, and Find Representative Texts with SemHash

Source: https://minish.ai/packages/semhash/introduction

This Python code demonstrates how to use SemHash to load a dataset, initialize SemHash from the dataset's text records, and then perform self-deduplication, outlier filtering, and representative sample selection. It assumes the 'datasets' library is installed.

```python
from datasets import load_dataset
from semhash import SemHash
# Load a dataset to deduplicate
texts = load_dataset("ag_news", split="train")["text"]
# Initialize a SemHash instance
semhash = SemHash.from_records(records=texts)
# Deduplicate the texts
deduplicated_texts = semhash.self_deduplicate().selected
# Filter outliers
filtered_texts = semhash.self_filter_outliers().selected
# Find representative texts
representative_texts = semhash.self_find_representative().selected
```

--------------------------------

### Use Model2Vec Model with SemHash

Source: https://minish.ai/packages/semhash/custom-encoders

This example demonstrates how to initialize SemHash with a custom Model2Vec embedding model. It loads a dataset, initializes the Model2Vec model from a pre-trained checkpoint, and then uses these to create a SemHash instance for deduplication.

```Python
from datasets import load_dataset
from model2vec import StaticModel
from semhash import SemHash
# Load a dataset to deduplicate
texts = load_dataset("ag_news", split="train")["text"]
# Load an embedding model (in this example, a multilingual model)
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
# Initialize a SemHash with the model and custom encoder
semhash = SemHash.from_records(records=texts, model=model)
# Deduplicate the texts
deduplicated_texts = semhash.self_deduplicate()

```

--------------------------------

### Load and Encode with Model2Vec-rs Library (Rust)

Source: https://minish.ai/packages/model2vec-rs/usage

Demonstrates loading a static embedding model from Hugging Face or a local path and encoding sentences using the model2vec-rs Rust library. It includes examples for default encoding and encoding with custom arguments like max length and batch size.

```Rust
use anyhow::Result;
use model2vec_rs::model::StaticModel;
fn main() -> Result<()> {
    // Load a model from the Hugging Face Hub or a local path.
    // Arguments: (repo_or_path, hf_token, normalize_embeddings, subfolder_in_repo)
    let model = StaticModel::from_pretrained(
        "minishlab/potion-base-8M", // Model ID from Hugging Face or local path to model directory
        None,                       // Optional: Hugging Face API token for private models
        None,                       // Optional: bool to override model's default normalization. `None` uses model's config.
        None                        // Optional: subfolder if model files are not at the root of the repo/path
    )?;
    let sentences = vec![
        "Hello world".to_string(),
        "Rust is awesome".to_string(),
    ];
    // Generate embeddings using default parameters
    // (Default max_length: Some(512), Default batch_size: 1024)
    let embeddings = model.encode(&sentences);
    // `embeddings` is a Vec<Vec<f32>>
    println!("Generated {} embeddings.", embeddings.len());
    // To generate embeddings with custom arguments:
    let custom_embeddings = model.encode_with_args(
        &sentences,
        Some(256), // Optional: custom max token length for truncation
        512,       // Custom batch size for processing
    );
    println!("Generated {} custom embeddings.", custom_embeddings.len());
    Ok(())
}

```

--------------------------------

### Load and Encode with Model2Vec-rs Library (Rust)

Source: https://minish.ai/packages/model2vec-rs

Demonstrates loading a static embedding model from Hugging Face or a local path and encoding sentences using the model2vec-rs Rust library. It includes examples for default encoding and encoding with custom arguments like max length and batch size.

```Rust
use anyhow::Result;
use model2vec_rs::model::StaticModel;
fn main() -> Result<()> {
    // Load a model from the Hugging Face Hub or a local path.
    // Arguments: (repo_or_path, hf_token, normalize_embeddings, subfolder_in_repo)
    let model = StaticModel::from_pretrained(
        "minishlab/potion-base-8M", // Model ID from Hugging Face or local path to model directory
        None,                       // Optional: Hugging Face API token for private models
        None,                       // Optional: bool to override model's default normalization. `None` uses model's config.
        None                        // Optional: subfolder if model files are not at the root of the repo/path
    )?;
    let sentences = vec![
        "Hello world".to_string(),
        "Rust is awesome".to_string(),
    ];
    // Generate embeddings using default parameters
    // (Default max_length: Some(512), Default batch_size: 1024)
    let embeddings = model.encode(&sentences);
    // `embeddings` is a Vec<Vec<f32>>
    println!("Generated {} embeddings.", embeddings.len());
    // To generate embeddings with custom arguments:
    let custom_embeddings = model.encode_with_args(
        &sentences,
        Some(256), // Optional: custom max token length for truncation
        512,       // Custom batch size for processing
    );
    println!("Generated {} custom embeddings.", custom_embeddings.len());
    Ok(())
}

```

--------------------------------

### Build Vicinity Vector Store with FAISS Backend

Source: https://minish.ai/packages/vicinity/usage

Demonstrates initializing a Vicinity vector store using the FAISS backend with the HNSW algorithm and Euclidean metric. This showcases configuring specific backend parameters for performance tuning.

```Python
from vicinity import Vicinity, Backend, Metric
vicinity = Vicinity.from_vectors_and_items(
    vectors=vectors,
    items=items,
    backend_type=Backend.FAISS,
    index_type="hnsw",
    metric=Metric.EUCLIDEAN
)
```

--------------------------------

### Build Vicinity Vector Store with Basic Backend

Source: https://minish.ai/packages/vicinity/usage

Initializes a Vicinity vector store using dummy data with the basic backend and cosine metric. This demonstrates the fundamental process of creating a vector store from vectors and associated items.

```Python
import numpy as np
from vicinity import Vicinity
# Create some dummy data as strings or other serializable objects
items = ["triforce", "master sword", "hylian shield", "boomerang", "hookshot"]
vectors = np.random.rand(len(items), 128)
# Initialize the Vicinity instance (using the basic backend and cosine metric)
vicinity = Vicinity.from_vectors_and_items(
    vectors=vectors,
    items=items,
)
```

--------------------------------

### Initialize Classifier with Model2Vec

Source: https://minish.ai/packages/model2vec/training

Demonstrates how to initialize a `StaticModelForClassification` using either a distilled model or a pre-trained model. This sets up the classifier for training on custom data.

```python
from model2vec.distill import distill
from model2vec.train import StaticModelForClassification
# From a distilled model
distilled_model = distill("baai/bge-base-en-v1.5")
classifier = StaticModelForClassification.from_static_model(model=distilled_model)
# From a pre-trained model: potion is the default
classifier = StaticModelForClassification.from_pretrained(model_name="minishlab/potion-base-32m")
```

--------------------------------

### Create Features with Tokenlearn

Source: https://minish.ai/packages/tokenlearn/usage

Creates features for Tokenlearn by processing a dataset using a specified model. It requires the model name, output directory, dataset path, dataset name, and dataset split as arguments.

```python
python3 -m tokenlearn.featurize \
    --model-name "baai/bge-base-en-v1.5" \
    --output-dir "data/c4_features" \
    --dataset-path "allenai/c4" \
    --dataset-name "en" \
    --dataset-split "train"
```

--------------------------------

### Create Features with Tokenlearn

Source: https://minish.ai/packages/tokenlearn

Creates features for Tokenlearn by processing a dataset using a specified model. It requires the model name, output directory, dataset path, dataset name, and dataset split as arguments.

```python
python3 -m tokenlearn.featurize \
    --model-name "baai/bge-base-en-v1.5" \
    --output-dir "data/c4_features" \
    --dataset-path "allenai/c4" \
    --dataset-name "en" \
    --dataset-split "train"
```

--------------------------------

### Query Top-K Nearest Neighbors in Vicinity

Source: https://minish.ai/packages/vicinity/usage

Performs a top-K nearest neighbor search on a Vicinity vector store. This example shows how to query for a specified number of most similar items to a given query vector.

```Python
# Create a query vector
query_vector = np.random.rand(128)
# Query for nearest neighbors with a top-k search
results = vicinity.query(query_vector, k=3)
```

--------------------------------

### Vicinity BASIC Backend Configuration

Source: https://minish.ai/packages/vicinity/supported-backends

Configuration for the BASIC backend in Vicinity, specifying the similarity metric. This backend is straightforward and relies on a single parameter for metric selection.

```python
BASIC:
  metric: cosine  # Similarity metric to use (cosine, euclidean)

```

--------------------------------

### Distill a Sentence Transformer Model with Model2Vec

Source: https://minish.ai/packages/model2vec/distillation

This Python code demonstrates how to distill a Sentence Transformer model using the `distill` function from the `model2vec.distill` module. The distilled model can be saved for later use. The process can be performed on a CPU and typically takes a few minutes.

```python
from model2vec.distill import distill
# Distill a Sentence Transformer model
m2v_model = distill(model_name="BAAI/bge-base-en-v1.5")
# Save the model
m2v_model.save_pretrained("m2v_model")
```

--------------------------------

### Vicinity PYNNDESCENT Backend Configuration

Source: https://minish.ai/packages/vicinity/supported-backends

Configuration for the PYNNDESCENT backend in Vicinity, enabling selection of the similarity metric and the number of neighbors for search.

```python
PYNNDESCENT:
  metric: cosine     # Similarity metric to use (cosine, euclidean, manhattan)
  n_neighbors: 15    # Number of neighbors to use for search

```

--------------------------------

### Use Sentence Transformer Model with SemHash

Source: https://minish.ai/packages/semhash/custom-encoders

This example shows how to integrate a Sentence Transformer model with SemHash for semantic deduplication. It involves loading a dataset, initializing a Sentence Transformer model, and then configuring SemHash to use this model for processing text records.

```Python
from datasets import load_dataset
from semhash import SemHash
from sentence_transformers import SentenceTransformer
# Load a dataset to deduplicate
texts = load_dataset("ag_news", split="train")["text"]
# Load a sentence-transformers model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
# Initialize a SemHash with the model and custom encoder
semhash = SemHash.from_records(records=texts, model=model)
# Deduplicate the texts
deduplicated_texts = semhash.self_deduplicate()

```

--------------------------------

### Push and Load Vector Store to Hugging Face Hub

Source: https://minish.ai/packages/vicinity/usage

This snippet demonstrates how to push a vector store to the Hugging Face Hub and then load it back. It utilizes the `push_to_hub` and `load_from_hub` methods, which require a repository ID. Optional parameters include an authentication token for private repositories and a flag to specify if the repository should be private.

```Python
vicinity.push_to_hub(repo_id='minishlab/my-vicinity-repo')
vicinity = Vicinity.load_from_hub(repo_id='minishlab/my-vicinity-repo')
```

--------------------------------

### Vicinity USEARCH Backend Configuration

Source: https://minish.ai/packages/vicinity/supported-backends

Configuration for the USEARCH backend in Vicinity, supporting various similarity metrics and parameters for graph construction and search.

```python
USEARCH:
  metric: cos             # Similarity metric to use (cos, ip, l2sq, hamming, tanimoto)
  connectivity: 16        # Number of connections per node in the graph
  expansion_add: 128      # Number of candidates considered during graph construction
  expansion_search: 64    # Number of candidates considered during search

```

--------------------------------

### Load Model2Vec Model from HuggingFace

Source: https://minish.ai/packages/model2vec/models

Demonstrates how to load a pre-trained Model2Vec model from the HuggingFace hub using the `from_pretrained` method. This is the standard way to integrate these models into your projects.

```Python
from sentence_transformers import SentenceTransformer

# Example: Loading the potion-base-32M model
model = SentenceTransformer('potion-base-32M')

# You can then use the model for inference:
# embeddings = model.encode('This is a sample sentence.')
```

--------------------------------

### Use Model2Vec for Embeddings

Source: https://minish.ai/packages/model2vec/introduction

Demonstrates how to load a pre-trained Model2Vec model and generate embeddings for a list of sentences. It requires the 'model2vec' library and a pre-trained model from HuggingFace Hub.

```Python
from model2vec import Model2Vec
model = Model2Vec.from_pretrained("minishlab/potion-base-8M")
embeddings = model.encode(["Hello world", "Static embeddings are great!"])
```

--------------------------------

### Vicinity ANNOY Backend Configuration

Source: https://minish.ai/packages/vicinity/supported-backends

Configuration for the ANNOY backend in Vicinity, allowing selection of the similarity metric and the number of trees for indexing. It also supports an optional dataset length parameter.

```python
ANNOY:
  metric: cosine  # Similarity metric to use (dot, euclidean, cosine)
  trees: 100      # Number of trees to use for indexing
  length: None    # Optional length of the dataset

```

--------------------------------

### Vicinity VOYAGER Backend Configuration

Source: https://minish.ai/packages/vicinity/supported-backends

Configuration for the VOYAGER backend in Vicinity, allowing specification of the similarity space and parameters for index construction.

```python
VOYAGER:
  metric: cosine  # Similarity space to use (cosine, l2)
  ef_construction: 200 # The number of vectors that this index searches through when inserting a new vector into the index
  m: 16               # The number of connections between nodes in the tree’s internal data structure

```

--------------------------------

### Load Dataset for Training

Source: https://minish.ai/packages/model2vec/training

Loads a dataset from the Hugging Face Hub for training the classification model. It splits the dataset into training and testing sets and extracts the text and label columns.

```python
from datasets import load_dataset
# Load the train and test splits of the dataset
train, test = load_dataset("setfit/subj", split=["train", "test"])
# Create X and y
X_train, y_train = train["text"], train["label"]
X_test, y_test   = test["text"], test["label"]
```

--------------------------------

### Save and Load Vicinity Vector Store

Source: https://minish.ai/packages/vicinity/usage

Demonstrates how to save a Vicinity vector store to disk and subsequently load it back. This is crucial for persisting and reusing trained vector stores.

```Python
vicinity.save('my_vector_store')
vicinity = Vicinity.load('my_vector_store')
```

--------------------------------

### Vicinity HNSW Backend Configuration

Source: https://minish.ai/packages/vicinity/supported-backends

Configuration for the HNSW backend in Vicinity, allowing the choice of similarity space and parameters for index construction and connections per layer.

```python
HNSW:
  metric: cosine      # Similarity space to use (cosine, l2)
  ef_construction: 200 # Size of the dynamic list during index construction
  m: 16               # Number of connections per layer

```

--------------------------------

### Evaluate Vector Store Performance

Source: https://minish.ai/packages/vicinity/usage

This code evaluates the performance of a vector store by calculating queries per second (QPS) and recall. It requires the full dataset vectors and a subset of these vectors to be used as query vectors. The evaluation can be configured with parameters like `k` for the number of nearest neighbors and `epsilon` for the recall calculation threshold.

```Python
# Use the first 1000 vectors as query vectors
query_vectors = vectors[:1000]
# Evaluate the Vicinity instance by measuring the queries per second and recall
qps, recall = vicinity.evaluate(
    full_vectors=vectors,
    query_vectors=query_vectors,
)
```

--------------------------------

### Train Model with Tokenlearn

Source: https://minish.ai/packages/tokenlearn/usage

Trains a Tokenlearn model using pre-generated features. This command requires the model name, the path to the features data, and a path to save the trained model.

```python
python3 -m tokenlearn.train \
    --model-name "baai/bge-base-en-v1.5" \
    --data-path "data/c4_features" \
    --save-path "<path-to-save-model>"
```

--------------------------------

### Use Model2Vec for Embeddings

Source: https://minish.ai/packages/model2vec

Demonstrates how to load a pre-trained Model2Vec model and generate embeddings for a list of sentences. It requires the 'model2vec' library and a pre-trained model from HuggingFace Hub.

```Python
from model2vec import Model2Vec
model = Model2Vec.from_pretrained("minishlab/potion-base-8M")
embeddings = model.encode(["Hello world", "Static embeddings are great!"])
```

--------------------------------

### Train Model with Tokenlearn

Source: https://minish.ai/packages/tokenlearn

Trains a Tokenlearn model using pre-generated features. This command requires the model name, the path to the features data, and a path to save the trained model.

```python
python3 -m tokenlearn.train \
    --model-name "baai/bge-base-en-v1.5" \
    --data-path "data/c4_features" \
    --save-path "<path-to-save-model>"
```

--------------------------------

### Train and Evaluate Classifier

Source: https://minish.ai/packages/model2vec/training

Trains the initialized classifier using the prepared training data and then evaluates its performance on the test data. The `fit` method trains the model, and the `evaluate` method returns the results.

```python
# Train the classifier
classifier = classifier.fit(X_train, y_train)
# Evaluate the classifier
results = classifier.evaluate(X_test, y_test)
```

--------------------------------

### Save and Load Model2Vec Classifier Pipeline

Source: https://minish.ai/packages/model2vec/training

Converts a trained classifier into a scikit-learn compatible pipeline, allowing it to be saved locally or pushed to the HuggingFace Hub. It also shows how to load the pipeline back for inference.

```python
pipeline = classifier.to_pipeline()
pipeline.save_pretrained(path)
pipeline.push_to_hub("my_org/my_classifier")

from model2vec.inference import StaticModelPipeline
pipeline = StaticModelPipeline.from_pretrained("my_org/my_classifier")
```

--------------------------------

### Initialize SemHash Instance

Source: https://minish.ai/packages/semhash/deduplication

Initializes a SemHash instance with a dataset. You can specify the model to use, with 'minishlab/potion-base-8M' as the default for English text and 'minishlab/potion-multilingual-128M' for multilingual data. Custom SentenceTransformer models are also supported.

```Python
from datasets import load_dataset
from semhash import SemHash
# Initialize a SemHash instance
semhash = SemHash.from_records(records=texts)
```

--------------------------------

### Vicinity FAISS Backend Configuration

Source: https://minish.ai/packages/vicinity/supported-backends

Configuration for the FAISS backend in Vicinity, supporting various index types and associated parameters like nlist, m, nbits, and refine_nbits. The similarity metric can also be specified.

```python
FAISS:
  metric: cosine       # Similarity metric to use (cosine, l2)
  index_type: hnsw     # Type of FAISS index (flat, ivf, hnsw, lsh, scalar, pq, ivf_scalar, ivfpq, ivfpqr)
  nlist: 100           # Number of cells for IVF indexes
  m: 8                 # Number of subquantizers for PQ and HNSW indexes
  nbits: 8             # Number of bits for LSH and PQ indexes
  refine_nbits: 8      # Number of bits for the refinement stage in IVFPQR indexes

```
Tokens:
Raw