Zvec

https://zvec.org/en/docs
Admin
Zvec is an open-source, in-process vector database that provides fast, scalable similarity search...
Tokens:34,493
Snippets:225
Trust Score:4.8
Update:7 months ago
Show doc for...
Tokens:
Raw
### Install Zvec Python Package

Source: https://zvec.org/en/docs/quickstart

Installs the Zvec Python client library using pip. This is the first step to using Zvec in your Python projects.

```python
pip install zvec
```

--------------------------------

### Install Git using apt-get

Source: https://zvec.org/en/docs/benchmarks

Installs the Git version control system on Ubuntu. This is a prerequisite for cloning repositories. It updates the package list and then installs git.

```bash
apt-get update
apt install git
```

--------------------------------

### Retrieve Zvec Document by ID

Source: https://zvec.org/en/docs/quickstart

Fetches a specific document from a Zvec collection using its unique identifier. This is a direct lookup operation.

```python
collection.get(id="document_id")
```

--------------------------------

### Perform Basic Zvec Similarity Search

Source: https://zvec.org/en/docs/quickstart

Executes a basic similarity search in a Zvec collection using a query vector. Results are returned ranked by their similarity score to the query vector.

```python
collection.query(vector=[...], limit=10)
```

--------------------------------

### Install VectorDBBench and dependencies

Source: https://zvec.org/en/docs/benchmarks

Clones the VectorDBBench repository, installs its dependencies using pip, and optionally configures it to use the Aliyun PyPI mirror for faster downloads. This command sets up the benchmarking framework.

```bash
# Clone VectorDBBench
git clone https://github.com/zilliztech/VectorDBBench.git
cd VectorDBBench

# Install deps
pip install -U pip
pip install -e .

# If you experience slow downloads or connection issues, you can try Aliyun PyPI mirror
# pip install -U pip -i https://mirrors.aliyun.com/pypi/simple
# pip install -e . -i https://mirrors.aliyun.com/pypi/simple
```

--------------------------------

### Clone Zvec Repository and Install from Source (Bash)

Source: https://zvec.org/en/docs/build

Clones the Zvec repository, including submodules, and then installs the package using pip. This is the primary method for building Zvec from source.

```bash
# Clone the repository
git clone --recurse-submodules https://github.com/alibaba/zvec.git
cd zvec

# Install from source
pip install .
```

--------------------------------

### View Zvec Collection Statistics

Source: https://zvec.org/en/docs/quickstart

Retrieves and displays statistics for a Zvec collection. This can include information about the number of documents, index status, and memory usage.

```python
print(collection.stats)
```

--------------------------------

### Install Python 3.11+ and set up virtual environment

Source: https://zvec.org/en/docs/benchmarks

Installs Python 3.11 or higher along with necessary development headers and creates a virtual environment for project dependencies. This ensures a clean and isolated Python environment.

```bash
apt-get update
apt install python3-full python3-venv python3-dev

cd /opt
python3 -m venv venv
source venv/bin/activate
```

--------------------------------

### View Zvec Collection Schema

Source: https://zvec.org/en/docs/quickstart

Retrieves and displays the schema definition of a Zvec collection. The schema defines the structure of documents, including scalar fields and vector embeddings.

```python
print(collection.schema)
```

--------------------------------

### Optimize Zvec Collection

Source: https://zvec.org/en/docs/quickstart

Optimizes a Zvec collection to improve query performance. This operation reorganizes data within the collection for faster retrieval.

```python
collection.optimize()
```

--------------------------------

### Install Zvec Python package

Source: https://zvec.org/en/docs/benchmarks

Installs a specific version (v0.1.1) of the Zvec Python package using pip. This command ensures that the Zvec library is available for use in the current Python environment.

```bash
pip install zvec==v0.1.1
```

--------------------------------

### Perform Filtered Zvec Similarity Search

Source: https://zvec.org/en/docs/quickstart

Performs a similarity search in a Zvec collection that includes a filter condition. Only documents matching the filter are considered in the search results.

```python
collection.query(vector=[...], filter="field_name = 'value'", limit=10)
```

--------------------------------

### Delete Zvec Documents by Filter

Source: https://zvec.org/en/docs/quickstart

Deletes multiple documents from a Zvec collection based on a specified filter condition. All documents matching the filter will be removed.

```python
collection.delete_by_filter(filter="publish_year < 1900")
```

--------------------------------

### Custom Embedding Functions

Source: https://zvec.org/en/docs/embedding

Guide on creating custom embedding functions in Zvec using protocol base classes and framework-specific base classes.

```APIDOC
## Custom Embedding Functions

### Description
Zvec allows for the creation of custom embedding functions by providing protocol base classes and framework-specific base classes. This enables integration of custom models or logic for generating embeddings.

### Protocol Base Classes
- `DenseEmbeddingFunction[T]`: Protocol for dense embeddings.
- `SparseEmbeddingFunction[T]`: Protocol for sparse embeddings.

### Framework-Specific Base Classes
- `SentenceTransformerFunctionBase`: Base class for Sentence Transformers models.
- `QwenFunctionBase`: Base class for Qwen Dashscope API.

### Example 1: Custom Dense Embedding from Scratch

#### Description
An example demonstrating how to implement a custom dense embedding function from scratch.

#### Method
N/A (Class Definition and Method Calls)

#### Endpoint
N/A

#### Parameters
(Defined within the custom class constructor and methods)

#### Request Example
```python
from zvec.extension import DenseEmbeddingFunction
from zvec.common.constants import TEXT, DenseVectorType
import numpy as np

class MyCustomDenseEmbedding(DenseEmbeddingFunction[TEXT]):
    """Custom dense embedding function example"""
    
    def __init__(self, model_name: str = "custom-model", **kwargs):
        self._model_name = model_name
        self._dimension = 768  # Custom dimension
        self._extra_params = kwargs
        # Initialize your model
        self._model = self._load_model()
    
    @property
    def dimension(self) -> int:
        """Return embedding vector dimension"""
        return self._dimension
    
    @property
    def extra_params(self) -> dict:
        """Return extra parameters"""
        return self._extra_params
    
    def _load_model(self):
        """Load your custom model"""
        # Implement your model loading logic here
        pass
    
    def embed(self, input: str) -> DenseVectorType:
        """
        Generate dense embedding vector
        
        Args:
            input: Input text
            
        Returns:
            DenseVectorType: List of floats, length = self.dimension
        """
        # Input validation
        if not isinstance(input, str):
            raise TypeError(f"Expected str, got {type(input).__name__}")
        
        input = input.strip()
        if not input:
            raise ValueError("Input cannot be empty")
        
        # Generate embedding using your model
        # embedding = self._model.encode(input)
        # return embedding.tolist()
        
        # Example: return random vector
        return np.random.randn(self._dimension).tolist()
    
    def __call__(self, input: str) -> DenseVectorType:
        """Make the function callable"""
        return self.embed(input)


# Use custom embedding
custom_emb = MyCustomDenseEmbedding(model_name="my-model")
vector = custom_emb.embed("Test text")
print(f"Dimensions: {len(vector)}")
```

#### Response
#### Success Response (N/A - Method Return)
- **embed() return value** (DenseVectorType) - The generated dense embedding vector.
```

--------------------------------

### QwenReRanker - Dashscope API Reranking

Source: https://zvec.org/en/docs/reranker

This section explains how to use the QwenReRanker for reranking via the Dashscope API. It requires an API key and provides an example of its usage.

```APIDOC
## QwenReRanker - Dashscope API Reranking

### Description
Uses Qwen Dashscope API for reranking. Requires a Dashscope API key.

### Method
N/A (This is a class for API interaction)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
```python
from zvec.extension import QwenReRanker
from zvec import Doc

reranker = QwenReRanker(
    query="What is a vector database",
    model="gte-rerank-v2",
    api_key="your-dashscope-api-key",
    topn=3,
    rerank_field="content",
)

documents = {
    "vector1": [
        Doc(
            id="1",
            fields={
                "content": "Vector databases store and retrieve vectors"
            },
        ),
        Doc(
            id="2",
            fields={
                "content": "Relational databases store structured data"
            },
        ),
        Doc(
            id="3",
            fields={
                "content": "Vector retrieval is based on similarity computation"
            },
        ),
    ],
}

# Perform reranking
reranked_docs = reranker.rerank(documents)

for doc in reranked_docs:
    print(doc)
```

### Response
#### Success Response (200)
N/A (API interaction, returns reranked documents)

#### Response Example
```
Doc(id='1', score=0.95, fields={'content': 'Vector databases store and retrieve vectors'})
Doc(id='3', score=0.85, fields={'content': 'Vector retrieval is based on similarity computation'})
Doc(id='2', score=0.2, fields={'content': 'Relational databases store structured data'})
```
```

--------------------------------

### Index DDL Operations

Source: https://zvec.org/en/docs/collections/schema-evolution

Manage indexes on vector and scalar fields to optimize search performance and filtering capabilities.

```APIDOC
## Index DDL Operations

### Create an Index

Creates or replaces an index on a specified field. Vector fields require an index for similarity search, while scalar fields can use inverted indexes for filtering.

*   **Method**: POST (Conceptual - API call)
*   **Endpoint**: `/collections/{collection_name}/indexes`

#### Parameters

*   `field_name` (string) - Required - The name of the field to create an index on.
*   `index_param` (object) - Required - Defines the type and parameters of the index.
    *   For Vector Fields:
        *   `HnswIndexParam` (object) - Parameters for HNSW index.
        *   `IVFIndexParam` (object) - Parameters for IVF index.
        *   `FlatIndexParam` (object) - Parameters for FLAT index.
            *   `metric_type` (enum) - Required - The distance metric (e.g., `COSINE`, `EUCLIDEAN`).
    *   For Scalar Fields:
        *   `InvertIndexParam` (object) - Parameters for inverted index.

### Request Example (Vector Field - FLAT Index)

```json
{
  "field_name": "dense_embedding",
  "index_param": {
    "type": "FlatIndexParam",
    "metric_type": "COSINE"
  }
}
```

### Request Example (Scalar Field - Inverted Index)

```json
{
  "field_name": "publish_year",
  "index_param": {
    "type": "InvertIndexParam"
  }
}
```

### Drop an Index

Removes an index from a scalar field. Dropping indexes on vector fields is not allowed as they are mandatory for similarity search.

*   **Method**: DELETE (Conceptual - API call)
*   **Endpoint**: `/collections/{collection_name}/indexes/{field_name}`

#### Parameters

*   `field_name` (string) - Required - The name of the field whose index should be dropped. Must be a scalar field.
```

--------------------------------

### Create Custom Sparse Embedding from Scratch (Python)

Source: https://zvec.org/en/docs/embedding

Demonstrates how to create a custom sparse embedding function by inheriting from SparseEmbeddingFunction. This involves implementing tokenizer loading and the embedding logic. The example returns a simple term frequency vector.

```python
from zvec.extension import SparseEmbeddingFunction
from zvec.common.constants import TEXT, SparseVectorType
from typing import Dict


class MyCustomSparseEmbedding(SparseEmbeddingFunction[TEXT]):
    """Custom sparse embedding function example"""
    
    def __init__(self, vocab_size: int = 30000, **kwargs):
        self._vocab_size = vocab_size
        self._extra_params = kwargs
        self._tokenizer = self._load_tokenizer()
    
    @property
    def extra_params(self) -> dict:
        return self._extra_params
    
    def _load_tokenizer(self):
        """Load tokenizer"""
        # Implement your tokenizer loading logic
        pass
    
    def embed(self, input: str) -> SparseVectorType:
        """
        Generate sparse embedding vector
        
        Args:
            input: Input text
            
        Returns:
            SparseVectorType: Dictionary {dimension_index: weight}, contains only non-zero values
        """
        if not isinstance(input, str):
            raise TypeError(f"Expected str, got {type(input).__name__}")
        
        input = input.strip()
        if not input:
            raise ValueError("Input cannot be empty")
        
        # Implement your sparse embedding logic
        # tokens = self._tokenizer.tokenize(input)
        # sparse_vec = self._compute_sparse_representation(tokens)
        
        # Example: return simple term frequency vector
        sparse_vec = {
            100: 0.5,
            250: 1.2,
            500: 0.8
        }
        
        # Ensure sorted by index
        return dict(sorted(sparse_vec.items()))
    
    def __call__(self, input: str) -> SparseVectorType:
        return self.embed(input)


# Use custom sparse embedding
sparse_emb = MyCustomSparseEmbedding(vocab_size=50000)
sparse_vec = sparse_emb.embed("Test text")
print(f"Non-zero dimensions: {len(sparse_vec)}")

```

--------------------------------

### Python Vocabulary Mapping and Sparse Vector Example

Source: https://zvec.org/en/docs/concepts/vector-embedding

Demonstrates a Python dictionary for vocabulary mapping, where terms are assigned unique integer IDs. It also shows a sparse vector represented as a dictionary of {index: weight}, highlighting how non-zero dimensions correspond to terms in the vocabulary. This format is efficient for high-dimensional data where most values are zero.

```python
vocab = {
    "animal": 124,
    "dog": 309,
    "pet": 1822,
    "puppy": 4017,
    "cat": 5001,
    "kitten": 7890,
    # ... (many other terms fill the rest of the dictionary)
    # Total size ≈ 50,000
}

sparse_vector = {
    4017: 2.31,   # "puppy"
    309: 1.85,    # "dog"
    1822: 1.12,   # "pet"
    124: 0.76     # "animal"
}
# The remaining ~49,996 dimensions are implicitly zero.
```

--------------------------------

### View Collection Path - Python & Node.js

Source: https://zvec.org/en/docs/collections/inspect

Get the filesystem path where the collection data is stored. This is useful for locating collection files on disk for backup or direct inspection.

```python
print(collection.path)
```

```javascript
console.log(collection.path);
```

--------------------------------

### Use Custom SentenceTransformer Model (Python)

Source: https://zvec.org/en/docs/embedding

Shows how to create a custom dense embedding function by inheriting from SentenceTransformerFunctionBase. This allows using different Sentence Transformers models. The example initializes with 'all-mpnet-base-v2' and demonstrates using a multilingual model.

```python
from zvec.extension.sentence_transformer_function import SentenceTransformerFunctionBase
from zvec.extension import DenseEmbeddingFunction
from zvec.common.constants import TEXT, DenseVectorType
from typing import Literal, Optional


class CustomSentenceTransformerEmbedding(
    SentenceTransformerFunctionBase, 
    DenseEmbeddingFunction[TEXT]
):
    """Using custom Sentence Transformer model"""
    
    def __init__(
        self,
        model_name: str = "all-mpnet-base-v2",  # Use a different model
        model_source: Literal["huggingface", "modelscope"] = "huggingface",
        normalize_embeddings: bool = True,
        **kwargs
    ):
        # Initialize base class
        SentenceTransformerFunctionBase.__init__(
            self, 
            model_name=model_name,
            model_source=model_source,
        )
        
        self._normalize_embeddings = normalize_embeddings
        self._extra_params = kwargs
        
        # Load model and get dimension
        model = self._get_model()
        self._dimension = model.get_sentence_embedding_dimension()
    
    @property
    def dimension(self) -> int:
        return self._dimension
    
    @property
    def extra_params(self) -> dict:
        return self._extra_params
    
    def embed(self, input: str) -> DenseVectorType:
        if not isinstance(input, str):
            raise TypeError(f"Expected str, got {type(input).__name__}")
        
        input = input.strip()
        if not input:
            raise ValueError("Input cannot be empty")
        
        model = self._get_model()
        embedding = model.encode(
            input,
            convert_to_numpy=True,
            normalize_embeddings=self._normalize_embeddings
        )
        
        return embedding.tolist()
    
    def __call__(self, input: str) -> DenseVectorType:
        return self.embed(input)


# Use custom model
# Use larger MPNet model (768 dimensions)
custom_emb = CustomSentenceTransformerEmbedding(
    model_name="all-mpnet-base-v2"
)
vector = custom_emb.embed("High-quality text embedding")
print(f"Dimensions: {len(vector)}")  # 768

# Use multilingual model
multilingual_emb = CustomSentenceTransformerEmbedding(
    model_name="paraphrase-multilingual-MiniLM-L12-v2"
)

```

--------------------------------

### API-Based Reranking with QwenReRanker

Source: https://zvec.org/en/docs/reranker

Performs reranking using the Qwen Dashscope API. Requires a Dashscope API key and specifies the model, query, and fields to rerank. Ensure 'openai' and 'dashscope' packages are installed.

```python
from zvec.extension import QwenReRanker
from zvec import Doc

reranker = QwenReRanker(
    query="What is a vector database",
    model="gte-rerank-v2",
    api_key="your-dashscope-api-key",
    topn=3,
    rerank_field="content",
)

documents = {
    "vector1": [
        Doc(
            id="1",
            fields={
                "content": "Vector databases store and retrieve vectors"
            },
        ),
        Doc(
            id="2",
            fields={
                "content": "Relational databases store structured data"
            },
        ),
        Doc(
            id="3",
            fields={
                "content": "Vector retrieval is based on similarity computation"
            },
        ),
    ],
}

# Perform reranking
reranked_docs = reranker.rerank(documents)

for doc in reranked_docs:
    print(doc)
```

--------------------------------

### Custom Dense Embedding Function Implementation

Source: https://zvec.org/en/docs/embedding

Example of creating a custom dense embedding function by inheriting from `DenseEmbeddingFunction`. This allows for custom model loading, dimension specification, and embedding generation logic. Input validation is included.

```python
from zvec.extension import DenseEmbeddingFunction
from zvec.common.constants import TEXT, DenseVectorType
from typing import Optional
import numpy as np


class MyCustomDenseEmbedding(DenseEmbeddingFunction[TEXT]):
    """Custom dense embedding function example"""
    
    def __init__(self, model_name: str = "custom-model", **kwargs):
        self._model_name = model_name
        self._dimension = 768  # Custom dimension
        self._extra_params = kwargs
        # Initialize your model
        self._model = self._load_model()
    
    @property
    def dimension(self) -> int:
        """Return embedding vector dimension"""
        return self._dimension
    
    @property
    def extra_params(self) -> dict:
        """Return extra parameters"""
        return self._extra_params
    
    def _load_model(self):
        """Load your custom model"""
        # Implement your model loading logic here
        # e.g., return YourModelClass.from_pretrained(self._model_name)
        pass
    
    def embed(self, input: str) -> DenseVectorType:
        """
        Generate dense embedding vector
        
        Args:
            input: Input text
            
        Returns:
            DenseVectorType: List of floats, length = self.dimension
        """
        # Input validation
        if not isinstance(input, str):
            raise TypeError(f"Expected str, got {type(input).__name__}")
        
        input = input.strip()
        if not input:
            raise ValueError("Input cannot be empty")
        
        # Generate embedding using your model
        # embedding = self._model.encode(input)
        # return embedding.tolist()
        
        # Example: return random vector
        return np.random.randn(self._dimension).tolist()
    
    def __call__(self, input: str) -> DenseVectorType:
        """Make the function callable"""
        return self.embed(input)


# Use custom embedding
custom_emb = MyCustomDenseEmbedding(model_name="my-model")
vector = custom_emb.embed("Test text")
print(f"Dimensions: {len(vector)}")
```

--------------------------------

### Delete Zvec Document by ID

Source: https://zvec.org/en/docs/quickstart

Deletes a specific document from a Zvec collection using its ID. This is a direct removal operation.

```python
collection.delete(ids="document_id")
```

--------------------------------

### Build Index for Cohere 1M dataset

Source: https://zvec.org/en/docs/benchmarks

Initiates the index building process for the Cohere 1M dataset using VectorDBBench with Zvec. It specifies parameters like dataset path, database label, concurrency levels, quantization type, M, and EF-search values.

```bash
vectordbbench zvec --path Performance768D1M --db-label 16c64g-v0.1 --case-type Performance768D1M --num-concurrency 12,14,16,18,20 --quantize-type int8 --m 15 --ef-search 180
```

--------------------------------

### Build Index for Cohere 10M dataset

Source: https://zvec.org/en/docs/benchmarks

Initiates the index building process for the Cohere 10M dataset using VectorDBBench with Zvec. It specifies parameters like dataset path, database label, concurrency levels, quantization type, M, and EF-search values.

```bash
vectordbbench zvec --path Performance768D10M --db-label 16c64g-v0.1 --case-type Performance768D10M --num-concurrency 12,14,16,18,20 --quantize-type int8 --m 50 --ef-search 118 --is-using-refiner
```

--------------------------------

### Create and Open a Collection in Python

Source: https://zvec.org/en/docs/collections/create

This Python code snippet illustrates the process of creating and opening a Zvec collection. It utilizes the `create_and_open()` function, passing the desired storage `path`, a pre-defined `schema`, and the `collection_option` object. The function returns a `Collection` handle, ready for subsequent data operations like insertions and queries.

```python
import zvec

collection = zvec.create_and_open(
    path="/path/to/my/collection",
    schema=collection_schema,
    option=collection_option,
)
```

--------------------------------

### Set Custom Scikit-build Directory for Troubleshooting (Bash)

Source: https://zvec.org/en/docs/build

Sets an environment variable to specify a custom build directory for scikit-build. This helps in managing build caches and diagnosing build errors by providing access to detailed logs.

```bash
export SKBUILD_BUILD_DIR=/tmp/build  # Or other directory
pip install .
```

--------------------------------

### Run Benchmark for Cohere 1M dataset

Source: https://zvec.org/en/docs/benchmarks

Executes the benchmark tests for the Cohere 1M dataset using VectorDBBench with Zvec. This command runs the performance evaluation, skipping index dropping and loading if they have already been performed.

```bash
vectordbbench zvec --path Performance768D1M --db-label 16c64g-v0.1 --case-type Performance768D1M --num-concurrency 12,14,16,18,20 --quantize-type int8 --m 15 --ef-search 180 --skip-drop-old --skip-load
```

--------------------------------

### Custom Reranking Functions

Source: https://zvec.org/en/docs/reranker

Guide on creating custom reranking functions by inheriting from the `RerankFunction` base class.

```APIDOC
## Custom Reranking Functions

### Description
Learn how to create your own reranking functions. Reranking functions need to inherit from the `RerankFunction` base class (exported as `ReRanker`).

### Method
N/A (Guide for custom implementation)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### Run Benchmark for Cohere 10M dataset

Source: https://zvec.org/en/docs/benchmarks

Executes the benchmark tests for the Cohere 10M dataset using VectorDBBench with Zvec. This command runs the performance evaluation, skipping index dropping and loading if they have already been performed.

```bash
vectordbbench zvec --path Performance768D10M --db-label 16c64g-v0.1 --case-type Performance768D10M --num-concurrency 12,14,16,18,20 --quantize-type int8 --m 50 --ef-search 118 --is-using-refiner --skip-drop-old --skip-load
```

--------------------------------

### Drop a Scalar Field from a Collection

Source: https://zvec.org/en/docs/collections/schema-evolution

Permanently removes a scalar field and all its associated data from a Zvec collection. This operation is irreversible.

```python
collection.drop_column(field_name="publish_year")
```

```javascript
await collection.drop_column({ field_name: "publish_year" });
```

--------------------------------

### Configure Zvec Global Settings (Node.js)

Source: https://zvec.org/en/docs/config

Initializes Zvec with custom global settings, including log type, log level, and the number of query threads. This function should be called once at application startup before any database operations. Note: The Node.js code is not provided in the input text, but the structure is implied.

```javascript
// Node.js equivalent (conceptual, actual implementation may vary)
// const zvec = require('zvec');
// zvec.init({
//     logType: zvec.LogType.CONSOLE,
//     logLevel: zvec.LogLevel.WARN,
//     queryThreads: 4
// });
```

--------------------------------

### Use Custom Qwen Reranker

Source: https://zvec.org/en/docs/reranker

Demonstrates how to initialize and use a custom Qwen reranker for reordering documents based on relevance to a query. Requires API key and specifies the top N documents to consider.

```python
from zvec.extension import CustomQwenReRanker

custom_qwen_reranker = CustomQwenReRanker(
    query="What is a vector database",
    api_key="your-dashscope-api-key",
    topn=5
)
reranked = custom_qwen_reranker.rerank(documents, rerank_field="text")
```

--------------------------------

### Configure Collection Options in Python

Source: https://zvec.org/en/docs/collections/create

This Python code snippet demonstrates how to configure optional settings for a Zvec collection. It shows the creation of a `CollectionOption` object, specifying `read_only` and `enable_mmap` parameters to control the collection's runtime behavior. These options influence how the collection interacts with the file system and data modification capabilities.

```python
import zvec

collection_option = zvec.CollectionOption(read_only=False, enable_mmap=True)
```

--------------------------------

### DefaultLocalReRanker - Local Cross-Encoder Reranking

Source: https://zvec.org/en/docs/reranker

This section describes how to use the DefaultLocalReRanker for local reranking using a Cross-Encoder model. It includes initialization and usage examples.

```APIDOC
## DefaultLocalReRanker - Local Cross-Encoder Reranking

### Description
Uses a Cross-Encoder model for reranking.

### Method
N/A (This is a class for local execution)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
```python
from zvec.extension import DefaultLocalReRanker
from zvec import Doc

# Initialize reranker
reranker = DefaultLocalReRanker(
    query="What are machine learning algorithms",
    topn=5,
    rerank_field="content"  # Specify the field to rerank
)

# Prepare document list
documents = {
    "vector1": [
        Doc(
            id="1",
            fields={
                "content": "Machine learning is a subset of artificial intelligence that focuses on building systems that can learn from data."
            },
        ),
        Doc(
            id="2",
            fields={
                "content": "The weather is nice today with clear skies and sunshine."
            },
        ),
        Doc(
            id="3",
            fields={
                "content": "Deep learning is a specialized branch of machine learning using neural networks with multiple layers."
            },
        ),
    ],
}

# Perform reranking
reranked_docs = reranker.rerank(documents)

for doc in reranked_docs:
    print(doc)
```

### Response
#### Success Response (200)
N/A (Local execution, returns reranked documents)

#### Response Example
```
Doc(id='1', score=0.9, fields={'content': 'Machine learning is a subset of artificial intelligence that focuses on building systems that can learn from data.'})
Doc(id='3', score=0.8, fields={'content': 'Deep learning is a specialized branch of machine learning using neural networks with multiple layers.'})
Doc(id='2', score=0.1, fields={'content': 'The weather is nice today with clear skies and sunshine.'})
```
```

--------------------------------

### View Collection Options - Python & Node.js

Source: https://zvec.org/en/docs/collections/inspect

Retrieve and display the runtime options configured for a collection. These options govern behavior such as read-only mode and memory mapping, affecting how the collection operates.

```python
print(collection.option)
```

```javascript
console.log(collection.option);
```

--------------------------------

### Drop an Index from a Scalar Field

Source: https://zvec.org/en/docs/collections/schema-evolution

Removes an index from a scalar field in a Zvec collection. Dropping indexes on vector fields is not permitted as they are required for similarity search.

```python
collection.drop_index(field_name="publish_year")
```

```javascript
await collection.drop_index({ field_name: "publish_year" });
```

--------------------------------

### Column DDL Operations

Source: https://zvec.org/en/docs/collections/schema-evolution

Manage scalar fields within a collection, including adding new fields, removing existing ones, and modifying their names or data types.

```APIDOC
## Column DDL Operations

### Add a Column

Adds a new scalar field to an existing collection.

*   **Method**: POST (Conceptual - API call)
*   **Endpoint**: `/collections/{collection_name}/columns`

#### Parameters

*   `field_schema` (object) - Required - Defines the name and data type of the new field.
    *   `name` (string) - Required - The name of the new field.
    *   `data_type` (enum) - Required - The data type of the new field (e.g., `INT32`, `FLOAT`, `STRING`).
*   `expression` (string) - Required - Specifies the default value for existing documents. Must evaluate to a number for numerical fields.

### Request Example

```json
{
  "field_schema": {
    "name": "rating",
    "data_type": "INT32"
  },
  "expression": "5"
}
```

### Drop a Column

Permanently removes a scalar field and all its associated data from the collection.

*   **Method**: DELETE (Conceptual - API call)
*   **Endpoint**: `/collections/{collection_name}/columns/{field_name}`

#### Parameters

*   `field_name` (string) - Required - The name of the field to drop.

### Alter a Column

Renames a column or updates its schema (e.g., changes data type if compatible).

*   **Method**: PUT (Conceptual - API call)
*   **Endpoint**: `/collections/{collection_name}/columns`

#### Parameters

*   `old_name` (string) - Optional - The current name of the field to alter.
*   `new_name` (string) - Optional - The new name for the field.
*   `field_schema` (object) - Optional - The updated schema for the field.
    *   `name` (string) - Required if `new_name` is not provided - The name of the field.
    *   `data_type` (enum) - Required if changing data type - The new data type for the field.

### Request Example (Rename)

```json
{
  "old_name": "publish_year",
  "new_name": "release_year"
}
```

### Request Example (Change Type)

```json
{
  "field_schema": {
    "name": "rating",
    "data_type": "FLOAT"
  }
}
```

### View the Current Schema

Retrieves the current schema definition for a collection.

*   **Method**: GET (Conceptual - API call)
*   **Endpoint**: `/collections/{collection_name}/schema`

#### Response Example

```json
{
  "schema": {
    "fields": [
      { "name": "id", "data_type": "UUID" },
      { "name": "title", "data_type": "STRING" },
      { "name": "release_year", "data_type": "INT32" },
      { "name": "rating", "data_type": "INT32" }
    ]
  }
}
```
```

--------------------------------

### Configure Zvec Global Settings (Python)

Source: https://zvec.org/en/docs/config

Initializes Zvec with custom global settings, including log type, log level, and the number of query threads. This function should be called once at application startup before any database operations.

```python
import zvec

zvec.init(
    log_type=zvec.LogType.CONSOLE,
    log_level=zvec.LogLevel.WARN,
    query_threads=4,
)
```

--------------------------------

### Alter a Scalar Field in a Collection

Source: https://zvec.org/en/docs/collections/schema-evolution

Renames a scalar field or updates its schema in a Zvec collection. Can be used to change the field name or its data type, provided the change is compatible.

```python
# Rename
collection.alter_column(old_name="publish_year", new_name="release_year")   

# Change type (if compatible)
updated = zvec.FieldSchema(name="rating", data_type=zvec.DataType.FLOAT)
collection.alter_column(field_schema=updated)
```

```javascript
// Rename
await collection.alter_column({ old_name: "publish_year", new_name: "release_year" });

// Change type (if compatible)
const updated = new zvec.FieldSchema({ name: "rating", dataType: zvec.DataType.FLOAT });
await collection.alter_column({ field_schema: updated });
```

--------------------------------

### Configure Network for Chinese Users Downloading Models (Python)

Source: https://zvec.org/en/docs/embedding

Provides two methods for configuring network settings to reliably download models for users in mainland China. It shows how to use ModelScope as a source or set a Hugging Face mirror endpoint via an environment variable.

```python
import os
from zvec.extension import DefaultLocalDenseEmbedding

# Option 1: Use ModelScope
embedding = DefaultLocalDenseEmbedding(model_source="modelscope")

# Option 2: Use Hugging Face mirror in Python
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
embedding = DefaultLocalDenseEmbedding(model_source="huggingface")

```

--------------------------------

### Create an Index on a Field

Source: https://zvec.org/en/docs/collections/schema-evolution

Creates or replaces an index on a specified field in a Zvec collection to accelerate search performance. Supports vector fields (HNSW, IVF, FLAT) and scalar fields (Inverted Index).

```python
import zvec

# Replace the existing HNSW index with a FLAT index
collection.create_index(
    field_name="dense_embedding",
    index_param=zvec.FlatIndexParam(metric_type=zvec.MetricType.COSINE),
)

# Create an inverted index
collection.create_index(
    field_name="publish_year",
    index_param=zvec.InvertIndexParam(),
)
```

```javascript
const zvec = require('zvec');

// Replace the existing HNSW index with a FLAT index
await collection.create_index({
    field_name: "dense_embedding",
    index_param: new zvec.FlatIndexParam({ metric_type: zvec.MetricType.COSINE }),
});

// Create an inverted index
await collection.create_index({
    field_name: "publish_year",
    index_param: new zvec.InvertIndexParam(),
});
```

--------------------------------

### Implement Custom Qwen Embeddings with QwenFunctionBase (Python)

Source: https://zvec.org/en/docs/embedding

Demonstrates how to create a custom dense embedding function using the QwenFunctionBase class for the Qwen Dashscope API. It initializes the base class, defines embedding logic, and handles input validation.

```python
from zvec.extension.qwen_function import QwenFunctionBase
from zvec.extension import DenseEmbeddingFunction
from zvec.common.constants import TEXT, DenseVectorType
from typing import Optional


class CustomQwenEmbedding(QwenFunctionBase, DenseEmbeddingFunction[TEXT]):
    """Custom Qwen embedding implementation"""
    
    def __init__(
        self,
        api_key: str,
        model: str = "text-embedding-v3",
        **kwargs
    ):
        # Initialize base class with API key
        QwenFunctionBase.__init__(self, api_key=api_key)
        
        self._model = model
        self._extra_params = kwargs
        self._dimension = None  # Will be set after first call
    
    @property
    def dimension(self) -> int:
        if self._dimension is None:
            # Get dimension from first embedding call
            test_result = self.embed("test")
            self._dimension = len(test_result)
        return self._dimension
    
    @property
    def extra_params(self) -> dict:
        return self._extra_params
    
    def embed(self, input: str) -> DenseVectorType:
        if not isinstance(input, str):
            raise TypeError(f"Expected str, got {type(input).__name__}")
        
        input = input.strip()
        if not input:
            raise ValueError("Input cannot be empty")
        
        # Use the base class's embed_text method
        result = self._embed_text(
            text=input,
            model=self._model
        )
        
        return result
    
    def __call__(self, input: str) -> DenseVectorType:
        return self.embed(input)


# Use custom Qwen embedding
custom_qwen_emb = CustomQwenEmbedding(
    api_key="your-dashscope-api-key",
    model="text-embedding-v3"
)
vector = custom_qwen_emb.embed("Custom Qwen embedding")

```

--------------------------------

### Dense Vector Example

Source: https://zvec.org/en/docs/concepts/vector-embedding

Illustrates a dense vector, which is a fixed-length list of numbers generated by embedding models to capture semantic meaning. These vectors are often high-dimensional and generated by deep learning models.

```python
# Example: 384-dimensional dense vector from a neural network model
dense_vector = [ 0.012, -0.034, 0.005, 0.041, -0.022, ..., 0.018 ]  # Length = 384
```

--------------------------------

### Set CMAKE_GENERATOR to Unix Makefiles (Bash)

Source: https://zvec.org/en/docs/build

Configures scikit-build to use Unix Makefiles as the build system generator instead of the default Ninja. This can be useful for compatibility or specific build environments.

```bash
export CMAKE_GENERATOR="Unix Makefiles"
pip install .
```

--------------------------------

### Sparse Vector Example

Source: https://zvec.org/en/docs/concepts/vector-embedding

Demonstrates a sparse vector, represented as a dictionary mapping terms to their relevance weights. Sparse vectors are high-dimensional where only a few dimensions are non-zero, often used for text data with term importance.

```python
# Example: Sparse vector over a vocabulary of 50,000 terms, stored as {term: weight}
sparse_vector = {
    "puppy": 2.31,
    "dog": 1.85,
    "pet": 1.12,
    "animal": 0.76
}
# The remaining ~49,996 dimensions are implicitly zero.
```

--------------------------------

### QwenSparseEmbedding

Source: https://zvec.org/en/docs/embedding

Provides sparse embedding functionality using the Qwen model via the Dashscope API. Requires a Dashscope API key.

```APIDOC
## QwenSparseEmbedding

### Description
Provides sparse embedding functionality using the Qwen model via the Dashscope API. Requires a Dashscope API key. Visit the Dashscope Console to obtain your API key.

### Method
N/A (Class Initialization and Method Calls)

### Endpoint
N/A

### Parameters
#### Initialization Parameters
- **api_key** (str) - Required - Your Dashscope API key.
- **dimension** (int) - Required - The required embedding vector dimension for the Dashscope API (e.g., 256).

### Request Example
```python
from zvec.extension import QwenSparseEmbedding

embedding_func = QwenSparseEmbedding(
    api_key="your-dashscope-api-key",
    dimension=256,  # dashscope api required input dimension
)
sPARSE_vec = embedding_func.embed("sparse vector")
```

### Response
#### Success Response (N/A - Method Return)
- **embed() return value** (list[float]) - The generated sparse embedding vector.
```

--------------------------------

### Perform Multi-Vector Search with Re-ranking in Python

Source: https://zvec.org/en/docs/data-operations/query

Combines multiple vector embeddings for search and applies re-ranking. This allows for more nuanced search results by weighting different embedding types. It requires specifying multiple VectorQuery objects and a WeightedReRanker configuration.

```python
import zvec

result = collection.query(  
    topk=10,
    vectors=[
        zvec.VectorQuery(field_name="dense_embedding", vector=[0.1] * 768),
        zvec.VectorQuery(field_name="sparse_embedding", vector={1: 0.1, 37: 0.43}),
    ],
    reranker=zvec.WeightedReRanker(  
        topn=3,
        metric=zvec.MetricType.IP,
        weights={
            "dense_embedding": 1.2,
            "sparse_embedding": 1.0,
        },
    ),
)
print(result)
```

--------------------------------

### Add a Scalar Field to a Collection

Source: https://zvec.org/en/docs/collections/schema-evolution

Adds a new scalar field to an existing Zvec collection. Requires a FieldSchema object defining the field's name and data type, and an expression for default values in existing documents. Currently supports numerical scalar fields.

```python
import zvec

new_field = zvec.FieldSchema(name="rating", data_type=zvec.DataType.INT32)
collection.add_column(field_schema=new_field, expression="5")
```

```javascript
const newField = new zvec.FieldSchema({ name: "rating", dataType: zvec.DataType.INT32 });
await collection.add_column({ field_schema: newField, expression: "5" });
```
Tokens:
Raw