VectorDBBench

https://github.com/zilliztech/vectordbbench
Admin
VectorDBBench is a comprehensive benchmarking tool that enables performance and cost-effectiveness...
Tokens:40,034
Snippets:432
Trust Score:8.4
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Connection and Setup Options

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Configuration flags for managing database connections and benchmark execution stages.

```text
--config-file PATH              Read configuration from yaml file
--drop-old / --skip-drop-old    Drop old or skip [default: drop-old]
--load / --skip-load            Load or skip [default: load]
--search-serial / --skip-search-serial
                                Search serial or skip [default: search-serial]
--search-concurrent / --skip-search-concurrent
                                Search concurrent or skip [default: search-concurrent]
```

--------------------------------

### Install VectorDB Bench

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/00-index.md

Install the core package and optional database-specific client dependencies.

```bash
# Basic installation (includes Milvus client)
pip install vectordb-bench

# Add specific database clients
pip install 'vectordb-bench[qdrant]'
pip install 'vectordb-bench[elastic]'
pip install 'vectordb-bench[pgvector]'
pip install 'vectordb-bench[opensearch]'
```

--------------------------------

### Run the benchmark server

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Start the benchmark server using the module flag or the initialization command.

```shell
python -m vectordb_bench
```

```shell
init_bench
```

--------------------------------

### Install database client dependencies

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Commands to install the base package and specific database extras via pip.

```bash
# Milvus (default)
pip install vectordb-bench

# Add specific database
pip install 'vectordb-bench[qdrant]'
pip install 'vectordb-bench[elastic]'
pip install 'vectordb-bench[pgvector]'

# Add opensearch (required for AWS OpenSearch and AOSS)
pip install 'vectordb-bench[opensearch]'

# Full installation
pip install 'vectordb-bench[test,pinecone,qdrant,elastic,pgvector,opensearch]'
```

--------------------------------

### Install VectorDBBench

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Install the base package or specific database client dependencies.

```shell
pip install vectordb-bench
```

```shell
pip install 'vectordb-bench[pinecone]'
```

--------------------------------

### Initialize a Milvus client

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

Example of instantiating a specific vector database client using the VectorDB interface.

```python
from vectordb_bench.backend.clients import DB

# Initialize Milvus client
milvus_cls = DB.Milvus.init_cls
client = milvus_cls(
    dim=768,
    db_config={"uri": "http://localhost:19530"},
    db_case_config=HNSWConfig(m=16, ef_construction=128),
    collection_name="test_collection",
    drop_old=True
)
```

--------------------------------

### Install Hologres Dependencies

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Install the necessary packages to run benchmarks on Hologres.

```shell
pip install 'vectordb-bench[hologres]' 'psycopg[binary]' pgvector
```

--------------------------------

### Define CLI for Zilliz Client

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Example implementation of a cli.py file for the Zilliz client, utilizing Pydantic and Click for parameter handling.

```python
from typing import Annotated, Unpack

import click
import os
from pydantic import SecretStr

from vectordb_bench.cli.cli import (
    CommonTypedDict,
    cli,
    click_parameter_decorators_from_typed_dict,
    run,
)
from vectordb_bench.backend.clients import DB


class ZillizTypedDict(CommonTypedDict):
    uri: Annotated[
        str, click.option("--uri", type=str, help="uri connection string", required=True)
    ]
    user_name: Annotated[
        str, click.option("--user-name", type=str, help="Db username", required=True)
    ]
    password: Annotated[
        str,
        click.option("--password",
                     type=str,
                     help="Zilliz password",
                     default=lambda: os.environ.get("ZILLIZ_PASSWORD", ""),
                     show_default="$ZILLIZ_PASSWORD",
                     ),
    ]
    level: Annotated[
        str,
        click.option("--level", type=str, help="Zilliz index level", required=False),
    ]


@cli.command()
@click_parameter_decorators_from_typed_dict(ZillizTypedDict)
def ZillizAutoIndex(**parameters: Unpack[ZillizTypedDict]):
    from .config import ZillizCloudConfig, AutoIndexConfig

    run(
        db=DB.ZillizCloud,
        db_config=ZillizCloudConfig(
            db_label=parameters["db_label"],
            uri=SecretStr(parameters["uri"]),
            user=parameters["user_name"],
            password=SecretStr(parameters["password"]),
        ),
        db_case_config=AutoIndexConfig(
            params={parameters["level"]},
        ),
        **parameters,
    )
```

--------------------------------

### Install project requirements

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Install the package in editable mode with test or pinecone dependencies.

```shell
pip install -e '.[test]'

pip install -e '.[pinecone]'
```

--------------------------------

### Define HNSWConfig for index tuning

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Example of a DBCaseConfig subclass for HNSW index parameters, including methods to retrieve index and search parameters.

```python
class HNSWConfig(DBCaseConfig):
    ""HNSW index parameters"""
    m: int = 16                         # Max connections
    ef_construction: int = 128          # Build parameter
    ef_search: int = 100                # Search parameter
    
    def index_param(self) -> dict:
        return {"m": self.m, "ef_construction": self.ef_construction}
    
    def search_param(self) -> dict:
        return {"ef_search": self.ef_search}
```

--------------------------------

### View CLI Help Documentation

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Displays the available command-line options and configuration parameters for Elastic Cloud benchmarks.

```text
$ vectordbbench elasticcloudhnsw --help
Usage: vectordbbench elasticcloudhnsw [OPTIONS]

Options:
  # Connection
  --cloud-id TEXT                 Elastic Cloud ID  [required]
  --password TEXT                 Elastic Cloud password  [required]

  # HNSW Index Parameters
  --m INTEGER                     HNSW M parameter  [default: 16]
  --ef-construction INTEGER       HNSW efConstruction parameter  [default: 100]
  --num-candidates INTEGER        Number of candidates for search  [default: 100]
  --element-type [float|byte]     Element type for vectors (float: 4 bytes, byte: 1 byte)  [default: float]

  # Index Configuration
  --number-of-shards INTEGER      Number of shards  [default: 1]
  --number-of-replicas INTEGER    Number of replicas  [default: 0]
  --refresh-interval TEXT         Index refresh interval  [default: 30s]
  --merge-max-thread-count INTEGER
                                  Maximum thread count for merge  [default: 8]
  --use-force-merge BOOLEAN       Whether to use force merge  [default: True]
  --use-routing BOOLEAN           Whether to use routing  [default: False]
  --use-rescore BOOLEAN           Whether to use rescore  [default: False]
  --oversample-ratio FLOAT        Oversample ratio for rescore  [default: 2.0]

  # Common Options
  --case-type [CapacityDim128|CapacityDim960|Performance768D100M|...]
                                  Case type
  --db-label TEXT                 Db label, default: date in ISO format
  --k INTEGER                     K value for number of nearest neighbors to search  [default: 100]
  --num-concurrency TEXT          Comma-separated list of concurrency values  [default: 1,5,10,20,30,40,60,80]
  --help                          Show this message and exit.
```

--------------------------------

### Use init context manager

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

Demonstrates managing database connections and performing operations within the context manager.

```python
with client.init():
    embeddings = [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]
    client.insert_embeddings(embeddings, metadata=[1, 2])
    results = client.search_embedding([0.1, 0.2, 0.3], k=10)
```

--------------------------------

### Initialize and Run Benchmarks

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Commands to initialize the environment and execute benchmarks via the CLI.

```shell
init_bench
```

```shell
vectordbbench [OPTIONS] COMMAND [ARGS]...
```

--------------------------------

### TestResult JSON Schema

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/07-task-runner.md

Example JSON structure for a task result file.

```json
{
  "task_label": "my-benchmark-run",
  "timestamp": "2026-08-21T10:30:00",
  "case_results": [
    { "case_type": "...", "db_label": "..." },
    ...
  ]
}
```

--------------------------------

### Main Entry Point Usage

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

The base command structure for invoking the benchmark suite.

```bash
vectordbbench [OPTIONS] COMMAND [ARGS]...
```

--------------------------------

### CaseResult JSON Schema

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/07-task-runner.md

Example JSON structure for an individual case result file.

```json
{
  "case_type": "Performance768D1M",
  "case_label": "Performance Test (768 Dim 1M)",
  "db_label": "pgvector-hnsw",
  "metric": {
    "qps": 1234.5,
    "serial_latency_p99": 0.85,
    "serial_latency_p95": 0.72,
    "recall": 0.95,
    "load_duration": 3600.0,
    "optimize_duration": 300.0,
    "conc_num_list": [1, 10, 20],
    "conc_qps_list": [1234.5, 8900.0, 12345.0],
    "conc_latency_p99_list": [0.85, 1.2, 1.5],
    "conc_latency_p95_list": [0.72, 0.95, 1.2]
  },
  "result_label": "succeed"
}
```

--------------------------------

### Initialize BenchMarkRunner

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/02-benchmark-interface.md

Configure the runner with data source settings and collection management preferences.

```python
from vectordb_bench.interface import BenchMarkRunner

runner = BenchMarkRunner()
runner.set_drop_old(True)
runner.set_download_address(use_aliyun=False, use_ir_datasets=False)
```

--------------------------------

### BenchMarkRunner.__init__

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/02-benchmark-interface.md

Initializes a new instance of the BenchMarkRunner with default settings.

```APIDOC
## BenchMarkRunner()

### Description
Initializes the runner with default settings (S3 data source, drop_old=True).
```

--------------------------------

### Run Benchmark with Context Note

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Execute a benchmark while providing a JSON file for deployment and resource context.

```shell
vectordbbench zillizautoindex \
  --note-file ./run-context.json \
  <other options>
```

--------------------------------

### Qdrant Cloud CLI Usage

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Commands and options for benchmarking Qdrant Cloud vector database.

```bash
vectordbbench qdrantcloud [OPTIONS]
```

```text
--url TEXT               Qdrant API URL [required]
--api-key TEXT           API key [required]
```

```text
--m INTEGER              HNSW M parameter
--ef-construction INTEGER
                         HNSW ef_construction
--ef-search INTEGER      HNSW ef_search
```

--------------------------------

### ADBPG Nova CLI options reference

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

List of available command-line flags for configuring ADBPG Nova benchmarks.

```text
  --user-name TEXT                Db username  [required]
  --password TEXT                 Postgres database password  [$POSTGRES_PASSWORD]
  --host TEXT                     Db host  [required]
  --port INTEGER                  Postgres database port  [default: 5432]
  --db-name TEXT                  Db name  [required]
  --algorithm TEXT                algorithm  [default: novamr]
  --hnsw-m INTEGER                hnsw_m  [default: 16]
  --ef-construction INTEGER       ef_construction  [default: 200]
  --ef-search INTEGER             ef_search  [default: 100]
  --max-scan-points INTEGER       max scan points  [default: 2000]
  --quantize-rescore-amp FLOAT    fastann.quantize_rescore_amp  [default: 1.0]
  --nova-adaptive-gamma FLOAT     fastann.nova_adaptive_gamma  [default: 0.0]
  --auto-reduction/--no-auto-reduction  Index WITH auto_reduction=on  [default: False]
```

--------------------------------

### Verify Database Connection

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/06-implementation-guide.md

Use the Python client to initialize a connection and verify the client instance is active.

```python
from vectordb_bench.backend.clients.new_database.config import NewDatabaseConfig

config = NewDatabaseConfig(
    host="localhost",
    port=9000,
    user_name="user",
    password=SecretStr("pass")
)

client = NewDatabaseClient(
    dim=768,
    db_config=config.to_dict(),
    db_case_config=None,
    collection_name="test",
    drop_old=True
)

with client.init():
    print(f"Connected: {client.client is not None}")
```

--------------------------------

### Assembler.assemble_all

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/07-task-runner.md

Builds a fully configured TaskRunner instance from a list of task configurations and a dataset source.

```APIDOC
## assemble_all

### Description
Validates all configurations and assembles them into a TaskRunner ready for execution.

### Signature
`assemble_all(run_id: str, task_label: str, task_configs: list[TaskConfig], dataset_source: DatasetSource) -> TaskRunner`

### Parameters
- **run_id** (str) - Required - Unique run identifier
- **task_label** (str) - Required - Human-readable label
- **task_configs** (list[TaskConfig]) - Required - Benchmark configurations
- **dataset_source** (DatasetSource) - Required - Dataset source (S3, AliyunOSS, IR_DATASETS)

### Returns
- **TaskRunner** - Fully configured TaskRunner ready to execute.

### Raises
- **FilterNotSupportedError** - If database doesn't support required filter type
- **ModuleNotFoundError** - If database client module not installed
```

--------------------------------

### VolcMySQL CLI Options

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

List of specific command-line parameters available for the VolcMySQL benchmark tool.

```text
  --username TEXT                  Username  [required]
  --password TEXT                  Password  [required]
  --host TEXT                      Db host  [default: 127.0.0.1]
  --port INTEGER                   DB Port  [default: 3306]
  --m INTEGER                      M parameter in HNSW vector indexing
  --ef-search INTEGER              Session variable loose_hnsw_ef_search
  --ef-construction INTEGER        HNSW ef_construction
  --quant-algorithm [NONE|SQ|PQ]   Quantization algorithm
  --quant-type [16_bit|8_bit|4_bit|binary]
                                   Quantization type
```

--------------------------------

### Register New Client in Initialization

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Update the DB enum and initialization properties in clients/__init__.py to include the new client.

```python
#clients/__init__.py

# Add NewClient to the DB enum
class DB(Enum):
    ...
    DB.NewClient = "NewClient"

    @property
    def init_cls(self) -> Type[VectorDB]:
        ...
        if self == DB.NewClient:
            from .new_client.new_client import NewClient
            return NewClient
        ...

    @property
    def config_cls(self) -> Type[DBConfig]:
        ...
        if self == DB.NewClient:
            from .new_client.config import NewClientConfig
            return NewClientConfig
        ...

    def case_config_cls(self, ...)
        if self == DB.NewClient:
            from .new_client.config import NewClientCaseConfig
            return NewClientCaseConfig
```

--------------------------------

### Configure Aliyun OSS benchmark execution

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Sets environment variables for Aliyun OSS data source and executes the pgvectorhnsw benchmark.

```bash
export DATASET_SOURCE=AliyunOSS
export DATASET_LOCAL_DIR=/data/vectordb_bench/dataset
export NUM_PER_BATCH=1000
export LOAD_CONCURRENCY=8

vectordbbench pgvectorhnsw \
  --user-name postgres \
  --password password \
  --host localhost \
  --db-name vectordb \
  --case-type Performance768D1M
```

--------------------------------

### Configure Full-Text Search benchmark execution

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Sets environment variables for IR_DATASETS and executes the milvushnsw benchmark.

```bash
export DATASET_SOURCE=IR_DATASETS
export DATASET_LOCAL_DIR=/tmp/vectordb_bench/fts_dataset

vectordbbench milvushnsw \
  --uri http://localhost:19530 \
  --case-type FTSBm25Performance
```

--------------------------------

### Implement CLI Support

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/06-implementation-guide.md

Define CLI parameters using TypedDict and register the command to enable benchmarking via the command line.

```python
from typing import Annotated, Unpack
import click
import os
from pydantic import SecretStr

from vectordb_bench.cli.cli import (
    CommonTypedDict,
    cli,
    click_parameter_decorators_from_typed_dict,
    run,
)
from vectordb_bench.backend.clients import DB

class NewDatabaseTypedDict(CommonTypedDict):
    """CLI parameters for NewDatabase"""
    
    host: Annotated[
        str,
        click.option("--host", type=str, help="Database host", required=True),
    ]
    port: Annotated[
        int,
        click.option("--port", type=int, help="Database port", default=9000),
    ]
    user_name: Annotated[
        str,
        click.option("--user-name", type=str, help="Username", default=""),
    ]
    password: Annotated[
        str,
        click.option(
            "--password",
            type=str,
            help="Password",
            default=lambda: os.environ.get("NEWDB_PASSWORD", ""),
            show_default="$NEWDB_PASSWORD",
        ),
    ]

class NewDatabaseHNSWTypedDict(NewDatabaseTypedDict):
    """HNSW-specific parameters"""
    
    m: Annotated[
        int,
        click.option("--m", type=int, help="HNSW M parameter", default=16),
    ]
    ef_construction: Annotated[
        int,
        click.option("--ef-construction", type=int, help="ef_construction", default=128),
    ]
    ef_search: Annotated[
        int,
        click.option("--ef-search", type=int, help="ef_search", default=100),
    ]

@cli.command()
@click_parameter_decorators_from_typed_dict(NewDatabaseHNSWTypedDict)
def newdatabasehnsw(**parameters: Unpack[NewDatabaseHNSWTypedDict]):
    """NewDatabase with HNSW index"""
    from .config import NewDatabaseConfig, HNSWConfig
    
    run(
        db=DB.NewDatabase,
        db_config=NewDatabaseConfig(
            db_label=parameters["db_label"],
            host=parameters["host"],
            port=parameters["port"],
            user_name=parameters["user_name"],
            password=SecretStr(parameters["password"]),
        ),
        db_case_config=HNSWConfig(
            m=parameters["m"],
            ef_construction=parameters["ef_construction"],
            ef_search=parameters["ef_search"],
        ),
        **parameters,
    )
```

--------------------------------

### VectorDB.__init__

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

Initializes the vector database client, handles collection creation, and optionally drops existing collections.

```APIDOC
## Constructor: __init__

### Description
Initializes the vector database client. If drop_old is True, it drops existing collections with the same name and creates a new one.

### Parameters
- **dim** (int) - Required - Dimension of vectors in the dataset
- **db_config** (dict) - Required - Configuration for database connection (server, credentials, etc.)
- **db_case_config** (DBCaseConfig | None) - Optional - Case-specific configuration like index parameters
- **collection_name** (str) - Required - Name of collection/table to create or use
- **drop_old** (bool) - Optional - Whether to drop existing collection before initialization
- **kwargs** (Any) - Optional - Database-specific parameters
```

--------------------------------

### Integration Test via CLI

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/06-implementation-guide.md

Demonstrates running a benchmark against a local database container. Use the --dry-run flag to validate configuration before execution.

```bash
# Start your database
docker run -d -p 9000:9000 new-database:latest

# Run benchmark
vectordbbench newdatabasehnsw \
  --host localhost \
  --port 9000 \
  --user-name user \
  --password pass \
  --case-type Performance1536D50K \
  --db-label integration-test \
  --dry-run  # First, verify configuration

# Remove --dry-run to execute actual benchmark
vectordbbench newdatabasehnsw \
  --host localhost \
  --port 9000 \
  --user-name user \
  --password pass \
  --case-type Performance1536D50K \
  --db-label integration-test
```

--------------------------------

### Register the New Client

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/06-implementation-guide.md

Update the backend registry to include the new database client and its associated configuration classes.

```python
# Add to DB enum
class DB(Enum):
    ...
    NewDatabase = "NewDatabase"

# Add to init_cls property
@property
def init_cls(self) -> type[VectorDB]:
    ...
    if self == DB.NewDatabase:
        from .new_database.new_database import NewDatabaseClient
        return NewDatabaseClient

# Add to config_cls property
@property
def config_cls(self) -> type[DBConfig]:
    ...
    if self == DB.NewDatabase:
        from .new_database.config import NewDatabaseConfig
        return NewDatabaseConfig

# Add to case_config_cls method
def case_config_cls(self, case_type: CaseType) -> type[DBCaseConfig]:
    ...
    if self == DB.NewDatabase:
        if case_type in [CaseType.Performance768D1M, ...]:
            from .new_database.config import HNSWConfig
            return HNSWConfig
        return EmptyDBCaseConfig()
```

--------------------------------

### Run Benchmark with Config File

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Execute a benchmark using a specific YAML configuration file.

```bash
vectordbbench pgvectorhnsw --config-file my_config.yaml
```

--------------------------------

### CLI Help and Command Information

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Display help information for the main CLI or specific commands.

```text
$ vectordbbench --help
Usage: vectordbbench [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  pgvectorhnsw
  pgvectorivfflat
  vectorchordrq
  volcmysqlhnsw
  test
  weaviate
```

```text
$ vectordbbench pgvectorhnsw --help
Usage: vectordbbench pgvectorhnsw [OPTIONS]

Options:
  --config-file PATH              Read configuration from yaml file
  --drop-old / --skip-drop-old    Drop old or skip  [default: drop-old]
  --load / --skip-load            Load or skip  [default: load]
  --search-serial / --skip-search-serial
                                  Search serial or skip  [default: search-
                                  serial]
  --search-concurrent / --skip-search-concurrent
                                  Search concurrent or skip  [default: search-
                                  concurrent]
  --case-type [CapacityDim128|CapacityDim960|Performance768D100M|Performance768D10M|Performance768D1M|Performance768D10M1P|Performance768D1M1P|Performance768D10M99P|Performance768D1M99P|Performance1536D500K|Performance1536D5M|Performance1536D500K1P|Performance1536D5M1P|Performance1536D500K99P|Performance1536D5M99P|Performance1536D50K]
                                  Case type
  --db-label TEXT                 Db label, default: date in ISO format
```

--------------------------------

### Configure development and testing environment

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Sets debug logging and concurrency parameters for local testing with pgvectorhnsw.

```bash
export LOG_LEVEL=DEBUG
export NUM_PER_BATCH=10
export NUM_CONCURRENCY=1,2,4
export CONCURRENCY_DURATION=10
export RESULTS_LOCAL_DIR=./test_results

vectordbbench pgvectorhnsw \
  --user-name postgres \
  --password password \
  --host localhost \
  --db-name vectordb \
  --case-type Performance1536D50K \
  --db-label dev-test
```

--------------------------------

### Dataset and Case Configuration Options

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Flags for defining benchmark cases, labeling instances, and managing run metadata.

```text
--case-type TEXT                Benchmark case type (CapacityDim128, Performance768D1M, etc.)
--db-label TEXT                 Db label to distinguish instances [default: ISO timestamp]
--note TEXT                     Run context stored with each result [default: ""]
--note-file FILE                Read run context from UTF-8 text file
--dry-run                       Print configuration and exit without running
--custom-case-name TEXT         Custom case name (e.g., PerformanceCase1536D50K)
--custom-case-description TEXT  Custom case description
--custom-case-load-timeout INTEGER
                                Custom case load timeout [default: 36000]
--custom-case-optimize-timeout INTEGER
                                Custom case optimize timeout [default: 36000]
```

--------------------------------

### Define Batch Benchmark Configuration

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Configure multiple test scenarios in a single file by providing a list of configurations for each database type.

```yaml
pgvectorhnsw:
  - db_label: small-load
    user_name: postgres
    password: password
    host: localhost
    db_name: vectordb
    case_type: CapacityDim128
    m: 16
    
  - db_label: medium-perf
    user_name: postgres
    password: password
    host: localhost
    db_name: vectordb
    case_type: Performance768D1M
    m: 16
    ef_construction: 128
    ef_search: 100
    
  - db_label: large-perf
    user_name: postgres
    password: password
    host: localhost
    db_name: vectordb
    case_type: Performance768D10M
    m: 24
    ef_construction: 200
    ef_search: 150

elasticcloudhnsw:
  - db_label: elastic-1m
    cloud_id: my-deployment-id
    password: api-key
    case_type: Performance768D1M
    m: 16
    ef_construction: 100
```

--------------------------------

### Run ADBPG Nova benchmarks via CLI

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Execute performance benchmarks for ADBPG Nova using direct command-line arguments or a configuration file.

```shell
vectordbbench adbpgnova --case-type Performance1024D1M --k 10 \
--host <adbpg_host> --port 5432 --db-name postgres \
--user-name <username> --password <password> \
--algorithm novamr --hnsw-m 48 --ef-construction 600 \
--ef-search 130 --max-scan-points 5000 --quantize-rescore-amp 2.0
```

```shell
vectordbbench adbpgnova --config-file adbpg_bioasq1m_novamr.yml
```

--------------------------------

### Run Zvec Benchmark

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Executes the Zvec benchmark with specified performance parameters and quantization settings.

```bash
vectordbbench zvec --path Performance768D10M --db-label 16c64g-v0.1 \
    --case-type Performance768D10M --num-concurrency 12,14,16,18,20 \
    --quantize-type int8 --ef-search 118 --is-using-refiner
```

--------------------------------

### PolarDB CLI options reference

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

List of available command-line flags for configuring PolarDB benchmarks.

```text
  --username TEXT                  Username  [required]
  --password TEXT                  Password
  --host TEXT                      Db host  [default: 127.0.0.1]
  --port INTEGER                   Db Port  [default: 3306]
  --database TEXT                  Database name  [default: vectordbbench]
  --m INTEGER                      M parameter (max_degree) in HNSW
  --ef-construction INTEGER        ef_construction parameter in HNSW
  --ef-search INTEGER              polar_vector_index_hnsw_ef_search session variable
  --insert-workers INTEGER         Number of concurrent threads for data insertion
  --post-load-index / --inline-index
                                   Create index after load or inline at table creation
```

--------------------------------

### Define init method signature

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

Abstract method signature for the database connection context manager.

```python
@abstractmethod
@contextmanager
def init(self) -> None
```

--------------------------------

### Display AWS OpenSearch Help

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Lists available options and configuration parameters for the awsopensearch command.

```text
$ vectordbbench awsopensearch --help
Usage: vectordbbench awsopensearch [OPTIONS]

Options:
  # Sharding and Replication
  --number-of-shards INTEGER      Number of primary shards for the index
  --number-of-replicas INTEGER    Number of replica copies for each primary
                                  shard
  # Indexing Performance
  --index-thread-qty INTEGER      Thread count for native engine indexing
  --index-thread-qty-during-force-merge INTEGER
                                  Thread count during force merge operations
  --number-of-indexing-clients INTEGER
                                  Number of concurrent indexing clients
  # Index Management
  --number-of-segments INTEGER    Target number of segments after merging
  --refresh-interval TEXT         How often to make new data available for
                                  search
  --force-merge-enabled BOOLEAN   Whether to perform force merge operation
  --flush-threshold-size TEXT     Size threshold for flushing the transaction
                                  log
  --engine TEXT                   type of engine to use valid values [faiss, lucene, s3vector]
  # Memory Management
  --cb-threshold TEXT             k-NN Memory circuit breaker threshold

  --ondisk                        Ondisk mode with binary quantization(32x compression)
  --oversample-factor             Controls the degree of oversampling applied to minority classes in imbalanced datasets to improve model performance by balancing class distributions.(default 1.0)

  # Quantization Type
  --quantization-type TEXT        which type of quantization to use valid values [fp32, fp16, bq]
  --help                          Show this message and exit.
```

--------------------------------

### Configure AWS OpenSearch Serverless benchmark

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Configures batching, concurrency, and timeout settings for AWS OpenSearch Serverless benchmarks.

```bash
export NUM_PER_BATCH=50
export LOAD_CONCURRENCY=4
export CLOUD_INSERT_READINESS_TIMEOUT=300
export CLOUD_INSERT_READINESS_POLL_INTERVAL=10

vectordbbench awsopensearch \
  --serverless \
  --aws-region us-east-1 \
  --host <collection-id>.aoss.us-east-1.on.aws \
  --case-type Performance768D1M
```

--------------------------------

### Define Database and Index Configuration

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/06-implementation-guide.md

Create configuration classes inheriting from DBConfig and DBCaseConfig to manage database connection and index parameters.

```python
from pydantic import Field, SecretStr
from vectordb_bench.backend.clients.api import DBConfig, DBCaseConfig, EmptyDBCaseConfig

class NewDatabaseConfig(DBConfig):
    """Connection configuration for NewDatabase"""
    
    host: str = Field(..., description="Database host")
    port: int = Field(default=9000, description="Database port")
    user_name: str = Field(default="", description="Username")
    password: SecretStr = Field(default=SecretStr(""), description="Password")
    
    def to_dict(self) -> dict:
        """Convert config to dictionary for storage"""
        return {
            "host": self.host,
            "port": self.port,
            "user_name": self.user_name,
            "password": self.password.get_secret_value(),
            "db_label": self.db_label,
        }

class HNSWConfig(DBCaseConfig):
    """HNSW index configuration"""
    
    m: int = Field(default=16, description="M parameter")
    ef_construction: int = Field(default=128, description="ef_construction parameter")
    ef_search: int = Field(default=100, description="ef_search parameter")
    
    def index_param(self) -> dict:
        return {
            "m": self.m,
            "ef_construction": self.ef_construction,
        }
    
    def search_param(self) -> dict:
        return {
            "ef_search": self.ef_search,
        }
```

--------------------------------

### Run Benchmarks via CLI

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/00-index.md

Execute performance benchmarks for specific databases using the command line interface.

```bash
# Benchmark PostgreSQL with HNSW index
vectordbbench pgvectorhnsw \
  --user-name postgres \
  --password mypassword \
  --host localhost \
  --db-name vectordb \
  --case-type Performance768D1M \
  --m 16 \
  --ef-construction 128 \
  --ef-search 100

# Benchmark Milvus
vectordbbench milvushnsw \
  --uri http://localhost:19530 \
  --case-type Performance768D1M
```

--------------------------------

### VectorDB.init

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

A context manager that manages database connection lifecycles, essential for multiprocessing.

```APIDOC
## Method: init

### Description
Context manager that creates and destroys database connections. Essential for multiprocessing scenarios where connection pooling matters.

### Returns
- Context manager that yields a usable database connection.
```

--------------------------------

### Elastic Cloud CLI Usage

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Commands and options for benchmarking Elastic Cloud vector search.

```bash
vectordbbench elasticcloudhnsw [OPTIONS]
```

```text
--cloud-id TEXT          Elastic Cloud ID [required]
--password TEXT          Elastic Cloud password [required]
```

```text
--m INTEGER              HNSW M parameter [default: 16]
--ef-construction INTEGER
                         HNSW efConstruction [default: 100]
--num-candidates INTEGER Number of search candidates [default: 100]
--element-type [float|byte]
                         Vector element type [default: float]
```

```text
--number-of-shards INTEGER
                         Number of shards [default: 1]
--number-of-replicas INTEGER
                         Number of replicas [default: 0]
--refresh-interval TEXT  Index refresh interval [default: 30s]
--merge-max-thread-count INTEGER
                         Max merge threads [default: 8]
--use-force-merge BOOLEAN
                         Use force merge [default: True]
--use-routing BOOLEAN    Use routing [default: False]
--use-rescore BOOLEAN    Use rescore [default: False]
--oversample-ratio FLOAT Rescore oversample ratio [default: 2.0]
```

```bash
vectordbbench elasticcloudhnsw \
  --cloud-id <your-cloud-id> \
  --password mypassword \
  --case-type Performance768D1M \
  --m 16 \
  --ef-construction 100 \
  --db-label elastic-test
```

--------------------------------

### Define Benchmark Configuration in YAML

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

YAML structure for defining database connection parameters and benchmark settings.

```yaml
pgvectorhnsw:
  db_label: test-run
  user_name: postgres
  password: mypassword
  host: localhost
  db_name: vectordb
  case_type: Performance1536D50K
  m: 16
  ef_construction: 128
  ef_search: 100

milvushnsw:
  uri: http://localhost:19530
  case_type: Performance768D1M
  m: 16
  ef_construction: 128
```

--------------------------------

### EmptyDBCaseConfig Implementation

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

Default configuration for databases that do not require specific tuning parameters.

```python
class EmptyDBCaseConfig(BaseModel, DBCaseConfig):
    null: str | None = None

    def index_param(self) -> dict:
        return {}

    def search_param(self) -> dict:
        return {}
```

--------------------------------

### Run CloudInsertCase on Zilliz Cloud

Source: https://github.com/zilliztech/vectordbbench/blob/main/docs/release/2026-05-cloud-leaderboard.md

Measures write readiness and indexing latency for the LAION 100M dataset using a 10k batch size.

```bash
vectordbbench zillizautoindex \
  --case-type CloudInsertCase \
  --uri "$ZILLIZ_URI" \
  --token "$ZILLIZ_TOKEN" \
  --collection-name cloud_insert_laion100m_bs10k \
  --cloud-insert-batch-size 10000 \
  --load-concurrency 16 \
  --skip-search-serial \
  --skip-search-concurrent \
  --task-label cloud-insert-zilliz-12cu
```

--------------------------------

### Prepare dataset directory for dev container

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Create the local directory required for mounting in a development container environment.

```shell
# Mount local ~/vectordb_bench/dataset to contain's /tmp/vectordb_bench/dataset.
# If you are not comfortable with the path name, feel free to change it in devcontainer.json
mkdir -p ~/vectordb_bench/dataset
```

--------------------------------

### Run Batch Benchmarks

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Execute multiple benchmarks defined in a batch configuration file.

```bash
vectordbbench batchcli --batch-config-file batch_config.yaml
```

--------------------------------

### Run performance test on OpenSearch Serverless

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Execute a performance benchmark against an AOSS collection using the command line interface.

```shell
NUM_PER_BATCH=100 vectordbbench awsopensearch --db-label aoss \
  --serverless --aws-region us-east-1 \
  --host <collection-id>.aoss.us-east-1.on.aws --port 443 \
  --case-type Performance768D1M \
  --m 16 --ef-construction 200 --ef-search 40 \
  --number-of-shards 8 --number-of-replicas 0 \
  --engine faiss --metric-type cosine \
  --num-concurrency 80,100,120
```

--------------------------------

### Implement NewClient Class

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Define a new client class by inheriting from the VectorDB abstract base class.

```python
from ..api import VectorDB
class NewClient(VectorDB):
    # Implement the abstract methods defined in the VectorDB class
    # ...
```

--------------------------------

### Configuration File Format

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Required YAML structure for defining benchmark parameters in a configuration file.

```yaml
commandname:
   parameter_name: parameter_value
   parameter_name: parameter_value
```

```yaml
pgvectorhnsw:
  db_label: pgConfigTest
  user_name: vectordbbench
  password: vectordbbench
  db_name:  vectordbbench
  host: localhost
  m: 16
  ef_construction: 128
  ef_search: 128
milvushnsw:
  skip_search_serial: True
  case_type: Performance1536D50K
  uri: http://localhost:19530
  m: 16
  ef_construction: 128
  ef_search: 128
  drop_old: False
  load: False
elasticcloudhnsw:
  db_label: elastic-cloud-hnsw
  cloud_id: <your-cloud-id>
  password: <your-password>
  case_type: Performance768D1M
  m: 16
  ef_construction: 100
  num_candidates: 100
  number_of_shards: 1
  number_of_replicas: 0
  refresh_interval: 30s
  element_type: float
```

--------------------------------

### Run Doris Benchmark

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Executes the Doris benchmark, optionally including custom session variables for performance tuning.

```shell
NUM_PER_BATCH=1000000 vectordbbench doris --http-port=8030 --port=9030 --db-name=vector_test --case-type=Performance768D1M --stream-load-rows-per-batch=500000
```

```shell
NUM_PER_BATCH=1000000 vectordbbench doris --http-port=8030 --port=9030 --db-name=vector_test --case-type=Performance768D1M --stream-load-rows-per-batch=500000 --session-var enable_profile=True
```

--------------------------------

### Configure Logging

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Set environment variables for log level and output file path.

```bash
# Log level (DEBUG, INFO, WARNING, ERROR)
export LOG_LEVEL=INFO

# Log file path
export LOG_FILE=logs/vectordb_bench.log
```

--------------------------------

### Batch Configuration File Format

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

YAML structure for defining multiple test cases in a single batch configuration file.

```yaml
commandname:
  - parameter_name: parameter_value
    another_parameter_name: parameter_value
```

```yaml
pgvectorhnsw:
  - db_label: pgConfigTest
    user_name: vectordbbench
    password: vectordbbench
    db_name:  vectordbbench
    host: localhost
    m: 16
    ef_construction: 128
    ef_search: 128
milvushnsw:
  - skip_search_serial: True
    case_type: Performance1536D50K
    uri: http://localhost:19530
    m: 16
    ef_construction: 128
    ef_search: 128
    drop_old: False
    load: False
elasticcloudhnsw:
  - db_label: elastic-cloud-hnsw-test-1
    cloud_id: <your-cloud-id>
    password: <your-password>
    case_type: Performance768D1M
    m: 16
    ef_construction: 100
    num_candidates: 100
  - db_label: elastic-cloud-label-filter-0.1
    cloud_id: <your-cloud-id>
    password: <your-password>
    case_type: LabelFilterPerformanceCase
    dataset_with_size_type: "Medium OpenAI (1536dim, 500K)"
    label_percentage: 0.001
    m: 16
    ef_construction: 128
    num_candidates: 100
    num_concurrency: "1,5"
```

--------------------------------

### Define Basic Benchmark Configuration

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/05-configuration.md

Use this YAML structure to define standard performance test parameters for pgvector and Milvus instances.

```yaml
pgvectorhnsw:
  db_label: local-test
  user_name: postgres
  password: password
  host: localhost
  db_name: vectordb
  case_type: Performance768D1M
  m: 16
  ef_construction: 128
  ef_search: 100

milvushnsw:
  uri: http://localhost:19530
  case_type: Performance768D1M
  m: 16
  ef_construction: 128
  ef_search: 100
  db_label: milvus-local
```

--------------------------------

### Configure Tuning Parameters

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Set environment variables to control batch sizes and concurrency levels.

```bash
# Number of vectors per insert batch
export NUM_PER_BATCH=100

# Concurrency for data loading (0 = cpu_count)
export LOAD_CONCURRENCY=0

# Concurrency levels to test during search
export NUM_CONCURRENCY=1,5,10,20,30,40,60,80
```

--------------------------------

### Run vectorchordrq benchmark

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Executes the vectorchordrq benchmark with specified database connection and VectorChord index parameters.

```bash
vectordbbench vectorchordrq [OPTIONS]
```

```bash
vectordbbench vectorchordrq \
  --user-name postgres \
  --password mypassword \
  --host localhost \
  --db-name vectordb \
  --case-type Performance1536D50K \
  --lists 1000 \
  --probes 10 \
  --epsilon 1.9 \
  --build-threads 8
```

--------------------------------

### Run PostgreSQL Benchmark via CLI

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/README.md

Executes a performance benchmark for PostgreSQL with HNSW index using command-line arguments.

```bash
# PostgreSQL with HNSW index
vectordbbench pgvectorhnsw \
  --user-name postgres \
  --password mypassword \
  --host localhost \
  --db-name vectordb \
  --case-type Performance768D1M \
  --m 16 \
  --ef-construction 128
```

--------------------------------

### Run Batch Benchmark

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Command to execute tests defined in a batch configuration file.

```shell
vectordbbench batchcli --batch-config-file <your-yaml-configuration-file>
```

--------------------------------

### View OceanBase IVF Help

Source: https://github.com/zilliztech/vectordbbench/blob/main/README.md

Display available command-line options for the oceanbaseivf command.

```text
$ vectordbbench oceanbaseivf --help
Usage: vectordbbench oceanbaseivf [OPTIONS]

Options:
  [...]
  --host TEXT                     OceanBase host
  --user TEXT                     OceanBase username  [required]
  --password TEXT                 OceanBase database password
  --database TEXT                 DataBase name  [required]
  --port INTEGER                  OceanBase port  [required]
  --index-type [IVF_FLAT|IVF_SQ8|IVF_PQ]
                                  Type of index to use. Supported values:
                                  IVF_FLAT, IVF_SQ8, IVF_PQ  [required]
  --nlist INTEGER                 Number of cluster centers  [required]
  --sample_per_nlist INTEGER      The cluster centers are calculated by total
                                  sampling sample_per_nlist * nlist vectors
                                  [required]
  --ivf_nprobes TEXT              How many clustering centers to search during
                                  the query  [required]
  --m INTEGER                     The number of sub-vectors that each data
                                  vector is divided into during IVF-PQ
  --help                          Show this message and exit.                       Show this message and exit.
```

--------------------------------

### Define New Database Client Directory Structure

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/06-implementation-guide.md

The required file structure for a new database client within the vectordb_bench/backend/clients/ directory.

```text
new_database/
├── __init__.py
├── new_database.py
├── config.py
└── cli.py (optional)
```

--------------------------------

### Run benchmarks via CLI

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/README.md

Execute performance tests using the vectordbbench command. Use the dry-run flag to validate configurations before executing full benchmarks.

```bash
# Run with dry-run to verify configuration
vectordbbench pgvectorhnsw \
  --user-name postgres \
  --password pass \
  --host localhost \
  --db-name vectordb \
  --case-type Performance1536D50K \
  --dry-run

# Run small case for testing
vectordbbench pgvectorhnsw \
  --user-name postgres \
  --password pass \
  --host localhost \
  --db-name vectordb \
  --case-type Performance1536D50K
```

--------------------------------

### Prepare Filter Configuration

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

Optional hook for pre-processing filter conditions to minimize redundancy during benchmark execution.

```python
def prepare_filter(self, filters: Filter):
```

--------------------------------

### Configure OpenSearch Serverless

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Set the batch size specifically for OpenSearch Serverless.

```bash
# Number of vectors per bulk request for AOSS
export NUM_PER_BATCH=100
```

--------------------------------

### Set Multi-tenant Context

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/01-core-api.md

Configures the client with tenant labels to be used during multi-tenant testing scenarios.

```python
def set_multitenant_context(self, tenant_labels: list[str]) -> None
```

--------------------------------

### Configure Dataset Source

Source: https://github.com/zilliztech/vectordbbench/blob/main/_autodocs/04-cli-reference.md

Set the environment variable to define the dataset source provider.

```bash
# Use Aliyun OSS instead of AWS S3
export DATASET_SOURCE=AliyunOSS

# Use ir_datasets for full-text search
export DATASET_SOURCE=IR_DATASETS

# Default is S3
export DATASET_SOURCE=S3
```
Tokens:
Raw