SemaDB

https://semadb.com
Admin
SemaDB is a no-fuss multi-index hybrid vector database and search engine designed with a simple JSON...
Tokens:36,955
Snippets:217
Trust Score:5.3
Update:9 months ago
Show doc for...
Tokens:
Raw
### Install Python Libraries for SemaDB

Source: https://semadb.com/docs/getting-started

Installs the 'sentence-transformers' library for generating embeddings and the 'requests' library for making HTTP requests to the SemaDB API. These are essential for interacting with SemaDB in Python.

```bash
# Install the required libraries
pip install -U sentence-transformers requests

```

--------------------------------

### Python: Embed and Search Vectors with SemaDB

Source: https://semadb.com/docs/getting-started

This snippet demonstrates how to embed a search query using a pre-trained model and then perform a vector search against a SemaDB collection. It includes steps for normalizing the search vector and formatting the search payload. The output shows example results from the vector search.

```python
# Create a search string and embed it using the same method
search_sentence = "What can I wear on a hot day?"
search_vector = model.encode(search_sentence)
# Normalise the search vector for cosine distance, can be ignored if using
# euclidean distance
search_vector = search_vector / np.linalg.norm(search_vector)
print(search_vector.shape)

# Ask SemaDB to perform vector search
import pprint
payload = {
    "query": {
        "property": "vector",
        "vectorVamana": {
            "vector": search_vector.tolist(),
            "operator": "near",
            "searchSize": 75,
            "limit": 3
        }
    },
    # Restrict what is returned
    "select": ["sentence"],
    "limit": 3
}
response = requests.post(base_url+"/collections/mycollection/points/search", json=payload, headers=headers)
pprint.pprint(response.json())
```

--------------------------------

### Compile and Run SemaDB Standalone

Source: https://semadb.com/docs/deployment

This snippet shows how to clone the SemaDB repository, compile the Go code, and run it as a standalone process with a specified configuration file. Ensure you have Go installed or use pre-built binaries. The `SEMADB_CONFIG` environment variable points to the configuration file.

```shell
git clone --depth=1 https://github.com/Semafind/semadb.git
cd semadb
# Assuming you have Go installed to compile the code, or you can download the
# pre-built binaries.
go build
# Specify your own configuration file
SEMADB_CONFIG=config.yaml ./semadb

```

--------------------------------

### Deploy SemaDB using Docker Compose

Source: https://semadb.com/docs/deployment

Manage SemaDB deployment with Docker Compose. This example defines a single service for SemaDB, setting the image, hostname, restart policy, port mappings, volume mounts for configuration and data, and environment variables. It also includes resource limits and reservations. The hostname must match the server name in the configuration file for correct routing.

```yaml
services:
  semadb:
    image: ghcr.io/semafind/semadb:main
    # **The hostname must match the server name in the configuration file**.
    # This is how SemaDB knows which server it is itself, otherwise it will try
    # to indefinitely route requests.
    hostname: semadb
    restart: unless-stopped
    ports:
      - "8080:8080" # HTTP API as set in the configuration file
      # - "9898:9898" # RPC API
      # - "8090:8090" # Metrics
    volumes:
      - ./:/config
      - semadb_data_1:/semadb-data
    environment:
      - SEMADB_CONFIG=/config/config.yaml
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 12G
        reservations:
          cpus: '1.0'
          memory: 2G
  # Repeat the service definition for each server in the cluster although this
  # is unlikely to be on the same server. You'll probably run the above on
  # multiple servers.
  # semadb2:
  #   ...

volumes:
  semadb_data_1:

```

--------------------------------

### Configure SemaDB API Request Headers in Python

Source: https://semadb.com/docs/getting-started

Sets up the necessary headers for making authenticated requests to the SemaDB API. This includes the base URL, content type, and API credentials. The headers vary slightly for cloud-based versus self-hosted instances.

```python
import requests

base_url = "https://semadb.p.rapidapi.com"
# Or use an appropriate base_url if you are using a self-hosted instance, e.g.
# base_url = "http://localhost:8081/v2"

headers = {
	"content-type": "application/json",
	"X-RapidAPI-Key": "<SEMADB_API_KEY>",
	"X-RapidAPI-Host": "semadb.p.rapidapi.com"
    # Or if self-hosting
    # "X-User-Id": "<USER_ID>",
    # "X-User-Plan": "BASIC"
}

```

--------------------------------

### Bulk Insert Data into SemaDB using MessagePack in Python

Source: https://semadb.com/docs/getting-started

Demonstrates inserting a large amount of data into SemaDB using MessagePack, a binary format that is faster and more compact than JSON. This requires changing the 'content-type' header and using the 'msgpack' library for encoding the payload.

```python
import msgpack # pip install msgpack

payload = { "points": points } # Same as before
headers["content-type"] = "application/msgpack" # Change the content type to msgpack
response = requests.post(base_url+"/collections/mycollection/points", data=msgpack.dumps(payload), headers=headers)

```

--------------------------------

### SemaDB YAML Configuration Example

Source: https://semadb.com/docs/configuration

This snippet provides a comprehensive example of the SemaDB configuration file in YAML format. It illustrates settings for debug mode, cluster node details (root directory, servers, RPC parameters, shard limits), and HTTP API configurations (ports, metrics, authentication, IP whitelisting, user plans).

```yaml
# -------------------------------
# Configuration for SemaDB
# -------------------------------
# Enable debug mode, makes things more verbose and logs more information
# the &GLOBAL_DEBUG is a YAML anchor, it is used to reference the value
# in other parts of the configuration file.
debug: &GLOBAL_DEBUG true
# Enable pretty log output, makes logs more readable. Otherwise they are output
# in JSON format for easier parsing by log aggregators
prettyLogOutput: true
# -------------------------------
# The cluster node configuration
clusterNode:
  # -------------------------------
  # Root directory refers to the start of the filesystem where all data is stored
  rootDir: &ROOT_DIR ./data
  # List of known servers at the beginning. If you are running a single server,
  # it should be the hostname and port of the server itself. If it is a cluster,
  # it should be the hostname and port of all the servers in the cluster.
  # Important: The port refers to the RPC port defined below
  servers:
    - localhost:11001
  # -------------------------------
  # RPC Parameters
  # The full RPC address is constructed as rpcHost + rpcDomain + ":" + rpcPort
  rpcHost: localhost
  rpcDomain: "" # With preceding dot, e.g. ".example.com"
  rpcPort: 11001
  rpcTimeout: 300 # seconds
  rpcRetries: 2
  # -------------------------------
  # Node database parameters. This database usually store collection
  # information. The backup creates a local copy in the same filesystem.
  backupFrequency: 3600 # 1 hour
  backupCount: 3
  # -------------------------------
  # Maximum size of shards in bytes. Either the shard size is reached or the
  # shard point count is reached. These parameters are used when distributing
  # points to shards automatically. If you want a single shard, then set these
  # to something really high.
  maxShardSize: 2147483648 # 2GiB
  maxShardPointCount: 100000 # 100K
  # Maximum limit on search queries to avoid large results sets. It's a
  # precaution mainly, can be safely set to higher limit because the search
  # request limits are applied as well.
  maxSearchLimit: 75
  # -------------------------------
  # Shard manager configuration
  shardManager:
    rootDir: *ROOT_DIR
    # Shard timeout in seconds. After this period the shard database file is
    # closed and any shared in-memory decoded cache is released. Note that the
    # operating system cache holding the database file pages are untouched and
    # may still report high memory usage.
    shardTimeout: 300 # 5 minutes
    # Maximum shared shard cache in bytes. This size is shared amongst all the
    # shards and their indices. If the size is reached, least recently used
    # cache entries are evicted until the size is below the limit. This only
    # occurs after operations are complete and during an operation it may exceed
    # this limit to operate safely.
    maxCacheSize: 1073741824 # 1GiB
# -------------------------------
# The user facing HTTP API configuration
httpApi:
  debug: *GLOBAL_DEBUG
  # HTTP Parameters
  httpHost: ""
  httpPort: 8081
  enableMetrics: true
  metricsHttpHost: ""
  metricsHttpPort: 8091
  # The secret string that is used to authenticate the proxy requests. If the
  # secret is empty, then the check is disabled. Use this to prevent unauthorized
  # access to the HTTP API.
  proxySecret: ""
  # Alongside or instead of the secret, you can use IP whitelisting to allow
  # only certain IPs to access the HTTP API. Either list all the IPs or set to
  # ['*'] to allow all IPs.
  whiteListIPs:
    - "*"
  # User Plans are mainly used for rate limiting and to enforce quotas. It isn't
  # designed to monetize the service which may require subscriptions and payment
  # gateways. Here we are just interested in putting some safeguards in place
  # before someone accidentally or intentionally abuses the service, e.g. by
  # creating too many collections.
  userPlans:
    BASIC:
      name: Basic Plan
      maxCollections: 1
      maxCollectionPointCount: 1000000 # 1M
      # This is after the point is encoded and includes any vector fields.
      maxPointSize: 1024 # 1KiB
      shardBackupFrequency: 3600 # 1 hour
      shardBackupCount: 2

```

--------------------------------

### Run SemaDB from Source

Source: https://semadb.com/index

This code snippet shows how to compile and run the SemaDB server from its source code using Go. It requires Go to be installed and a configuration file (e.g., singleServer.yaml) to be present. The command sets the configuration path and then executes the main Go program.

```bash
SEMADB_CONFIG=./config/singleServer.yaml go run ./ 
```

--------------------------------

### SemaDB Composite Query Example (_and)

Source: https://semadb.com/docs/search/overview

This example demonstrates a composite query in SemaDB using the '_and' operator. It combines two conditions: an integer equality check on the 'size' field and a float lessThan check on the 'price' field.

```json
{
    "query": {
        "property": "_and",
        "_and": [
            {
                "property": "size",
                "integer": {
                    "value": 10,
                    "operator": "equals"
                }
            },
            {
                "property": "price",
                "float": {
                    "value": 100,
                    "operator": "lessThan"
                }
            }
        ]
    },
    "limit": 10
}
```

--------------------------------

### Insert Data API (MessagePack)

Source: https://semadb.com/docs/getting-started

This endpoint allows you to insert data points, including vectors and associated fields, into a specified collection using MessagePack binary format for efficiency.

```APIDOC
## POST /collections/{collectionId}/points

### Description
Inserts one or more data points into a specified collection using MessagePack encoding. This is recommended for bulk inserts due to its efficiency.

### Method
POST

### Endpoint
/collections/{collectionId}/points

### Parameters
#### Path Parameters
- **collectionId** (string) - Required - The ID of the collection to insert points into.

#### Query Parameters
None

#### Request Body
- **Content-Type**: `application/msgpack`
- The request body should be a MessagePack encoded object containing a `points` array. Each point object can have a `vector` field (array of floats) and other custom fields.

### Request Example
(Assuming `points` array is defined and `msgpack` library is used for encoding)
```
import msgpack

payload = { "points": points } # 'points' is a list of dictionaries
encoded_payload = msgpack.dumps(payload)

# Send encoded_payload with headers including 'content-type': 'application/msgpack'
```

### Response
#### Success Response (200)
- **message** (string) - Indicates the status of the operation (e.g., "success").
- **failedRanges** (array) - An array of any failed ranges, typically empty on success.

#### Response Example
```json
{
  "message": "success",
  "failedRanges": []
}
```
```

--------------------------------

### SemaDB Example Point Structure

Source: https://semadb.com/docs/concepts/indexing

This JSON object represents an example data point (or document) that conforms to the previously defined 'products' collection index schema. It includes indexed fields like SKU, description embedding, description, category, labels, size, price, and creation date, along with non-indexed fields like currency and updated date.

```json
{
    "SKU": "1234",
    "currency": "GBP",
    "descriptionEmbedding": [0.1, 0.2, 0.3, ...],
    "description": "This is a product description",
    "category": "electronics",
    "labels": ["new", "sale"],
    "size": 10,
    "price": 100.0,
    "dates": {
        "created": "2021-01-01",
        "updated": "2021-01-02"
    }
}
```

--------------------------------

### Kitchen Sink Search API

Source: https://semadb.com/docs/search/kitchen

An example of a complex search query that combines vector similarity with text search, including filters for stock and price, and weighting for hybrid search results.

```APIDOC
## POST /websites/semadb

### Description
This endpoint allows for complex search queries, combining vector similarity search with text-based search. It supports various filtering options, such as stock availability and price range, and allows for weighting different search components to influence the results. The endpoint can return specified fields and sort the results based on given criteria.

### Method
POST

### Endpoint
/websites/semadb

### Request Body
- **query** (object) - Required - The main search query object, which can include nested conditions like _or and _and.
  - **property** (string) - Required - The property to search on.
  - **vectorVamana** (object) - Optional - Configuration for vector similarity search.
    - **vector** (array) - Required - The vector to compare against.
    - **operator** (string) - Required - The similarity operator (e.g., 'near').
    - **searchSize** (integer) - Optional - The number of results to consider before applying limits.
    - **limit** (integer) - Optional - The maximum number of results to return.
    - **filter** (object) - Optional - A filter to apply to the vector search results.
    - **weight** (float) - Optional - The weight of this vector search component in a hybrid search.
  - **text** (object) - Optional - Configuration for text search.
    - **value** (string) - Required - The text value to search for.
    - **operator** (string) - Required - The text operator (e.g., 'containsAll').
    - **limit** (integer) - Optional - The maximum number of results to return.
    - **weight** (float) - Optional - The weight of this text search component in a hybrid search.
  - **integer** (object) - Optional - Configuration for integer field search.
    - **operator** (string) - Required - The integer operator (e.g., 'greaterThan').
    - **value** (integer) - Required - The integer value to compare against.
  - **float** (object) - Optional - Configuration for float field search.
    - **operator** (string) - Required - The float operator (e.g., 'lessThan').
    - **value** (float) - Required - The float value to compare against.
- **select** (array) - Required - An array of strings specifying which fields to return in the results.
- **sort** (array) - Optional - An array of objects specifying how to sort the results.
  - **field** (string) - Required - The field to sort by.
  - **descending** (boolean) - Required - Whether to sort in descending order.
- **limit** (integer) - Optional - The maximum number of results to return.

### Request Example
```json
{
    "query": {
        "property": "_or",
        "_or": [
            {
                "property": "productEmbedding",
                "vectorVamana": {
                    "vector": [1, 2],
                    "operator": "near",
                    "searchSize": 75,
                    "limit": 10,
                    "filter": {
                        "property": "_and",
                        "_and": [
                            {
                                "property": "stock",
                                "integer": {
                                    "operator": "greaterThan",
                                    "value": 0
                                }
                            },
                            {
                                "property": "price",
                                "float": {
                                    "operator": "lessThan",
                                    "value": 100
                                }
                            }
                        ]
                    },
                    "weight": 0.3
                }
            },
            {
                "property": "_and",
                "_and": [
                    {
                        "property": "description",
                        "text": {
                            "value": "summer dress",
                            "operator": "containsAll",
                            "limit": 10,
                            "weight": 0.7
                        }
                    },
                    {
                        "property": "stock",
                        "integer": {
                            "operator": "greaterThan",
                            "value": 0
                        }
                    }
                ]
            }
        ]
    },
    "select": ["title", "description", "price"],
    "sort": [
        {
        "field": "price",
        "descending": false
        }
    ],
    "limit": 5
}
```

### Response
#### Success Response (200)
- **title** (string) - The title of the product.
- **description** (string) - The description of the product.
- **price** (float) - The price of the product.

#### Response Example
```json
[
  {
    "title": "Summer Maxi Dress",
    "description": "A light and airy dress perfect for summer days.",
    "price": 75.50
  },
  {
    "title": "Casual Summer Dress",
    "description": "Comfortable cotton dress for everyday wear.",
    "price": 45.00
  }
]
```
```

--------------------------------

### Composite Queries Example

Source: https://semadb.com/docs/search/overview

Demonstrates how to construct composite queries using logical operators like `_and` and `_or` to combine multiple search criteria for more complex searches.

```APIDOC
## POST /collections/{collectionId}/points/search (Composite Query Example)

### Description
This example illustrates the use of composite queries by combining multiple conditions using the `_and` operator. This allows for sophisticated filtering based on multiple criteria simultaneously.

### Method
POST

### Endpoint
`/collections/{collectionId}/points/search`

### Parameters
#### Path Parameters
- **collectionId** (string) - Required - The ID of the collection to search within.

#### Query Parameters
None

#### Request Body
- **query** (object) - Required - The composite query object.
  - **property** (string) - Required - Must be `_and` or `_or` to combine sub-queries.
  - **_and** (array) - Required if `property` is `_and`. An array of query objects to be ANDed together.
  - **_or** (array) - Required if `property` is `_or`. An array of query objects to be ORed together.
    - Each element in `_and` or `_or` is a standard query object (e.g., filtering by 'size' or 'price').
- **limit** (integer) - Required - The maximum number of results to return.

### Request Example
```json
{
    "query": {
        "property": "_and",
        "_and": [
            {
                "property": "size",
                "integer": {
                    "value": 10,
                    "operator": "equals"
                }
            },
            {
                "property": "price",
                "float": {
                    "value": 100,
                    "operator": "lessThan"
                }
            }
        ]
    },
    "limit": 10
}
```

### Response
#### Success Response (200)
- **points** (array) - An array of search result objects matching all specified criteria.
  - **_id** (string) - The unique identifier of the point.
  - **[selected_field]** (any) - Any field that was specified in the `select` array (if provided).

#### Response Example
```json
{
  "points": [
    {
      "_id": "some-uuid-1",
      "description": "Small and cheap product",
      "price": 50
    },
    {
      "_id": "some-uuid-2",
      "description": "Small product just under budget",
      "price": 99
    }
  ]
}
```
```

--------------------------------

### Create a SemaDB Collection with Vamana Index in Python

Source: https://semadb.com/docs/getting-started

Defines and creates a new collection in SemaDB using the Vamana index type. This involves specifying parameters like vector size, distance metric, search size, degree bound, and alpha for the index configuration.

```python
payload = {
	"id": "mycollection",
    "indexSchema": {
        "vector": {
            "type": "vectorVamana",
            "vectorVamana": {
                "vectorSize": 384, # Sentence transformers give embeddings of size 384
                "distanceMetric": "cosine",
                "searchSize": 75, # How exhaustive the search should be?
                "degreeBound": 64, # How dense the graph should be?
                "alpha": 1.2, # How much longer edges should be preferred?
            }
        }
    }
}

response = requests.post(base_url + "/collections", json=payload, headers=headers)

print(response.json())

```

--------------------------------

### Create Collection API

Source: https://semadb.com/docs/getting-started

This endpoint allows you to create a new collection in SemaDB. A collection is a container for points (documents) that share the same search space and indexing schema.

```APIDOC
## POST /collections

### Description
Creates a new collection with a specified ID and index schema. The index schema defines how vector data within the collection will be indexed and searched.

### Method
POST

### Endpoint
/collections

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **id** (string) - Required - The unique identifier for the collection.
- **indexSchema** (object) - Required - Defines the schema for indexing, including vector properties like type, size, and distance metric.
  - **vector** (object) - Required - Specifies the indexing for vector fields.
    - **type** (string) - Required - The type of index to use (e.g., "vectorVamana").
    - **vectorVamana** (object) - Required if type is "vectorVamana" - Configuration for the Vamana index.
      - **vectorSize** (integer) - Required - The dimensionality of the vectors.
      - **distanceMetric** (string) - Required - The distance metric to use (e.g., "cosine", "euclidean").
      - **searchSize** (integer) - Optional - Controls the exhaustiveness of the search.
      - **degreeBound** (integer) - Optional - Controls the density of the graph.
      - **alpha** (float) - Optional - A parameter for preferring longer edges.

### Request Example
```json
{
	"id": "mycollection",
    "indexSchema": {
        "vector": {
            "type": "vectorVamana",
            "vectorVamana": {
                "vectorSize": 384,
                "distanceMetric": "cosine",
                "searchSize": 75,
                "degreeBound": 64,
                "alpha": 1.2
            }
        }
    }
}
```

### Response
#### Success Response (200)
- **message** (string) - Indicates the status of the operation (e.g., "success").
- **failedRanges** (array) - An array of any failed ranges, typically empty on success.

#### Response Example
```json
{
  "message": "success",
  "failedRanges": []
}
```
```

--------------------------------

### Python: Print JSON Response

Source: https://semadb.com/docs/getting-started

This basic Python snippet demonstrates how to parse and print a JSON response received from an API. It's a common utility for inspecting data returned by services like SemaDB.

```python
print(response.json())
```

--------------------------------

### SemaDB Single Vector Search Query Example

Source: https://semadb.com/docs/search/overview

This example demonstrates a single vector search query in SemaDB. It specifies a vector to search by, a search operator, and limits for both the internal vector search and the overall results. It also includes sorting by a specific field.

```json
{
    "query": {
        "property": "productEmbedding",
        "vectorVamana": {
            "vector": [1, 2],
            "operator": "near",
            "searchSize": 75,
            "limit": 10 
        }
    },
    "select": ["description", "price"],
    "sort": [
      {
        "field": "price",
        "descending": false
      }
    ],
    "limit": 5
}
```

--------------------------------

### SemaDB Collection Index Schema Example

Source: https://semadb.com/docs/concepts/indexing

This JSON object defines an example index schema for a collection named 'products' in SemaDB. It specifies different index types for various fields, such as vector embeddings, text, strings, integers, and floats. This schema dictates how data is structured for efficient searching.

```json
{
    "id": "products",
    "indexSchema": {
        "descriptionEmbedding": {
            "type": "vectorVamana",
            "vectorVamana": {
                "vectorSize": 386,
                "distanceMetric": "euclidean",
                "searchSize": 75,
                "degreeBound": 64,
                "alpha": 1.2
            }
        },
        "description": {
            "type": "text",
            "text": {
                "analyser": "standard"
            }
        },
        "category": {
            "type": "string",
            "string": {
                "caseSensitive": false
            }
        },
        "labels": {
            "type": "stringArray",
            "stringArray": {
                "caseSensitive": false
            }
        },
        "size": {
            "type": "integer"
        },
        "price": {
            "type": "float"
        },
        "dates.created": {
            "type": "string",
            "string": {
                "caseSensitive": false
            }
        }
    }
}
```

--------------------------------

### Composite Hybrid Query Example in SemaDB

Source: https://semadb.com/docs/search/hybrid

This example demonstrates how to construct a composite hybrid query in SemaDB. It combines vector search on 'productEmbedding' with text searches on 'description' and 'title'. The query uses the '_or' operator to merge results from different search types. Each search component includes parameters like 'vector', 'operator', 'searchSize', 'limit', and 'weight' to fine-tune the search and scoring.

```json
{
    "query": {
        "property": "_or",
        "_or": [
            {
                "property": "productEmbedding",
                "vectorVamana": {
                    "vector": [1, 2],
                    "operator": "near",
                    "searchSize": 75,
                    "limit": 10,
                    "weight": 0.2
                }
            },
            {
                "property": "description",
                "text": {
                    "value": "summer floral",
                    "operator": "containsAny",
                    "limit": 10,
                    "weight": 0.5
                }
            },
            {
                "property": "title",
                "text": {
                    "value": "maxi dress",
                    "operator": "containsAll",
                    "limit": 10,
                    "weight": 0.3
                }
            }
        ]
    },
    "select": ["title", "price"],
    "limit": 10
}
```

--------------------------------

### SemaDB Search Response Example (JSON)

Source: https://semadb.com/api-reference

This JSON response illustrates the structure of results returned by the SemaDB search endpoint. It includes an array of points, each with its unique ID, distance metric, relevance score, and associated properties like description and price.

```json
{
  "points": [
    {
      "_id": "faefe2b1-cf85-48db-9621-94b833ee9cc9",
      "_distance": 0,
      "_hybridScore": "-0,",
      "description": "A product",
      "price": 100
    },
    {
      "_id": "d2e3ebb5-149a-46e4-8b3f-cd6c38530da3",
      "_distance": "314402.94,",
      "_hybridScore": -314402.94,
      "description": "Another product",
      "price": 200
    }
  ]
}
```

--------------------------------

### Get Collection Details Response Sample (JSON)

Source: https://semadb.com/api-reference

This JSON sample illustrates a successful response for retrieving the details of a specific collection in SemaDB. It includes the collection's schema and shard information.

```json
{
  "id": "mycollection",
  "indexSchema": {
    "myvector": {
      "type": "vectorVamana",
      "vectorVamana": {
        "vectorSize": 2,
        "distanceMetric": "euclidean",
        "searchSize": 75,
        "degreeBound": 64,
        "alpha": 1.2
}
}
},
  "shards": [
    {
      "id": "fff3a226-b9f8-4375-8dbd-1a240e000705",
      "pointCount": 42
}
]

}
```

--------------------------------

### Basic Search Query for Price Range

Source: https://semadb.com/docs/search/basic

Demonstrates a basic search query in SemaDB to find documents within a specified price range. It utilizes the 'inRange' operator for float properties and sets a limit for the number of results. This is a fundamental example of querying inverted indices.

```json
{
    "query": {
        "property": "price",
        "float": {
            "value": 100,
            "operator": "inRange",
            "endValue": 200
        }
    },
    "limit": 10
}
```

--------------------------------

### Example SemaDB Point/Document Structure

Source: https://semadb.com/docs/concepts/point

Illustrates the structure of a typical SemaDB point (document), which is a JSON object. It demonstrates various field types including strings, integers, arrays, and nested objects. Fields used for indexing must match the index's expected format.

```json
{
    "city": "Edinburgh",
    "country": "Scotland",
    "areaCode": 0131,
    "population": 500000,
    "embedding": [0.1, 0.2, 0.3, 0.4, 0.5],
    "tags": ["capital", "historic", "tourist"],
    "link": "https://en.wikipedia.org/wiki/Edinburgh",
    "location": {
        "coordinates": [55.953251, -3.188267],
        "map": "https://www.openstreetmap.org/?mlat=55.953251&mlon=-3.188267"
    }
}
```

--------------------------------

### Run SemaDB with Docker

Source: https://semadb.com/index

These commands demonstrate how to run the SemaDB server using Docker and Podman container images. They include mounting configuration and data directories, exposing the necessary port, and specifying the configuration file via an environment variable. The examples show running the latest version from a registry and building/running a local image.

```bash
# Run latest version from ghcr.io
docker run -it --rm -v ./config:/config -e SEMADB_CONFIG=/config/singleServer.yaml -v ./data:/data -p 8081:8081 ghcr.io/semafind/semadb:main
# If using podman
podman run -it --rm -v ./config:/config:Z -e SEMADB_CONFIG=/config/singleServer.yaml -v ./data:/data:Z -p 8081:8081 ghcr.io/semafind/semadb:main

# Build and run local image
docker build -t semadb ./
docker run -it --rm -v ./config:/config -e SEMADB_CONFIG=/config/singleServer.yaml -v ./data:/data -p 8081:8081 semadb
# If using podman
podman build -t semadb ./
# The :Z argument relabels to access: see https://github.com/containers/podman/issues/3683
podman run -it --rm -v ./config:/config:Z -e SEMADB_CONFIG=/config/singleServer.yaml -v ./data:/data:Z -p 8081:8081 semadb
```

--------------------------------

### Delete Collection API

Source: https://semadb.com/docs/getting-started

Deletes a specified collection and all its associated data points. This is useful for cleaning up resources after use.

```APIDOC
## DELETE /collections/mycollection

### Description
Deletes a specified collection and all its associated data points. This is useful for cleaning up resources after use.

### Method
DELETE

### Endpoint
/collections/mycollection

### Parameters
#### Path Parameters
- **collection_name** (string) - Required - The name of the collection to delete.

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **message** (string) - A confirmation message indicating the collection was deleted.

#### Response Example
```json
{
  "message": "Collection 'mycollection' deleted successfully."
}
```
```

--------------------------------

### Insert Data API (JSON)

Source: https://semadb.com/docs/getting-started

This endpoint allows you to insert data points, including vectors and associated fields, into a specified collection using JSON format.

```APIDOC
## POST /collections/{collectionId}/points

### Description
Inserts one or more data points into a specified collection. Each point can contain a vector and arbitrary fields. This version uses JSON for the request body.

### Method
POST

### Endpoint
/collections/{collectionId}/points

### Parameters
#### Path Parameters
- **collectionId** (string) - Required - The ID of the collection to insert points into.

#### Query Parameters
None

#### Request Body
- **points** (array) - Required - An array of point objects to insert.
  - Each point object can have a **vector** field (array of floats) and other custom fields (e.g., **myfield**).

### Request Example
```json
{
  "points": [
    {'vector': [0.1, 0.2, ...], "myfield": 0},
    {'vector': [0.3, 0.4, ...], "myfield": 1}
  ]
}
```

### Response
#### Success Response (200)
- **message** (string) - Indicates the status of the operation (e.g., "success").
- **failedRanges** (array) - An array of any failed ranges, typically empty on success.

#### Response Example
```json
{
  "message": "success",
  "failedRanges": []
}
```
```

--------------------------------

### Vector Search API

Source: https://semadb.com/docs/getting-started

Performs a vector search within a specified collection using a query vector. This endpoint allows you to find semantically similar data points based on vector embeddings.

```APIDOC
## POST /collections/mycollection/points/search

### Description
Performs a vector search within a specified collection using a query vector. This endpoint allows you to find semantically similar data points based on vector embeddings.

### Method
POST

### Endpoint
/collections/mycollection/points/search

### Parameters
#### Query Parameters
None

#### Request Body
- **query** (object) - Required - Defines the search criteria, including the property to search on, the vector details, and search parameters.
  - **property** (string) - Required - The name of the vector property to search.
  - **vectorVamana** (object) - Required - Specifies the vector for the search.
    - **vector** (array) - Required - The embedding vector to search for.
    - **operator** (string) - Required - The search operator (e.g., 'near').
    - **searchSize** (integer) - Optional - The number of results to initially retrieve before ranking.
    - **limit** (integer) - Optional - The maximum number of results to return.
- **select** (array) - Optional - Specifies which fields to include in the returned results.
- **limit** (integer) - Optional - The maximum number of results to return.

### Request Example
```json
{
  "query": {
    "property": "vector",
    "vectorVamana": {
      "vector": [0.1, 0.2, ..., 0.9],
      "operator": "near",
      "searchSize": 75,
      "limit": 3
    }
  },
  "select": ["sentence"],
  "limit": 3
}
```

### Response
#### Success Response (200)
- **points** (array) - An array of found data points.
  - **_distance** (float) - The distance between the query vector and the result vector.
  - **_hybridScore** (float) - The score used to rank the results.
  - **_id** (string) - The unique identifier of the data point.
  - [other fields specified in 'select']

#### Response Example
```json
{
  "points": [
    {
      "_distance": 0.6202282,
      "_hybridScore": -0.6202282,
      "_id": "93687544-6aaa-437e-a3bc-2983b1104aea",
      "sentence": "This flowing floral maxi dress is perfect for a summer day."
    },
    {
      "_distance": 0.64442706,
      "_hybridScore": -0.64442706,
      "_id": "3d81846f-fa4f-4ab9-abd6-7c45a944f87a",
      "sentence": "A classic black turtleneck sweater is a versatile piece."
    },
    {
      "_distance": 0.6641861,
      "_hybridScore": -0.6641861,
      "_id": "667a3d27-f270-4c10-9352-77dfff328160",
      "sentence": "These ripped denim jeans are a must-have for any fashion-forward wardrobe."
    }
  ]
}
```
```

--------------------------------

### Deploy SemaDB using Docker

Source: https://semadb.com/docs/deployment

Deploy SemaDB as a Docker container. This command mounts a local configuration directory and a data volume, maps the container's port 8081 to the host, and runs the main Docker image. Remember to use a specific release tag instead of 'main' for stable deployments.

```docker
docker run -it --rm -v ./config:/config -e SEMADB_CONFIG=/config/semadb-config.yaml -v ./data:/data -p 8081:8081 ghcr.io/semafind/semadb:main

```

--------------------------------

### Generate Sentence Embeddings with Sentence Transformers in Python

Source: https://semadb.com/docs/getting-started

Generates vector embeddings for a list of sentences using the 'all-MiniLM-L6-v2' model from the Sentence Transformers library. It also includes normalization of embeddings, which is crucial for cosine distance calculations in SemaDB.

```python
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')

sentences = [
    "This flowing floral maxi dress is perfect for a summer day.",
    "These ripped denim jeans are a must-have for any fashion-forward wardrobe.",
    "A classic black turtleneck sweater is a versatile piece.",
]

embeddings = model.encode(sentences)
# We normalise the embeddings because for cosine distance SemaDB expects
# normalised vectors. We can skip this step for example if we use euclidean
# distance.
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)

print(embeddings.shape)

```

--------------------------------

### SemaDB User Plan Configuration Example

Source: https://semadb.com/docs/concepts/multi-tenancy

This snippet shows a sample configuration for defining different User Plans in SemaDB. User Plans are used to enforce limits on resources such as the maximum number of collections, maximum points per collection, and maximum point size, helping to manage usage and prevent abuse.

```yaml
userPlans:
  BASIC:
    name: Basic Plan
    maxCollections: 1
    maxCollectionPointCount: 1000000 # 1M
    # This is after the point is encoded and includes any vector fields.
    maxPointSize: 1024 # 1KiB
    shardBackupFrequency: 3600 # 1 hour
    shardBackupCount: 2

```

--------------------------------

### Python: Delete SemaDB Collection

Source: https://semadb.com/docs/getting-started

This snippet shows how to delete a specific collection from SemaDB using a DELETE request. It's a straightforward operation used for cleanup after performing operations on a collection. The response indicates the success or failure of the deletion.

```python
# Clean up by deleting the collection
response = requests.delete(base_url+"/collections/mycollection", headers=headers)

print(response.json())
```

--------------------------------

### Insert Vector Points into SemaDB Collection in Python

Source: https://semadb.com/docs/getting-started

Inserts generated vector embeddings along with associated data points into a specified SemaDB collection. Each point includes the 'vector' field and an optional 'myfield' for additional data. The data is prepared as a list of dictionaries.

```python
# Insert the points into the collection for searching
points = []
for i in range(embeddings.shape[0]):
    # Here the "vector" field is the one that is indexed as per the indexSchema
    # definition. But "myfield" is completely optional and can be any field you
    # want to store.
    points.append({'vector': embeddings[i].tolist(), "myfield": i})
payload = { "points": points }

response = requests.post(base_url+"/collections/mycollection/points", json=payload, headers=headers)

print(response.json())

```

--------------------------------

### Kubernetes StatefulSet Deployment for SemaDB

Source: https://semadb.com/docs/deployment

This YAML defines a Kubernetes StatefulSet to deploy SemaDB. It configures replicas, container image, resource requests/limits, ports, environment variables, volume mounts for configuration and data, readiness probes, and pod anti-affinity rules for distribution across nodes. It also includes volumeClaimTemplates for persistent storage.

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: semadb
  namespace: semadb
spec:
  selector:
    matchLabels:
      app: semadb
  serviceName: semadb
  replicas: 3
  podManagementPolicy: Parallel
  revisionHistoryLimit: 5
  template:
    metadata:
      labels:
        app: semadb
    spec:
      terminationGracePeriodSeconds: 40
      containers:
      - name: semadb
        image: ghcr.io/semafind/semadb:main
        resources:
          requests:
            cpu: "0.5"
            memory: 1Gi
          limits:
            cpu: "3.0"
            memory: 4Gi
        ports:
        - containerPort: 8080
          name: httpapi
        - containerPort: 9898
          name: rpcapi
        - containerPort: 8090
          name: metrics
        env:
        - name: SEMADB_CONFIG
          value: /etc/semadb/config.yaml
        volumeMounts:
          - mountPath: /semadb-data
            name: semadb-data-pvc
          - mountPath: /etc/semadb
            name: semadb-config-volume
        readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 3
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
            tcpSocket:
              host: ''
              port: 8080
      volumes:
        - name: semadb-config-volume
          configMap:
            name: semadb-config-map
            items:
              - key: semadb-config.yaml
                path: config.yaml
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: semadb
  volumeClaimTemplates:
  - metadata:
      name: semadb-data-pvc
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
      # storageClassName: ...

```
Tokens:
Raw