nrtSearch

https://github.com/yelp/nrtsearch
Admin
nrtSearch is a high-performance gRPC server with optional REST APIs built on Apache Lucene,...
Tokens:46,967
Snippets:439
Trust Score:9.1
License:Apache-2.0
Update:6 months ago
Show doc for...
Tokens:
Raw
### Start Index

Source: https://github.com/yelp/nrtsearch/wiki/Home

Start the index using a configuration file.

```bash
./build/install/platypus/bin/lucene-client startIndex -f startIndex.json
cat startIndex.json
{
  "indexName" : "testIdx"
}
```

--------------------------------

### Start NRTsearch Server with Configuration

Source: https://github.com/yelp/nrtsearch/blob/main/docs/server_configuration.md

Command to start the NRTsearch server, providing the server configuration file as an argument. Ensure the path to the executable and configuration file are correct.

```bash
./build/install/nrtsearch/bin/nrtsearch_server server_configuration.yaml
```

--------------------------------

### Install Docker Compose and Configure AWS Credentials

Source: https://github.com/yelp/nrtsearch/blob/main/docs/docker_compose.md

Install the required docker-compose binary and set up the boto.cfg file for S3 access.

```shell
# install docker-compose
shell% sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
shell% sudo chmod +x /usr/local/bin/docker-compose
# set the AWS credentials in your boto.cfg file, which must be located in the project root directory.
shell% cat boto.cfg
[default]
region = eu-central-1
aws_access_key_id = A--------QA
aws_secret_access_key = 7--------A
http_socket_timeout = 300
ec2_region_name = eu-central-1
ec2_region_endpoint = ec2.eu-central-1.amazonaws.com
```

--------------------------------

### Run REST Server (macOS Example)

Source: https://github.com/yelp/nrtsearch/blob/main/docs/introduction.md

Run the REST server using the appropriate platform binary. This example shows the command for macOS, specifying the gRPC and REST ports.

```bash
./build/install/nrtsearch/bin/http_wrapper-darwin-amd64 <gRPC_PORT> <REST_PORT>
```

--------------------------------

### Start nrtSearch Server

Source: https://context7.com/yelp/nrtsearch/llms.txt

Launch the nrtSearch server using a specific configuration file.

```bash
# Start server with configuration
./build/install/nrtsearch/bin/nrtsearch_server server_configuration.yaml
```

--------------------------------

### Build Platypus Server and Client

Source: https://github.com/yelp/nrtsearch/wiki/Home

Use Gradle to clean and install the distribution files.

```bash
./gradlew clean && ./gradlew installDist
```

--------------------------------

### Start Index

Source: https://context7.com/yelp/nrtsearch/llms.txt

Initializes an index in standalone, primary, or replica mode, with optional S3 restoration.

```bash
# Start standalone index
./build/install/nrtsearch/bin/nrtsearch_client startIndex -f startIndex.json

# startIndex.json for standalone
{
  "indexName": "testIdx"
}

# Start as NRT primary
{
  "indexName": "testIdx",
  "mode": "PRIMARY",
  "primaryGen": -1
}

# Start as NRT replica
{
  "indexName": "testIdx",
  "mode": "REPLICA",
  "primaryAddress": "primary-node",
  "port": 8001
}

# Start with restore from S3
{
  "indexName": "testIdx",
  "mode": "PRIMARY",
  "primaryGen": -1,
  "restore": {
    "serviceName": "my-service",
    "resourceName": "testIdx",
    "deleteExistingData": true
  }
}

# Using REST API
curl -XPOST localhost:8080/v1/start_index -d '{
  "indexName": "testIdx"
}'
```

--------------------------------

### Keyword Field Configuration Example (JSON)

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/keyword.md

Example JSON configuration for a Keyword field. Set 'type' to ATOM for exact matching.

```json
{
    "name": "keyword_field",
    "type": "ATOM",
    "search": true,
    "storeDocValues": true
}
```

--------------------------------

### Run Platypus Server

Source: https://github.com/yelp/nrtsearch/wiki/Home

Execute the server binary from the installation directory.

```bash
./build/install/platypus/bin/lucene-server
```

--------------------------------

### Start Docker Containers

Source: https://github.com/yelp/nrtsearch/blob/main/docs/docker_compose.md

Launch the primary and replica nodes using the docker-compose configuration.

```shell
shell% docker-compose -f docker-compose.yaml up
```

--------------------------------

### Initialize Replica Node Index

Source: https://github.com/yelp/nrtsearch/blob/main/docs/docker_compose.md

Configure the replica node to register with the primary and start the index.

```shell
shell% REPLICA_1_CONTAINER_ID=$(docker ps  | grep nrtsearch_replica-node-1_1 | awk '{print $1}')
shell% docker exec -it $REPLICA_1_CONTAINER_ID sh
# ./build/install/nrtsearch/bin/nrtsearch_client -h replica-node-1 -p 8002 createIndex --indexName  testIdx
# ./build/install/nrtsearch/bin/nrtsearch_client -h replica-node-1 -p 8002 settings -f docker-compose-config/settings_replica.json
# ./build/install/nrtsearch/bin/nrtsearch_client -h replica-node-1 -p 8002 registerFields -f docker-compose-config/registerFields.json
# ./build/install/nrtsearch/bin/nrtsearch_client -h replica-node-1 -p 8002 startIndex -f docker-compose-config/startIndex_replica.json
```

--------------------------------

### Example Text Field Configuration

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/text.md

An example JSON configuration for a Text field named 'text_field'. It enables search and doc values, and specifies the standard analyzer.

```json
{
    "name": "text_field",
    "type": "TEXT",
    "search": true,
    "storeDocValues": true,
    "analyzer": {
        "name": "standard"
    }
}
```

--------------------------------

### Example NRTsearch Server Configuration

Source: https://github.com/yelp/nrtsearch/blob/main/docs/server_configuration.md

An example YAML configuration file for the NRTsearch server. This includes settings for node name, host, ports, directories, thread pools, and AWS configuration.

```yaml
nodeName: "nrtsearch_primary"
hostName: "primary-node"
port: "8000"
replicationPort: "8001"
stateDir: "/user/app/primary_state"
indexDir: "/user/app/primary_index_base"
threadPoolConfiguration:
  search:
    maxThreads: 4
  index:
    useVirtualThreads: true
botoCfgPath: "/user/app/boto.cfg"
bucketName: "nrtsearch-bucket"
serviceName: "nrtsearch-service-test"
```

--------------------------------

### Search with InnerHit Examples

Source: https://github.com/yelp/nrtsearch/blob/main/docs/inner_hit.md

Examples of how to use InnerHit for various filtering scenarios including parent-only, child-only, and combined filtering.

```APIDOC
## POST /search

### Description
Perform a search request with InnerHit to retrieve nested menu items for businesses.

### Request Example (Combined Filtering)
{
  "indexName": "index_alpha",
  "query": {
    "termQuery": {
      "field": "business_name",
      "textValue": "restaurant_A"
    }
  },
  "retrieveFields": ["business_name"],
  "innerHit": {
    "query_nested_path": "menu",
    "query": {
      "rangeQuery": {
        "field": "menu.price",
        "upper": "6"
      }
    },
    "retrieve_fields": ["menu.food_name"]
  }
}
```

--------------------------------

### Initialize nrtSearch Cluster

Source: https://context7.com/yelp/nrtsearch/llms.txt

Commands to start the cluster and configure indices on primary and replica nodes.

```bash
# Start cluster
docker-compose -f docker-compose.yaml up

# Initialize primary index
docker exec -it primary-node sh
./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 createIndex --indexName testIdx
./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 settings -f docker-compose-config/settings_primary.json
./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 registerFields -f docker-compose-config/registerFields.json
./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 startIndex -f docker-compose-config/startIndex_primary.json

# Initialize replica index
docker exec -it replica-node-1 sh
./build/install/nrtsearch/bin/nrtsearch_client -h replica-node-1 -p 8002 createIndex --indexName testIdx
./build/install/nrtsearch/bin/nrtsearch_client -h replica-node-1 -p 8002 startIndex -f docker-compose-config/startIndex_replica.json
```

--------------------------------

### Run gRPC Server

Source: https://github.com/yelp/nrtsearch/blob/main/docs/introduction.md

Execute this command to start the NRT Search gRPC server locally.

```bash
./build/install/nrtsearch/bin/nrtsearch_server
```

--------------------------------

### Initialize Primary Node Index

Source: https://github.com/yelp/nrtsearch/blob/main/docs/docker_compose.md

Create the index, register fields, and start the index on the primary node.

```shell
shell% PRIMARY_CONTAINER_ID=$(docker ps | grep nrtsearch_primary-node | awk '{print $1}')
shell% docker exec -it $PRIMARY_CONTAINER_ID sh
# ./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 createIndex --indexName  testIdx
# ./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 settings -f docker-compose-config/settings_primary.json
# ./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 registerFields -f docker-compose-config/registerFields.json
# ./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 startIndex -f docker-compose-config/startIndex_primary.json
```

--------------------------------

### Index Settings Configuration File

Source: https://context7.com/yelp/nrtsearch/llms.txt

Example JSON file for configuring offline index settings.

```json
{
  "indexName": "testIdx",
  "directory": "MMapDirectory",
  "nrtCachingDirectoryMaxSizeMB": 60.0,
  "nrtCachingDirectoryMaxMergeSizeMB": 5.0,
  "concurrentMergeSchedulerMaxThreadCount": 8,
  "concurrentMergeSchedulerMaxMergeCount": 16,
  "indexMergeSchedulerAutoThrottle": false
}
```

--------------------------------

### Example JSON Configuration for Location Field

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/location.md

Illustrates how to configure a 'Location' field in NRT Search using JSON. This example enables searching and storing doc values for a field named 'location_field'.

```json
{
    "name": "location_field",
    "type": "LAT_LON",
    "search": true,
    "storeDocValues": true
}
```

--------------------------------

### Java Client Search Example

Source: https://github.com/yelp/nrtsearch/blob/main/docs/starter.md

Demonstrates how to perform a search query using the NRTSearch Java client. Requires setting up the LuceneServerStubBuilder and creating a SearchRequest.

```java
LuceneServerStubBuilder luceneServerStubBuilder = new LuceneServerStubBuilder("localhost", 8000);
LuceneServerGrpc.LuceneServerBlockingStub blockingStub = luceneServerStubBuilder.createBlockingStub();
SearchRequest request = SearchRequest.newBuilder()
  .setIndexName("testIdx")
  .setStartHit(0)
  .setTopHits(100)
  .addAllRetrieveFields(List.of("doc_id", "license_no", "vendor_name"))
  .setQueryText("vendor_name:first vendor")
  .build());
blockingStub.search(request);
```

--------------------------------

### Example Cluster Size Calculation

Source: https://github.com/yelp/nrtsearch/blob/main/docs/vector_search.md

Demonstrates calculating cluster size for 390 million documents with 512-dimensional embeddings.

```default
=> ~390M * 4 * (512 + 12) = ~817GB
```

--------------------------------

### Start NRTSearch Index

Source: https://github.com/yelp/nrtsearch/blob/main/docs/starter.md

Initiate the NRTSearch index to begin accepting documents and queries. Requires the index name specified in a JSON configuration.

```bash
# ./build/install/nrtsearch/bin/nrtsearch_client startIndex -f startIndex.json
```

```json
{
      "indexName" : "testIdx"
  }
```

--------------------------------

### Verify Docker and Docker Compose Versions

Source: https://github.com/yelp/nrtsearch/blob/main/docs/docker_compose.md

Check installed versions to ensure compatibility with the deployment configuration.

```shell
shell$ docker -v
Docker version 20.10.8, build 3967b7d28e

shell$ docker-compose -v
docker-compose version 1.27.4, build 40524192 # version 1.27 so the deploy.replicas option works
```

--------------------------------

### Prefix Query Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Use prefix queries to match terms that begin with a specified string. 'rewrite' and 'rewriteTopTermsSize' control how the query is executed.

```json
{
  "indexName": "testIdx",
  "topHits": 50,
  "retrieveFields": ["doc_id", "title"],
  "query": {
    "prefixQuery": {
      "field": "title",
      "prefix": "wid",
      "rewrite": "CONSTANT_SCORE_BOOLEAN",
      "rewriteTopTermsSize": 100
    }
  }
}
```

--------------------------------

### Configure Example Vector Field

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/vector.md

JSON configuration for a 128-dimensional vector field using L2 norm and HNSW indexing.

```json
{
    "name": "vector_field",
    "type": "VECTOR",
    "search": true,
    "storeDocValues": false,
    "vectorDimensions": 128,
    "vectorElementType": "VECTOR_ELEMENT_FLOAT",
    "vectorSimilarity": "l2_norm",
    "vectorIndexingOptions": {
        "type": "hnsw",
        "hnsw_m": 16,
        "hnsw_ef_construction": 100,
    }
}
```

--------------------------------

### Build NRT Search Locally

Source: https://github.com/yelp/nrtsearch/blob/main/docs/introduction.md

Use this command to clean, install distributions, and run tests for the NRT Search project locally. Tested on Java 17.

```bash
./gradlew clean installDist test
```

--------------------------------

### Configure Virtual Field JSON

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/virtual.md

Example JSON configuration for a virtual field using a JavaScript source script.

```json
{
    "name": "virtual_field",
    "type": "VIRTUAL",
    "script": {
        "lang": "js"
        "source": "field1 + field2"
    }
}
```

--------------------------------

### Get Live Settings with REST API

Source: https://context7.com/yelp/nrtsearch/llms.txt

Retrieve the current live settings for a specified index using the REST API.

```bash
curl -XGET localhost:8080/v2/live_settings/testIdx
```

--------------------------------

### Configure an Integer Field for Search

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/numeric.md

Example configuration for an integer field named 'int_field' that should be indexed for search and store doc values.

```json
{
    "name": "int_field",
    "type": "INT",
    "search": true,
    "storeDocValues": true
}
```

--------------------------------

### Configure Date Time Field via JSON

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/date_time.md

Example JSON object for initializing a date time field with epoch_millis format.

```json
{
    "name": "date_time_field",
    "type": "DATE_TIME",
    "search": true,
    "storeDocValues": true,
    "dateTimeFormat": "epoch_millis"
}
```

--------------------------------

### Example Query Vector

Source: https://github.com/yelp/nrtsearch/blob/main/docs/vector_search.md

A vector representation of a user's textual query, used to find similar documents.

```default
[1.0, 3.2, 1.3]
```

--------------------------------

### Vector Search (KNN) Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Execute K-nearest neighbor searches using 'knn' to find documents with embeddings similar to a query vector. 'num_candidates' influences accuracy and performance.

```json
{
  "indexName": "vector_index",
  "startHit": 0,
  "topHits": 10,
  "retrieveFields": ["doc_id", "title", "caption"],
  "knn": [
    {
      "field": "embeddings",
      "k": 10,
      "num_candidates": 100,
      "query_vector": [0.188423157, 0.246743672, 0.14576434, 0.8234, 0.1234]
    }
  ]
}
```

--------------------------------

### Configure Object Field JSON

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/object.md

Example JSON configuration for an object field containing TEXT and INT child fields.

```json
{
    "name": "object_field",
    "type": "OBJECT",
    "storeDocValues": true,
    "childFields": [
        {
            "name": "field1",
            "type": "TEXT",
            "search": true,
        },
        {
            "name": "field2",
            "type": "INT",
            "search": true,
        }
    ]
}
```

--------------------------------

### CSV Input for Document Loading

Source: https://github.com/yelp/nrtsearch/blob/main/docs/vector_search.md

Example CSV format for loading documents with vector fields using a client. Ensure vector embeddings are properly quoted.

```csv
photo_id,photo_embeddings
1,"[0.188423157, 0.246743672, …]"
2,"[0.188423157, 0.246743672, …]"
3,"[0.188423157, 0.246743672, …]"
```

--------------------------------

### POST /v1/create_index

Source: https://context7.com/yelp/nrtsearch/llms.txt

Creates a new index on the nrtSearch server. The index is created in a stopped state by default unless the start parameter is set to true.

```APIDOC
## POST /v1/create_index

### Description
Creates a new index on the nrtSearch server.

### Method
POST

### Endpoint
/v1/create_index

### Request Body
- **indexName** (string) - Required - The name of the index to create.
- **settings** (object) - Optional - Initial offline settings for the index.
- **liveSettings** (object) - Optional - Initial live settings for the index.
- **fields** (array) - Optional - List of field definitions to register.
- **start** (boolean) - Optional - Whether to start the index immediately.

### Request Example
{
  "indexName": "products",
  "settings": {
    "directory": "MMapDirectory"
  },
  "fields": [
    {"name": "product_id", "type": "ATOM"}
  ],
  "start": true
}
```

--------------------------------

### Vector Search with Filter Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Combine vector search with a 'filter' to narrow down results based on specific criteria. 'boost' can adjust the relevance of the vector search component.

```json
{
  "indexName": "vector_index",
  "topHits": 10,
  "retrieveFields": ["doc_id", "category"],
  "knn": [
    {
      "field": "embeddings",
      "k": 10,
      "num_candidates": 200,
      "filter": {
        "termQuery": {
          "field": "category",
          "textValue": "photos"
        }
      },
      "query_vector": [0.188423157, 0.246743672, 0.14576434],
      "boost": 1.5
    }
  ]
}
```

--------------------------------

### Run Yelp Reviews Indexing Test

Source: https://github.com/yelp/nrtsearch/blob/main/README.md

Execute this command to clean, install, and run performance tests for Yelp reviews indexing. It specifically targets the YelpReviewsTest.runYelpReviews method and provides detailed information.

```bash
./gradlew clean installDist :test -PincludePerfTests=* --tests "com.yelp.nrtsearch.yelp_reviews.YelpReviewsTest.runYelpReviews" --info
```

--------------------------------

### Create Index

Source: https://github.com/yelp/nrtsearch/wiki/Home

Initialize a new index with a specified name and root directory.

```bash
./build/install/platypus/bin/lucene-client createIndex --indexName  testIdx --rootDir testIdx
```

--------------------------------

### Generate API Documentation

Source: https://github.com/yelp/nrtsearch/wiki/Home

Generate HTML documentation from proto files using Gradle.

```bash
./gradlew buildDocs
```

--------------------------------

### Configure Boolean Field in JSON

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/boolean.md

Example configuration for a boolean field using JSON.

```json
{
    "name": "bool_field",
    "type": "BOOLEAN",
    "search": true,
    "storeDocValues": true
}
```

--------------------------------

### Configure Live Settings with CLI

Source: https://context7.com/yelp/nrtsearch/llms.txt

Configure live index settings using the CLI and a JSON file. These settings can be changed while the index is running.

```bash
./build/install/nrtsearch/bin/nrtsearch_client liveSettings -f liveSettings.json
```

--------------------------------

### Create Index with CLI

Source: https://context7.com/yelp/nrtsearch/llms.txt

Use the CLI to create a new index on the nrtSearch server. The index is created in a stopped state by default.

```bash
./build/install/nrtsearch/bin/nrtsearch_client createIndex --indexName testIdx
```

--------------------------------

### Example Vector Embeddings

Source: https://github.com/yelp/nrtsearch/blob/main/docs/vector_search.md

Represents a list of float numbers used to encode complex data for vector search.

```default
[[1.0, 0.3, 2.1], [0.01, 0.02, 1.24], ...]
```

--------------------------------

### Create Index

Source: https://github.com/yelp/nrtsearch/blob/main/README.md

Creates a new index using the client or REST API.

```bash
./build/install/nrtsearch/bin/nrtsearch_client createIndex --indexName  testIdx
```

```bash
curl -XPOST localhost:<REST_PORT>/v1/create_index -d '{"indexName": "testIdx"}'
```

--------------------------------

### Configure ID Field in JSON

Source: https://github.com/yelp/nrtsearch/blob/main/docs/field_types/id.md

Example configuration for an _ID field, requiring either store or storeDocValues to be true.

```json
{
    "name": "id_field",
    "type": "_ID",
    "storeDocValues": true
}
```

--------------------------------

### Create and Manage Snapshots

Source: https://context7.com/yelp/nrtsearch/llms.txt

Create point-in-time snapshots for backup and transactional purposes. Supports creating snapshots, backing up to S3, releasing snapshots, and retrieving snapshot generation information.

```bash
# Create snapshot
curl -XPOST localhost:8080/v1/create_snapshot -d '{
  "indexName": "testIdx",
  "openSearcher": true
}'
```

```bash
# Backup to S3
./build/install/nrtsearch/bin/nrtsearch_client backupIndex \
  --indexName testIdx \
  --serviceName my-service \
  --resourceName testIdx
```

```bash
# Release snapshot
curl -XPOST localhost:8080/v1/release_snapshot -d '{
  "indexName": "testIdx",
  "snapshotId": {
    "indexGen": 12345,
    "stateGen": 67890
  }
}'
```

```bash
# Get all snapshot generations
curl -XGET localhost:8080/v1/get_all_snapshot_index_gen/testIdx
```

--------------------------------

### Get Index Settings with REST API

Source: https://context7.com/yelp/nrtsearch/llms.txt

Retrieve the current settings for a specified index using the REST API.

```bash
curl -XGET localhost:8080/v2/settings/testIdx
```

--------------------------------

### Build gRPC Gateway

Source: https://github.com/yelp/nrtsearch/blob/main/README.md

Builds the gRPC gateway component.

```bash
./gradlew buildGrpcGateway
```

--------------------------------

### Get All Businesses (No InnerHit)

Source: https://github.com/yelp/nrtsearch/blob/main/docs/inner_hit.md

This query retrieves all parent documents, focusing only on the 'business_name' field. No nested document retrieval is involved.

```json
{
  "indexName": "index_alpha",
  "retrieveFields": ["business_name"]
}
```

--------------------------------

### Configure NRTsearch Server

Source: https://context7.com/yelp/nrtsearch/llms.txt

Set up the nrtSearch server using a YAML configuration file. This includes defining node names, ports, directories, thread pool configurations, remote storage settings, state backend, warmer, preloading, and plugins.

```yaml
# server_configuration.yaml
nodeName: "nrtsearch_primary"
hostName: "primary-node"
port: "8000"
replicationPort: "8001"
stateDir: "/data/state"
indexDir: "/data/index"

# Thread pool configuration
threadPoolConfiguration:
  search:
    maxThreads: 16
    maxBufferedItems: 2000
    useVirtualThreads: false
  index:
    maxThreads: 8
    useVirtualThreads: true
  fetch:
    maxThreads: 4

# Remote storage (S3)
bucketName: "nrtsearch-bucket"
serviceName: "my-nrtsearch-service"
botoCfgPath: "/etc/boto.cfg"

# State backend
stateConfig:
  backendType: "REMOTE"

# Warmer configuration
warmer:
  maxWarmingQueries: 1000
  warmingParallelism: 4
  warmOnStartup: true

# Index preloading
preload:
  enabled: true
  extensions: ["*"]

# Plugins
plugins:
  - "my-custom-plugin"
pluginSearchPath: "/opt/plugins"

```

--------------------------------

### Vector-Only Search Request

Source: https://github.com/yelp/nrtsearch/blob/main/docs/vector_search.md

Example JSON request for performing a vector-only KNN search. Specify the field, k, num_candidates, and the query_vector.

```default
{
  "indexName": "vector_test",
  "startHit": 0,
  "topHits": 10,
  "timeoutSec": 0,
  "retrieveFields": ["photo_id", "business_id", "caption"],
  "knn": [
    {
      "field": "photo_embeddings",
      "k": 3,
      "num_candidates": 1000,
      "query_vector": [0.188423157, 0.246743672, ...]
    }
  ]
}
```

--------------------------------

### Update Index Settings

Source: https://github.com/yelp/nrtsearch/wiki/Home

Configure index settings using a JSON file.

```bash
./build/install/platypus/bin/lucene-client settings -f settings.json
cat settings.json
{             "indexName": "testIdx",
              "indexVerbose": false,
              "directory": "MMapDirectory",
              "nrtCachingDirectoryMaxSizeMB": 0.0,
              "indexMergeSchedulerAutoThrottle": false,
              "concurrentMergeSchedulerMaxMergeCount": 16,
              "concurrentMergeSchedulerMaxThreadCount": 8
}
```

--------------------------------

### Configure Live Settings with REST API v2

Source: https://context7.com/yelp/nrtsearch/llms.txt

Use the v2 REST API to configure live index settings. These settings control refresh intervals, search parallelism, and merge policies.

```bash
curl -XPOST localhost:8080/v2/live_settings -d '{
  "indexName": "testIdx",
  "liveSettings": {
    "maxRefreshSec": {"value": 1.0},
    "minRefreshSec": {"value": 0.05},
    "maxSearcherAgeSec": {"value": 60.0},
    "indexRamBufferSizeMB": {"value": 256.0},
    "addDocumentsMaxBufferLen": {"value": 100},
    "sliceMaxDocs": {"value": 250000},
    "sliceMaxSegments": {"value": 5},
    "virtualShards": {"value": 4},
    "maxMergedSegmentMB": {"value": 5120},
    "segmentsPerTier": {"value": 10},
    "defaultSearchTimeoutSec": {"value": 30.0}
  }
}'
```

--------------------------------

### Add Documents to Primary

Source: https://github.com/yelp/nrtsearch/blob/main/docs/docker_compose.md

Index documents into the primary node to trigger replication.

```shell
shell% docker exec -it $PRIMARY_CONTAINER_ID sh
# ./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 addDocuments -i testIdx -f docker-compose-config/docs.csv -t csv
```

--------------------------------

### Geo Radius Query Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Filter documents based on geographic proximity using 'geoRadiusQuery'. Specify latitude, longitude, and radius for the search.

```json
{
  "indexName": "testIdx",
  "topHits": 50,
  "retrieveFields": ["doc_id", "name", "location"],
  "query": {
    "booleanQuery": {
      "clauses": [
        {
          "occur": "FILTER",
          "query": {
            "geoRadiusQuery": {
              "field": "location",
              "lat": 37.7749,
              "lon": -122.4194,
              "radius": "10 km"
            }
          }
        }
      ]
    }
  }
}
```

--------------------------------

### Nested Query Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Query nested child documents within parent documents using 'nestedQuery'. 'scoreMode' determines how child scores are aggregated.

```json
{
  "indexName": "restaurants",
  "topHits": 10,
  "retrieveFields": ["business_name"],
  "query": {
    "nestedQuery": {
      "path": "menu",
      "scoreMode": "MAX",
      "query": {
        "booleanQuery": {
          "clauses": [
            {
              "occur": "MUST",
              "query": {
                "matchQuery": {
                  "field": "menu.food_name",
                  "query": "burger"
                }
              }
            },
            {
              "occur": "FILTER",
              "query": {
                "rangeQuery": {
                  "field": "menu.price",
                  "upper": "15.0"
                }
              }
            }
          ]
        }
      }
    }
  }
}
```

--------------------------------

### Create Index with REST API

Source: https://context7.com/yelp/nrtsearch/llms.txt

Use the REST API to create a new index. You can specify initial settings and fields during creation.

```bash
curl -XPOST localhost:8080/v1/create_index -d '{
  "indexName": "testIdx"
}'
```

```bash
curl -XPOST localhost:8080/v1/create_index -d '{
  "indexName": "products",
  "settings": {
    "directory": "MMapDirectory",
    "nrtCachingDirectoryMaxSizeMB": 60.0,
    "concurrentMergeSchedulerMaxThreadCount": 8
  },
  "liveSettings": {
    "maxRefreshSec": 1.0,
    "minRefreshSec": 0.05,
    "indexRamBufferSizeMB": 256.0
  },
  "fields": [
    {"name": "product_id", "type": "ATOM", "storeDocValues": true},
    {"name": "title", "type": "TEXT", "search": true, "store": true}
  ],
  "start": true
}'
```

--------------------------------

### Retrieve Index Statistics

Source: https://context7.com/yelp/nrtsearch/llms.txt

Get statistics for an index, including document counts, size, and searcher information. Available via CLI or REST API.

```bash
# Using CLI
./build/install/nrtsearch/bin/nrtsearch_client stats --indexName testIdx
```

```bash
# Using REST API
curl -XGET localhost:8080/v1/stats/testIdx
```

```json
# Response
# {
#   "maxDoc": 10000,
#   "numDocs": 9500,
#   "dirSize": 52428800,
#   "state": "STARTED",
#   "currentSearcher": {
#     "version": 123456,
#     "numDocs": 9500,
#     "numSegments": 5
#   }
# }
```

--------------------------------

### Geo Bounding Box Query Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Perform a 'geoBoundingBoxQuery' to find documents within a specified geographic rectangular area defined by top-left and bottom-right coordinates.

```json
{
  "indexName": "testIdx",
  "topHits": 50,
  "query": {
    "geoBoundingBoxQuery": {
      "field": "location",
      "topLeft": {"latitude": 37.8, "longitude": -122.5},
      "bottomRight": {"latitude": 37.7, "longitude": -122.3}
    }
  }
}
```

--------------------------------

### POST /v2/live_settings

Source: https://context7.com/yelp/nrtsearch/llms.txt

Configures index settings that can be changed while the index is running, such as refresh intervals and search parallelism.

```APIDOC
## POST /v2/live_settings

### Description
Updates live settings for a running index.

### Method
POST

### Endpoint
/v2/live_settings

### Request Body
- **indexName** (string) - Required - The name of the index.
- **liveSettings** (object) - Required - The live settings to update.

### Request Example
{
  "indexName": "testIdx",
  "liveSettings": {
    "maxRefreshSec": {"value": 1.0}
  }
}
```

--------------------------------

### Phrase Query Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Employ phrase queries to find documents containing a specific sequence of terms. The 'slop' parameter allows for flexibility in term proximity.

```json
{
  "indexName": "testIdx",
  "topHits": 10,
  "retrieveFields": ["doc_id", "title", "description"],
  "query": {
    "phraseQuery": {
      "field": "description",
      "terms": ["high", "quality", "product"],
      "slop": 2
    }
  }
}
```

--------------------------------

### Perform search with custom highlighting options

Source: https://github.com/yelp/nrtsearch/blob/main/docs/highlighting.md

Configures custom tags, fragment sizes, and specific highlight queries for the search response.

```json
{
  "indexName": "test_index",
  "topHits": 2,
  "query": {
    "matchQuery": {
      "field": "comment",
      "query": "food"
    }
  },
  "highlight": {
    "fields": ["comment"],
    "fieldSettings": {
      "comment": {
        "preTags": ["<START>"],
        "postTags": ["<END>"],
        "fragmentSize": 18,
        "maxNumberOfFragments": 3,
        "highlightQuery": {
          "matchQuery": {
            "field": "comment",
            "query": "food is good"
          }
        }
      }
    }
  }
}
```

```json
{
  "hits": [{
    "highlights": {
      "comment": {
        "fragments": ["the <START>food<END> here <START>is<END> amazing", "service was <START>good<END>"]
      }
    }
  }, {
    "highlights": {
      "comment": {
        "fragments": ["The <START>food<END> here <START>is<END> pretty", "This <START>is<END> my first time", "pretty <START>good<END>, the service"]
      }
    }
  }]
}
```

--------------------------------

### Configure Index Settings with CLI

Source: https://context7.com/yelp/nrtsearch/llms.txt

Configure offline index settings using the CLI and a JSON file. These settings can only be changed while the index is stopped.

```bash
./build/install/nrtsearch/bin/nrtsearch_client settings -f settings.json
```

--------------------------------

### Hybrid Search (Vector + Text) Example

Source: https://context7.com/yelp/nrtsearch/llms.txt

Perform hybrid search by combining a traditional text query ('matchQuery') with a vector search ('knn'). This allows for both semantic and keyword-based matching.

```json
{
  "indexName": "vector_index",
  "topHits": 10,
  "retrieveFields": ["doc_id", "title", "description"],
  "query": {
    "matchQuery": {
      "field": "title",
      "query": "sunset beach"
    }
  },
  "knn": [
    {
      "field": "embeddings",
      "k": 50,
      "num_candidates": 500,
      "query_vector": [0.188423157, 0.246743672, 0.14576434]
    }
  ]
}
```

--------------------------------

### Initialize SpanMultiTermQuery in Java

Source: https://github.com/yelp/nrtsearch/blob/main/docs/queries/span_multi_term_query_wrapper.md

Wraps a WildcardQuery into a SpanMultiTermQuery to enable span-based query nesting.

```java
WildcardQuery wildcard = new WildcardQuery(new Term(“field”, “bro?n”));
SpanQuery spanWildcard = new SpanMultiTermQuery<WildcardQuery>(wildcard);
// do something with spanWildcard, such as use it in a SpanFirstQuery
```

--------------------------------

### Register Fields

Source: https://github.com/yelp/nrtsearch/wiki/Home

Define index fields and their properties via a JSON configuration.

```bash
./build/install/platypus/bin/lucene-client registerFields -f registerFields.json
cat registerFields.json
{             "indexName": "testIdx",
              "field":
              [
                      { "name": "vendor_name", "type": "TEXT" , "search": true, "store": true, "tokenize": true},
                      { "name": "license_no",  "type": "INT", "multiValued": true, "storeDocValues": true}
              ]
}
```

--------------------------------

### Match All Query Proto Definition

Source: https://github.com/yelp/nrtsearch/blob/main/docs/queries/match_all.md

The protocol buffer definition for the Match All Query.

```protobuf
message MatchAllQuery {
}
```

--------------------------------

### Term Query Examples

Source: https://context7.com/yelp/nrtsearch/llms.txt

Use term queries to match exact values in text, numeric, or boolean fields. Ensure the field type matches the value type provided.

```json
{
  "indexName": "testIdx",
  "topHits": 50,
  "retrieveFields": ["doc_id", "title", "category", "status"],
  "query": {
    "booleanQuery": {
      "clauses": [
        {
          "occur": "MUST",
          "query": {
            "termQuery": {
              "field": "category",
              "textValue": "electronics"
            }
          }
        },
        {
          "occur": "MUST",
          "query": {
            "termQuery": {
              "field": "quantity",
              "intValue": 100
            }
          }
        },
        {
          "occur": "MUST",
          "query": {
            "termQuery": {
              "field": "is_active",
              "booleanValue": true
            }
          }
        }
      ]
    }
  }
}
```

--------------------------------

### Restart and Restore Index from S3

Source: https://github.com/yelp/nrtsearch/blob/main/docs/docker_compose.md

Restart nrtSearch containers and initiate index restoration from S3. This involves downing containers, rebuilding images, bringing them up, and then starting the index with the restoration configuration.

```shell
docker-compose down
sehll% docker images | grep nrtsearch | awk '{print "docker rmi -f "$3}' | sh  # need to rebuild the images with the new config
docker-compose -f docker-compose.yaml up
PRIMARY_CONTAINER_ID=$(docker ps | grep nrtsearch_primary-node | awk '{print $1}')
docker exec -it $PRIMARY_CONTAINER_ID sh
# ./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 startIndex -f docker-compose-config/startIndex_primary.json
# ./build/install/nrtsearch/bin/nrtsearch_client -h primary-node -p 8000 search -f docker-compose-config/search.json
...
fields {
  key: "license_no"
  value {
    fieldValue {
      intValue: 111
    }
    fieldValue {
      intValue: 222
    }
  }
}
```
Tokens:
Raw