Monstache

https://rwynn.github.io/monstache-site
Admin
Monstache is a sync daemon written in Go that continuously indexes MongoDB collections into...
Tokens:27,199
Snippets:360
Trust Score:9.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Highlight Response Example

Source: https://rwynn.github.io/monstache-site/advanced

Example of the JSON response containing highlighted search terms.

```json
"hits" : [ { 
    "highlight" : { 
        "file.content" : [ "I like to program in <em>golang</em>.\n\n" ] 
    } 
} ]

```

--------------------------------

### Insert MongoDB documents for routing example

Source: https://rwynn.github.io/monstache-site/advanced

Example MongoDB collection setup for comments associated with a post identifier.

```javascript
use blog;
db.comments.insert({title: "Did you read this?", post_id: "123"});
db.comments.insert({title: "Yeah, it's good", post_id: "123"});

```

--------------------------------

### Elasticsearch Highlight Example

Source: https://rwynn.github.io/monstache-site/advanced

Example of Elasticsearch highlight output, showing content with highlighted terms.

```json
"hits" : [{
    "highlight" : {
        "attachment.content" : [ "I like to program in <em>golang</em>." ]
    }
}]


```

--------------------------------

### Example Golang Map Plugin

Source: https://rwynn.github.io/monstache-site/advanced

This example plugin converts all top-level string values within a document to uppercase. It demonstrates how to access and modify the document within the `Map` function.

```go
package main
import (
    "github.com/rwynn/monstache/monstachemap"
    "strings"
)
// a plugin to convert document values to uppercase
func Map(input *monstachemap.MapperPluginInput) (output *monstachemap.MapperPluginOutput, err error) {
    doc := input.Document
    for k, v := range doc {
        switch v.(type) {
        case string:
            doc[k] = strings.ToUpper(v.(string))
        }
    }
    output = &monstachemap.MapperPluginOutput{Document: doc}
    return
}
```

--------------------------------

### Start Monstache Service

Source: https://rwynn.github.io/monstache-site/advanced

Execute the Monstache process with verbose logging enabled.

```bash
monstache -verbose -f CONFIG.toml

```

--------------------------------

### Build Monstache from Development Version

Source: https://rwynn.github.io/monstache-site/about

Install the latest development version of Monstache using Git and Go modules. Ensure you have Go 1.11 or later installed.

```bash
cd ~/build # somewhere outside your $GOPATH
git clone https://github.com/rwynn/monstache.git
cd monstache
git checkout <branch-to-build>
go install
# the resulting monstache binary will be in $GOPATH/bin
```

--------------------------------

### MongoDB Document Examples

Source: https://rwynn.github.io/monstache-site/advanced

These are example documents from a MongoDB collection.

```json
{ "_id": "london", "type": "branch", "name": "London Westminster", "city": "London", "country": "UK" }
```

```json
{ "_id": "alice", "type": "employee", "name": "Alice Smith", "branch": "london" }
```

--------------------------------

### Verify Monstache Installation

Source: https://rwynn.github.io/monstache-site/start

Run this command after installation to check the Monstache version. Ensure the binary is in your system's PATH.

```bash
monstache -v
# 6.7.4
```

--------------------------------

### Monstache High Availability and Worker Configuration

Source: https://rwynn.github.io/monstache-site/advanced

Example TOML configuration for workers and command-line arguments for starting Monstache in a high-availability cluster with distributed workers.

```toml
// config.toml
workers = ["Tom", "Dick", "Harry"]


```

--------------------------------

### Sort and Project Documents

Source: https://rwynn.github.io/monstache-site/advanced

Example of sorting documents in the current collection and projecting specific fields.

```toml
[[script]]
namespace = "test.test"
script = """
module.exports = function(doc) {
    doc.twoAgesSortedByName = find({}, {
            sort: ["name"],
            limit: 2,
            select: {
              age: 1
            }
    });
    return doc;
}
"""
```

--------------------------------

### Run Monstache Docker with Golang Plugin

Source: https://rwynn.github.io/monstache-site/advanced

Example of running Monstache via Docker, mounting a local Golang plugin into the container and specifying its path.

```bash
docker run --rm --net=host -v ~/plugin:/tmp/plugin rwynn/monstache:6.7.4 -mapper-plugin-path /tmp/plugin/plugin.so
```

--------------------------------

### Run Monstache Process

Source: https://rwynn.github.io/monstache-site/about

Start a Monstache process in a terminal. This is a prerequisite for running the tests locally without Docker.

```bash
monstache -verbose
```

--------------------------------

### Configure Pipelines and Scripts in Monstache

Source: https://rwynn.github.io/monstache-site/about

Example configuration for defining aggregation pipelines and custom scripts to process documents during synchronization.

```toml
direct-read-namespaces = [test.test]
change-stream-namespaces = [test.test]
[[pipeline]]
script = """
module.exports = function(ns, changeStream) {
  if (changeStream) {
    return [
      { $match: {"fullDocument.foo": 1} }
    ];
  } else {
    return [
      { $match: {"foo": 1} }
    ];
  }
}
"""
[[script]]
namespace = "test.test"
script = """
module.exports = function(doc, ns) {
  doc.extra = pipe([
    { $match: {foo: 1} },
    { $limit: 1 },
    { $project: { _id: 0, foo: 1}}
  ]);
  return doc;
}
"""
```

--------------------------------

### Run Aggregation Pipeline

Source: https://rwynn.github.io/monstache-site/advanced

Example of using the pipe function to execute an aggregation pipeline and store results in a document field.

```toml
[[script]]
namespace = "test.test"
script = """
module.exports = function(doc, ns) {
  doc.extra = pipe([
    { $match: {foo: 1} },
    { $limit: 1 },
    { $project: { _id: 0, foo: 1}}
  ]
  // optional , { database: "foo", collection: "bar"} // defaults to same namespace
  );
  return doc;
}
"""
```

--------------------------------

### Monstache TOML Configuration Example

Source: https://rwynn.github.io/monstache-site/start

This TOML configuration demonstrates essential settings for Monstache, including MongoDB and Elasticsearch connection URLs, direct read namespaces for initial seeding, and change stream namespaces for real-time synchronization.

```toml
# connection settings

# connect to MongoDB using the following URL
mongo-url = "mongodb://someuser:password@localhost:40001"
# connect to the Elasticsearch REST API at the following node URLs
elasticsearch-urls = ["https://es1:9200", "https://es2:9200"]

# frequently required settings

# if you need to seed an index from a collection and not just listen and sync changes events
# you can copy entire collections or views from MongoDB to Elasticsearch
direct-read-namespaces = ["mydb.mycollection", "db.collection", "test.test", "db2.myview"]

# if you want to use MongoDB change streams instead of legacy oplog tailing use change-stream-namespaces
# change streams require at least MongoDB API 3.6+
# if you have MongoDB 4+ you can listen for changes to an entire database or entire deployment
# in this case you usually don't need regexes in your config to filter collections unless you target the deployment.
# to listen to an entire db use only the database name.  For a deployment use an empty string.
change-stream-namespaces = ["mydb.mycollection", "db.collection", "test.test"]

# additional settings

# if you don't want to listen for changes to all collections in MongoDB but only a few
# e.g. only listen for inserts, updates, deletes, and drops from mydb.mycollection
# this setting does not initiate a copy, it is only a filter on the change event listener
namespace-regex = '^mydb\.mycollection$'
# compress requests to Elasticsearch
gzip = true
# generate indexing statistics
stats = true
# index statistics into Elasticsearch
index-stats = true
```

--------------------------------

### Monstache Systemd Service Configuration

Source: https://rwynn.github.io/monstache-site/advanced

Example systemd service file for running Monstache as a background service. It includes settings for description, execution, watchdog, and restart behavior.

```ini
[Unit]
Description=monstache sync service

[Service]
Type=notify
ExecStart=/usr/local/bin/monstache -f /etc/monstache/config.toml
WatchdogSec=30s
Restart=always

[Install]
WantedBy=multi-user.target
```

--------------------------------

### Start Monstache Worker Processes

Source: https://rwynn.github.io/monstache-site/advanced

Launch Monstache processes, assigning each to a specific worker name for distributed task handling.

```bash
monstache -f config.toml -worker Tom
monstache -f config.toml -worker Dick
monstache -f config.toml -worker Harry


```

--------------------------------

### Configure Elasticsearch Ingest Pipeline

Source: https://rwynn.github.io/monstache-site/config

This example shows how to configure an Elasticsearch ingest pipeline for extracting file information when `index-files` is enabled. The pipeline must be created before running Monstache if `disable-file-pipeline-put` is true.

```json
PUT _ingest/pipeline/attachment
{
  "description" : "Extract file information",
  "processors" : [
    {
      "attachment" : {
        "field" : "file",
        "indexed_chars" : -1
      }
    }
  ]
}

```

--------------------------------

### Start Monstache HA Cluster with Workers

Source: https://rwynn.github.io/monstache-site/advanced

Initiate Monstache processes across multiple hosts for high availability, each assigned a worker role.

```bash
// on host A
monstache -cluster-name HA -worker Tom -f config.toml
monstache -cluster-name HA -worker Dick -f config.toml
monstache -cluster-name HA -worker Harry -f config.toml

// on host B
monstache -cluster-name HA -worker Tom -f config.toml
monstache -cluster-name HA -worker Dick -f config.toml
monstache -cluster-name HA -worker Harry -f config.toml

// on host C
monstache -cluster-name HA -worker Tom -f config.toml
monstache -cluster-name HA -worker Dick -f config.toml
monstache -cluster-name HA -worker Harry -f config.toml


```

--------------------------------

### Configure Elasticsearch Thread Pool Bulk Queue Size

Source: https://rwynn.github.io/monstache-site/config

This example demonstrates how to configure the Elasticsearch thread pool's bulk queue size. Adjusting this value can help prevent bulk indexing failures when dealing with high concurrency.

```yaml
thread_pool:
    bulk:
    queue_size: 200

```

--------------------------------

### Query Elasticsearch with routing parameter

Source: https://rwynn.github.io/monstache-site/advanced

Example cURL request to search for documents on a specific shard using the routing parameter.

```bash
$ curl -H "Content-Type:application/json" -XGET 'http://localhost:9200/blog.comments/_search?routing=123' -d '
{
   "query":{
      "match_all":{}
   }
}'

```

--------------------------------

### Define a global document mapping script

Source: https://rwynn.github.io/monstache-site/advanced

Assign a function to module.exports to manipulate documents before indexing. This example omits specific fields from the document.

```javascript
module.exports = function(doc, ns, updateDesc) {
    // the doc namespace e.g. test.test is passed as the 2nd arg
        // if available, an object containing the update description is passed as the 3rd arg
    return _.omit(doc, "password", "secret");
}
```

--------------------------------

### Elasticsearch Document After Update

Source: https://rwynn.github.io/monstache-site/advanced

This example shows the Elasticsearch document after an update operation. A new merge patch has been added, reflecting the changes and preserving history.

```json
_source : {
  "age" : 21,
  "friends" : [
    1,
    3
  ],
  "json-merge-patches" : [
    {
      "p" : "{\"age\":16,\"friends\":[1,2,3],\"name\":\"Joe\"}",
      "ts" : 1487263414,
      "v" : 1
    },
    {
      "p" : "{\"age\":21,\"friends\":[1,3]}",
      "ts" : 1487263746,
      "v" : 2
    }
  ],
  "name" : "Joe"
}
```

--------------------------------

### Elasticsearch Document with Merge Patch

Source: https://rwynn.github.io/monstache-site/advanced

This is an example of how a document appears in Elasticsearch after an insert operation when merge patches are enabled. It includes the original source and a timestamped merge patch.

```json
_source : {
  "age" : 16,
  "friends" : [
    1,
    2,
    3
  ],
  "json-merge-patches" : [
    {
      "p" : "{\"age\":16,\"friends\":[1,2,3],\"name\":\"Joe\"}",
      "ts" : 1487263414,
      "v" : 1
    }
  ],
  "name" : "Joe"
}
```

--------------------------------

### Enable Templating for Configuration Files

Source: https://rwynn.github.io/monstache-site/config

When enabled, configuration files passed with -f are interpreted as Go templates. This allows dynamic configuration using environment variables. Environment variables are accessible within the template via the `.` symbol and the `index` function.

```go
[[script]]
namespace = "mydb.mycollection"
script = """
module.exports = function(doc) {
    if ( doc.score > {{index . "THRESHOLD"}} ) {
      doc.important = true;
    }
    return doc;
}
"""

```

--------------------------------

### Build Monstache from Source

Source: https://rwynn.github.io/monstache-site/start

Clone the Monstache repository and build the binary using Go. This method is useful if you need to customize or develop plugins.

```bash
cd ~/build # somewhere outside your $GOPATH
git clone https://github.com/rwynn/monstache.git
cd monstache
git checkout <branch-or-tag-to-build>
go install
# monstache binary should now be in $GOPATH/bin
```

--------------------------------

### Configure index-as-update settings

Source: https://rwynn.github.io/monstache-site/config

Settings to mitigate serialization issues when index-as-update is enabled by limiting connections and increasing buffer capacity.

```toml
elasticsearch-max-conns = 1

[gtm-settings]
buffer-size = 2048
buffer-duration = 4s
```

--------------------------------

### View Generated Elasticsearch Requests

Source: https://rwynn.github.io/monstache-site/advanced

Example of the bulk index requests generated by Monstache for the parent and child documents.

```json
{"index":{"_id":"5a84a8b826993bde57c12893","_index":"test.test","_type":"_doc","routing":"5a84a8b826993bde57c12893","version":6522523668566769665,"version_type":"external"}}
{"my_join_field":{"name":"question"},"text":"This is a question","type":"question"}

{"index":{"_id":"5a84a92b26993bde57c12894","_index":"test.test","_type":"_doc","routing":"5a84a8b826993bde57c12893","version":6522524162488008705,"version_type":"external"}}
{"my_join_field":{"name":"answer","parent":"5a84a8b826993bde57c12893"},"question":"5a84a8b826993bde57c12893","text":"This is an answer","type":"answer"}

```

--------------------------------

### Initialize Elasticsearch Mapping (Pre-v5)

Source: https://rwynn.github.io/monstache-site/advanced

REST commands to set up attachment mappings for Elasticsearch versions prior to 5.

```json
POST /users.fs.files
{
  "mappings": {
    "fs.files": {
      "properties": {
    "file": { "type": "attachment" }
}}}}

POST /posts.fs.files
{
  "mappings": {
    "fs.files": {
      "properties": {
    "file": { "type": "attachment" }
}}}}

```

--------------------------------

### Replace Reference ID with Document

Source: https://rwynn.github.io/monstache-site/advanced

Example of replacing an author reference ID with the actual user document from a different collection.

```toml
[[script]]
namespace = "test.posts"
script = """
module.exports = function(post) {
    if (post.author) { // author is a an object id reference
        post.author = findId(post.author, {
          database: "test",
          collection: "users"
        });
    }
    return post;
}
"""
```

--------------------------------

### Disable Indexing for Merge Patches

Source: https://rwynn.github.io/monstache-site/advanced

An Elasticsearch index template example to disable indexing for the merge patch attribute, improving search performance.

```json
PUT /_template/test.test
{
    "template" : "test.test",
    "mappings" : {
    "test" : {
        "json-merge-patches" : { "index" : false }
    }
    }
}
```

--------------------------------

### Run Monstache with Configuration File

Source: https://rwynn.github.io/monstache-site/start

Execute Monstache using a specified TOML configuration file. This allows for detailed control over connection settings and synchronization behavior.

```bash
monstache -f /path/to/config.toml
```

--------------------------------

### Compile Golang Plugin

Source: https://rwynn.github.io/monstache-site/advanced

Use the `go build` command with the `-buildmode=plugin` flag to compile your Golang plugin into a shared object (.so) file.

```bash
go build -buildmode=plugin -o myplugin.so myplugin.go
```

--------------------------------

### Initialize Elasticsearch Ingest Pipeline (v5+)

Source: https://rwynn.github.io/monstache-site/advanced

REST command to create an attachment pipeline for Elasticsearch version 5 and above.

```json
PUT /_ingest/pipeline/attachment
{
  "description" : "Extract file information",
  "processors" : [
    {
      "attachment" : {
    "field" : "file"
      }
    }
  ]
}

```

--------------------------------

### Golang Plugin Function Signatures

Source: https://rwynn.github.io/monstache-site/advanced

Implement one or more of these functions in your Golang plugin. Ensure your plugin is built with the exact same source code as the loading program.

```go
func Map(input *monstachemap.MapperPluginInput) (output *monstachemap.MapperPluginOutput, err error)
```

```go
func Filter(input *monstachemap.MapperPluginInput) (keep bool, err error)
```

```go
func Pipeline(ns string, changeStream bool) (stages []interface{}, err error)
```

```go
func Process(input*monstachemap.ProcessPluginInput) error
```

--------------------------------

### Configure Monstache Middleware

Source: https://rwynn.github.io/monstache-site/advanced

Set up the Monstache configuration file to handle document transformation and routing logic.

```toml
[[script]]
namespace = "test.test"
routing = true
script = """
module.exports = function(doc) {
        var routing;
        if (doc.type === "question") {
                routing = doc._id;
                doc.my_join_field = {
                   name: "question"
                }
        } else if (doc.type === "answer") {
                routing = doc.question;
                doc.my_join_field = {
                  name: "answer",
                  parent: routing
                };
        }
        if (routing) {
                doc._meta_monstache = { routing: routing };
        }
        return doc;
}
"""

```

--------------------------------

### Filter Change Stream Events by Field

Source: https://rwynn.github.io/monstache-site/advanced

Use a change stream pipeline to listen for specific field changes in MongoDB 3.6+. This example filters updates to only sync when 'foo' or 'bar' fields change.

```toml
change-stream-namespaces = ["test.test"]

[[pipeline]]
namespace = "test.test"
script = """
module.exports = function(ns, changeStream) {
  if (changeStream) {
    return [
      {
        $match: {
          $or: [
            { "updateDescription": {$exists: false} },
            { "updateDescription.updatedFields.foo": {$exists: true}},
            { "updateDescription.updatedFields.bar": {$exists: true}}
          ]
        }
      }
    ];
  } else {
    return [];
  }
}
"""

```

--------------------------------

### Run Monstache Tests

Source: https://rwynn.github.io/monstache-site/about

Execute the Monstache test suite from the command line. This requires local MongoDB and Elasticsearch servers to be running.

```bash
cd monstache
go test -v
```

--------------------------------

### Run Monstache with Golang Plugin

Source: https://rwynn.github.io/monstache-site/advanced

Execute the Monstache binary, specifying the path to your compiled plugin using the `-mapper-plugin-path` argument.

```bash
$GOPATH/bin/monstache -mapper-plugin-path /path/to/myplugin.so
```

--------------------------------

### Indexing Metadata Configuration

Source: https://rwynn.github.io/monstache-site/advanced

This section details how to override indexing metadata for individual documents by setting the `_meta_monstache` field. It explains the available properties within `_meta_monstache` and provides an example of a Javascript script that configures parent-child relationships and custom index routing in Elasticsearch.

```APIDOC
## Indexing Metadata Configuration

### Description
Override the indexing metadata for an individual document by setting a special field named `_meta_monstache` on the document returned from your Javascript function.

### Properties of `_meta_monstache`

- **routing** (string) - The routing value.
- **index** (string) - The name of the index to use.
- **type** (string) - The document type.
- **parent** (string) - The document parent.
- **version** (string) - The document version.
- **versionType** (string) - The document version type.
- **pipeline** (string) - The name of a pipeline to apply to the document.
- **retryOnConflict** (integer) - Control how retry works on conflicts.
- **skip** (boolean) - Set this boolean to true to skip indexing.
- **id** (string) - Override the ID used to index the document.

### Example Script for Parent-Child Relationship and Custom Indexing

This example demonstrates how to configure parent-child relationships and custom index routing in Elasticsearch using a Javascript script within Monstache.

```javascript
[[script]]
namespace = "test.company"
routing = true
script = """
module.exports = function(doc, ns) {
        // var meta = { type: doc.type, index: 'company' };
    var meta = { type: doc.type, index: ns.split(".")[1] };
    if (doc.type === "employee") {
        meta.parent = doc.branch;
    }
    doc._meta_monstache = meta;
    return _.omit(doc, "branch", "type");
}
"""
```

#### Explanation of the Example:

- **Namespace**: The `namespace` is set to `"test.company"`, indicating the MongoDB database and collection. Monstache will transform this to `company` in Elasticsearch.
- **Routing**: `routing = true` enables document routing.
- **Script Logic**: 
  - The script defines `meta` with the document's `type` and sets the `index` to the collection name extracted from the namespace (`ns.split(".")[1]`).
  - If the document `type` is `"employee"`, the `meta.parent` is set to the `doc.branch` value, establishing a parent-child relationship.
  - The `_meta_monstache` object is assigned to the document.
  - `_.omit(doc, "branch", "type")` removes the `branch` and `type` fields from the document before indexing, as this information is now handled by Elasticsearch's `_parent` and `_type` fields respectively.

This configuration routes documents to the `company` index, uses the document's `type` as the Elasticsearch type, and indexes employees as children of their respective branches.
```

--------------------------------

### Customize index and type with routing

Source: https://rwynn.github.io/monstache-site/advanced

Advanced script configuration to dynamically set index, type, and routing metadata based on document content.

```toml
[[script]]
namespace = "blog.comments"
routing = true
script = """
module.exports = function(doc) {
    if (doc.score >= 100) {
        // NOTE: prefix dynamic index with namespace for proper cleanup on drops
        doc._meta_monstache = { index: "blog.comments.highscore", type: "highScoreComment", routing: doc.post_id };
    } else {
        doc._meta_monstache = { routing: doc.post_id };
    }
    return doc;
}
"""

```

--------------------------------

### Configure document filters

Source: https://rwynn.github.io/monstache-site/advanced

Use filter configurations to ignore documents based on their content. Return true to include or false to ignore.

```toml
[[filter]]
namespace = "db.collection"
script = """
module.exports = function(doc, ns, updateDesc) {
    return !!doc.interesting;
}
"""

[[filter]]
namespace = "db2.collection2"
path = "path/to/script.js"
```

--------------------------------

### Query Options Configuration

Source: https://rwynn.github.io/monstache-site/advanced

Structure of the optional options object used to configure database, collection, projection, sorting, and limits.

```javascript
var options = {
    database: "test",
    collection: "test",
    // to omit _id set the _id key to 0 in select
    select: {
        age: 1
    },
    // only applicable to find...
    sort: ["name"],
    limit: 2
}
```

--------------------------------

### File Namespaces Configuration

Source: https://rwynn.github.io/monstache-site/config

Configure namespaces for indexing files uploaded via GridFS. This is required when `index-files` is enabled.

```APIDOC
## file-namespaces

`[]string` (default `nil`) (env var name `MONSTACHE_FILE_NS`)

The file_namespaces config must be set when index-files is enabled. file_namespaces must be set to an array of MongoDB namespace strings. Files uploaded through gridfs to any of the namespaces in file_namespaces will be retrieved and their raw content indexed into Elasticsearch via either the mapper-attachments or ingest-attachment plugin.

**Note:**
This option may be passed on the command line as `./monstache --file-namespace test.foo --file-namespace test.bar`
If specified as an environment variable the value should be namespaces separated only by the `env-delimiter` which defaults to a comma. E.g. `MONSTACHE_FILE_NS=test.foo,test.bar`
```

--------------------------------

### Configure Monstache for GridFS

Source: https://rwynn.github.io/monstache-site/advanced

Add these settings to your TOML configuration file to enable GridFS indexing.

```toml
index-files = true

direct-read-namespaces = ["users.fs.files", "posts.fs.files"]

file-namespaces = ["users.fs.files", "posts.fs.files"]

file-highlighting = true

```

--------------------------------

### Options Object Configuration

Source: https://rwynn.github.io/monstache-site/advanced

The optional configuration object used by retrieval functions to specify target database, collection, and query modifiers.

```APIDOC
### Options Object Fields
- **database** (string) - Optional - Target database name. Defaults to current document's database.
- **collection** (string) - Optional - Target collection name. Defaults to current document's collection.
- **select** (object) - Optional - Projection object (e.g., {age: 1}).
- **sort** (array) - Optional - Sort criteria (e.g., ["name"]).
- **limit** (number) - Optional - Maximum number of documents to return.
```

--------------------------------

### Add File to GridFS

Source: https://rwynn.github.io/monstache-site/advanced

Use the mongofiles command to upload a file to MongoDB for indexing.

```bash
mongofiles -d users put resume.docx

```

--------------------------------

### Configure Index Template for Bulk Indexing

Source: https://rwynn.github.io/monstache-site/config

Use this index template to optimize indexing throughput by adjusting shard replicas and refresh intervals before running Monstache.

```json
{
  "index_patterns": ["test.*"],
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 0,
    "refresh_interval": "30s"
  }
}
```

--------------------------------

### Monstache Configuration Parameters

Source: https://rwynn.github.io/monstache-site/config

Configuration settings for connecting Monstache to MongoDB and defining indexing behavior.

```APIDOC
## Configuration Parameters

### Parameters
- **type** (string) - Optional - Override the default index type (default: _doc for ES 6.2+).
- **pipeline** (string) - Optional - Name of an existing Elasticsearch pipeline for data transformation.
- **max-file-size** (int) - Optional - Max bytes for GridFS file indexing (default: 0).
- **merge-patch-attribute** (string) - Optional - Property name for merge patches (default: json-merge-patches).
- **mongo-url** (string) - Required - MongoDB connection string (env: MONSTACHE_MONGO_URL).
- **mongo-config-url** (string) - Optional - Config server URL for sharded clusters (env: MONSTACHE_MONGO_CONFIG_URL).
- **mongo-pem-file** (string) - Optional - Path to local certificate for SSL (env: MONSTACHE_MONGO_PEM).
- **mongo-validate-pem-file** (boolean) - Optional - Enable/disable TLS verification (env: MONSTACHE_MONGO_VALIDATE_PEM).
- **mongo-oplog-database-name** (string) - Optional - Database name for oplog (default: local).
- **mongo-oplog-collection-name** (string) - Optional - Collection name for oplog (default: oplog.rs).

### Dial Settings (mgo driver only)
- **ssl** (bool) - Optional - Use TLS for connection.
- **timeout** (int) - Optional - Initial connection timeout in seconds.
- **read-timeout** (int) - Optional - Read timeout in seconds.
- **write-timeout** (int) - Optional - Write timeout in seconds.

### Session Settings (mgo driver only)
- **socket-timeout** (int) - Optional - Socket timeout in seconds.
```

--------------------------------

### GTM Settings Configuration

Source: https://rwynn.github.io/monstache-site/config

Configure settings for the Go Transaction Manager (GTM) to control event processing and buffering.

```APIDOC
## gtm-settings

`TOML table` (default `nil`)

The following gtm configuration properties are available. See gtm for details

### channel-size

`int` (default 512)

Controls the size of the go channels created for processing events. When many events are processed at once a larger channel size may prevent blocking in gtm.

### buffer-size

`int` (default 32)

Determines how many documents are buffered by a gtm worker go routine before they are batch fetched from MongoDB. When many documents are inserted or updated at once it is better to fetch them together.

### buffer-duration

`string` (default 75ms)

A string representation of a golang duration. Determines the maximum time a buffer is held before it is fetched in batch from MongoDB and flushed for indexing.

### max-await-time

`string` (default "")

A string represetation of a golang duration, e.g. "10s". If set, will be converted and passed at the `maxAwaitTimeMS` option for change streams. This determines the maximum amount of time in milliseconds the server waits for new data changes to report to the change stream cursor before returning an empty batch.
```

--------------------------------

### Highlight Matching Terms

Source: https://rwynn.github.io/monstache-site/advanced

Enable highlighting to see where search terms appear in the extracted text.

```bash
curl -H "Content-Type:application/json" localhost:9200/users.fs.files/_search?pretty -d '{
    "fields": ["file.content"],
    "query": {
        "match": {
            "file.content": "golang"
        }
    },
    "highlight": {
        "fields": {
            "file.content": {
            }
        }
    }
}'

```

```bash
curl -H "Content-Type:application/json" localhost:9200/users.fs.files/_search?pretty -d '{
    "_source": ["attachment.content"],
    "query": {
        "match": {
            "attachment.content": "golang"
        }
    },
    "highlight": {
        "fields": {
            "attachment.content": {
            }
        }
    }
}'

```

--------------------------------

### Gzip Compression Configuration

Source: https://rwynn.github.io/monstache-site/config

Enable gzip compression for requests sent to Elasticsearch to reduce network bandwidth.

```APIDOC
## gzip

`boolean` (default `false`)

When gzip is true, monstache will compress requests to Elasticsearch. If you enable gzip in monstache and are using Elasticsearch prior to version 5 you will also need to update the Elasticsearch config file to set http.compression: true. In Elasticsearch version 5 and above http.compression is enabled by default. Enabling gzip compression is recommended if you enable the index-files setting.
```

--------------------------------

### Pipeline Configuration

Source: https://rwynn.github.io/monstache-site/config

Defines a pipeline of aggregation stages to be run on data. This is specified as an array of TOML tables, each containing a namespace and a script.

```APIDOC
## pipeline

### Description
When pipeline is given monstache will call the function specified to determine an array of aggregation pipeline stages to run. See the section Middleware for more information.

### Type
[] array of TOML table

### Default
nil

#### namespace

### Description
The MongoDB namespace, db.collection, to apply the script to. If you omit the namespace the pipeline function with be applied to all namespaces.

### Type
string

### Default
""

#### script

### Description
An inline script. You can use TOML multiline syntax here. The function should take 2 arguments, a namespace and a boolean indicating whether or not the data is a change stream. The function should return an array of aggregation pipeline stages. Note, for change streams the root of the pipeline will be the change event with a field `fullDocument` representing the changed doc. You should alter your pipeline stages according to this boolean. Monstache needs the change event data so do not replace the root of the document in your pipeline for change streams.

### Type
string

### Default
""
```

--------------------------------

### Graylog Address Configuration

Source: https://rwynn.github.io/monstache-site/config

Configure the address of a Graylog server to redirect logs to in GELF format.

```APIDOC
## graylog-addr

`string` (default "") (env var name `MONSTACHE_GRAYLOG_ADDR`)

The address of a graylog server to redirect logs to in GELF
```

--------------------------------

### Monstache Configuration Settings

Source: https://rwynn.github.io/monstache-site/config

Configuration options for controlling Monstache synchronization behavior and Elasticsearch connectivity.

```APIDOC
## Configuration Settings

### direct-read-stateful
- **Type**: boolean (default `false`)
- **Description**: When true, monstache tracks completed direct read namespaces in a `directreads` collection. On restart, it skips namespaces already marked as complete.

### disable-change-events
- **Type**: boolean (default `false`)
- **Description**: When true, monstache will not listen to oplog change events or watch collections.

### disable-file-pipeline-put
- **Type**: boolean (default `false`)
- **Description**: When true, monstache will not auto-create the `attachment` ingest pipeline for GridFS files.

### dropped-databases
- **Type**: boolean (default `true`)
- **Description**: When false, monstache will not delete mapped indexes in Elasticsearch if a MongoDB database is dropped.

### dropped-collections
- **Type**: boolean (default `true`)
- **Description**: When false, monstache will not delete mapped indexes in Elasticsearch if a MongoDB collection is dropped.

### elasticsearch-user
- **Type**: string (default `""`)
- **Description**: Optional Elasticsearch username for basic authentication. Env var: `MONSTACHE_ES_USER`.

### elasticsearch-password
- **Type**: string (default `""`)
- **Description**: Optional Elasticsearch password for basic authentication. Env var: `MONSTACHE_ES_PASS`.

### elasticsearch-urls
- **Type**: []string (default `[ "http://localhost:9200" ]`)
- **Description**: An array of URLs to connect to the Elasticsearch REST Interface. Env var: `MONSTACHE_ES_URLS`.

### elasticsearch-healthcheck-timeout-startup
- **Type**: int (default `15`)
- **Description**: Seconds to wait for the initial Elasticsearch health check before exiting.

### elasticsearch-healthcheck-timeout
- **Type**: int (default `5`)
- **Description**: Seconds to wait for post-initial Elasticsearch health checks.

### elasticsearch-version
- **Type**: string
- **Description**: Manually specify the Elasticsearch server version to determine API interaction logic.

### elasticsearch-max-conns
- **Type**: int (default `4`)
- **Description**: Size of the Elasticsearch HTTP connection pool, determining concurrency of bulk indexing.

### elasticsearch-retry
- **Type**: boolean (default `false`)
- **Description**: When true, failed requests to Elasticsearch are retried with an exponential backoff policy.

### elasticsearch-client-timeout
- **Type**: int (default `0`)
- **Description**: Seconds before a request to Elasticsearch times out. 0 disables the timeout.
```

--------------------------------

### Sync Timeout Configuration

Source: https://rwynn.github.io/monstache-site/config

Configures the maximum time in seconds an operation will wait for a connection to a usable server before returning an error. Must be greater than 0.

```APIDOC
## sync-timeout

### Description
Amount of time in seconds an operation will wait before returning an error in case a connection to a usable server can't be established. Must be greater than 0.

### Type
int

### Default
30
```

--------------------------------

### Configure Monstache routing script

Source: https://rwynn.github.io/monstache-site/advanced

TOML configuration for a Monstache script that sets the routing metadata based on the document's post_id.

```toml
[[script]]
namespace = "blog.comments"
routing = true
script = """
module.exports = function(doc) {
    doc._meta_monstache = { routing: doc.post_id };
    return doc;
}
"""

```

--------------------------------

### Configure Time Machine Namespaces

Source: https://rwynn.github.io/monstache-site/config

Specify MongoDB namespaces to enable time machine functionality. Changes to documents in these namespaces will be logged in timestamped Elasticsearch indexes. Can be set via command line or environment variable.

```bash
./monstache --time-machine-namespace test.foo --time-machine-namespace test.bar
```

```bash
MONSTACHE_TIME_MACHINE_NS=test.foo,test.bar
```

--------------------------------

### Change Stream Namespaces Configuration

Source: https://rwynn.github.io/monstache-site/config

Configuration for enabling and specifying MongoDB change stream namespaces for real-time data tailing.

```APIDOC
## change-stream-namespaces

### Description
This option allows you to opt in to using MongoDB change streams, requiring MongoDB 3.6 or above. When enabled, Monstache tails specified namespaces using the `watch` API, disabling direct oplog tailing.

### Parameters
- **change-stream-namespaces** ([]string) - Optional - An array of namespaces (e.g., `db.collection`, `db`, or `''` for the entire deployment) to be tailed using the `watch` API. (env var name `MONSTACHE_CHANGE_STREAM_NS`)

### Notes
- If using MongoDB 4+, you can specify a database name to tail all collections within that database, or an empty string to tail the entire deployment.
- Command line: `./monstache --change-stream-namespace test.foo`
- Environment variable: `MONSTACHE_CHANGE_STREAM_NS=test.foo,test.bar` (namespaces separated by `env-delimiter`, defaults to comma).
- If `change-stream-namespaces` is enabled, the same database cannot be used for both change streams and config storage.
```

--------------------------------

### Configure Time Machine Index Suffix Format

Source: https://rwynn.github.io/monstache-site/config

Define the date format for the suffix of time machine index names. Defaults to '2006-01-02' (Year-Month-Day). Consult Go's time formatting documentation for options.

```string
time-machine-index-suffix
```

--------------------------------

### Run Specific Release Monstache Docker Image

Source: https://rwynn.github.io/monstache-site/advanced

Commands to pull and run specific release versions of the Monstache Docker image.

```bash
docker run rwynn/monstache:6.7.4 -v
```

```bash
docker run rwynn/monstache:5.7.4 -v
```
Tokens:
Raw