ElastAlert

https://github.com/yelparchive/elastalert
Admin
ElastAlert is a modular alerting framework for detecting anomalies, spikes, and other patterns of...
Tokens:52,725
Snippets:405
Trust Score:8.1
License:Apache-2.0
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install ElastAlert module

Source: https://github.com/yelparchive/elastalert/blob/master/README.md

Installs the module after cloning, requiring setuptools>=11.3.

```bash
pip install "setuptools>=11.3"
```

```bash
python setup.py install
```

--------------------------------

### MongoRulesLoader example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/adding_loaders.rst

Example implementation of a RulesLoader that retrieves rules from a MongoDB database.

```APIDOC
## MongoRulesLoader

### Description
A custom RulesLoader that loads rules from a MongoDB database.

### Constructor
`__init__(self, conf)`
- **Parameters**:
  - `conf` (dict) - Required - Configuration dictionary containing `mongo_url` and `mongo_db` keys.

### Methods

#### get_names(conf, use_rule=None)
- **Parameters**:
  - `conf` (dict) - Required - Configuration dictionary.
  - `use_rule` (str) - Optional - If provided, return a list containing only this rule name.
- **Returns**: List of rule names from the database.

#### get_hashes(conf, use_rule=None)
- **Parameters**:
  - `conf` (dict) - Required - Configuration dictionary.
  - `use_rule` (str) - Optional - If provided, return a list containing only this rule name.
- **Returns**: Dictionary mapping rule names to their hashes.

#### get_yaml(rule)
- **Parameters**:
  - `rule` (str) - Required - The rule name.
- **Returns**: The YAML content of the rule as a string.

### Example Code
```python
from pymongo import MongoClient
from elastalert.loaders import RulesLoader
import yaml

class MongoRulesLoader(RulesLoader):
    def __init__(self, conf):
        super(MongoRulesLoader, self).__init__(conf)
        self.client = MongoClient(conf['mongo_url'])
        self.db = self.client[conf['mongo_db']]
        self.cache = {}

    def get_names(self, conf, use_rule=None):
        if use_rule:
            return [use_rule]

        rules = []
        self.cache = {}
        for rule in self.db.rules.find():
            self.cache[rule['name']] = yaml.load(rule['yaml'])
            rules.append(rule['name'])

        return rules

    def get_hashes(self, conf, use_rule=None):
        if use_rule:
            return [use_rule]

        hashes = {}
        self.cache = {}
        for rule in self.db.rules.find():
            self.cache[rule['name']] = rule['yaml']
            hashes[rule['name']] = rule['hash']

        return hashes

    def get_yaml(self, rule):
        if rule in self.cache:
            return self.cache[rule]

        self.cache[rule] = yaml.load(self.db.rules.find_one({'name': rule})['yaml'])
        return self.cache[rule]
```
```

--------------------------------

### Example output of top_count_keys

Source: https://github.com/yelparchive/elastalert/blob/master/README.md

Example output that may appear in an alert when top_count_keys is used, showing counts for each value.

```text
ip_address:
127.0.0.1: 7
10.0.0.1: 2
192.168.0.1: 1

status:
200: 9
500: 1
```

--------------------------------

### wildcard filter example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/writing_filters.rst

Example of using the wildcard filter for wildcard matches.

```yaml
filter:
- query:
    wildcard:
      field: "foo*bar"
```

--------------------------------

### Install ElastAlert via pip

Source: https://github.com/yelparchive/elastalert/blob/master/README.md

Installs the latest released version of ElastAlert using pip.

```bash
pip install elastalert
```

--------------------------------

### Whitelist rule configuration example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Example configuration for the whitelist rule type, showing how to define a list of whitelisted values and file paths using the '!file' syntax. The 'compare_key' field must be in this list to avoid matching.

```yaml
whitelist:
    - value1
    - value2
    - "!file /tmp/whitelist1.txt"
    - "!file /tmp/whitelist2.txt"
```

--------------------------------

### Example alert output

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/adding_rules.rst

Example alert output from the custom rule, showing the rule name, the match string from get_match_str, and the included fields from the match.

```default
Example login rule

userXYZ logged in between 20:00 and 24:00

@timestamp: 2015-03-02T22:23:24Z
username: userXYZ
```

--------------------------------

### Spike rule configuration and examples

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

This block contains the spike rule configuration and several worked examples. The configuration uses 'timeframe', 'spike_height', 'spike_type', and 'threshold_cur' or 'threshold_ref' to define alert conditions. The examples show event counts per hour and explain why alerts are or are not triggered based on the reference window, spike height, and thresholds.

```default
" Alert if at least 15 events occur within two hours and less than a quarter of that number occurred within the previous two hours. "
timeframe: hours: 2
spike_height: 4
spike_type: up
threshold_cur: 15

hour1: 5 events (ref: 0, cur: 5) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour2: 5 events (ref: 0, cur: 10) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour3: 10 events (ref: 5, cur: 15) - No alert because (a) spike_height not met, (b) ref window not filled
hour4: 35 events (ref: 10, cur: 45) - Alert because (a) spike_height met, (b) threshold_cur met, (c) ref window filled

hour1: 20 events (ref: 0, cur: 20) - No alert because ref window not filled
hour2: 21 events (ref: 0, cur: 41) - No alert because ref window not filled
hour3: 19 events (ref: 20, cur: 40) - No alert because (a) spike_height not met, (b) ref window not filled
hour4: 23 events (ref: 41, cur: 42) - No alert because spike_height not met

hour1: 10 events (ref: 0, cur: 10) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour2: 0 events (ref: 0, cur: 10) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour3: 0 events (ref: 10, cur: 0) - No alert because (a) threshold_cur not met, (b) ref window not filled, (c) spike_height not met
hour4: 30 events (ref: 10, cur: 30) - No alert because spike_height not met
hour5: 5 events (ref: 0, cur: 35) - Alert because (a) spike_height met, (b) threshold_cur met, (c) ref window filled

" Alert if at least 5 events occur within two hours, and twice as many events occur within the next two hours. "
timeframe: hours: 2
spike_height: 2
spike_type: up
threshold_ref: 5

hour1: 20 events (ref: 0, cur: 20) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour2: 100 events (ref: 0, cur: 120) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour3: 100 events (ref: 20, cur: 200) - No alert because ref window not filled
hour4: 100 events (ref: 120, cur: 200) - No alert because spike_height not met

hour1: 0 events (ref: 0, cur: 0) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour2: 20 events (ref: 0, cur: 20) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour3: 100 events (ref: 0, cur: 120) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour4: 100 events (ref: 20, cur: 200) - Alert because (a) spike_height met, (b) threshold_ref met, (c) ref window filled

hour1: 1 events (ref: 0, cur: 1) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour2: 2 events (ref: 0, cur: 3) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour3: 2 events (ref: 1, cur: 4) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour4: 1000 events (ref: 3, cur: 1002) - No alert because threshold_ref not met
hour5: 2 events (ref: 4, cur: 1002) - No alert because threshold_ref not met
hour6: 4 events: (ref: 1002, cur: 6) - No alert because spike_height not met

hour1: 1000 events (ref: 0, cur: 1000) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour2: 0 events (ref: 0, cur: 1000) - No alert because (a) threshold_ref not met, (b) ref window not filled
hour3: 0 events (ref: 1000, cur: 0) - No alert because (a) spike_height not met, (b) ref window not filled
hour4: 0 events (ref: 1000, cur: 0) - No alert because spike_height not met
hour5: 1000 events (ref: 0, cur: 1000) - No alert because threshold_ref not met
hour6: 1050 events (ref: 0, cur: 2050)- No alert because threshold_ref not met
hour7: 1075 events (ref: 1000, cur: 2125) Alert because (a) spike_height met, (b) threshold_ref met, (c) ref window filled

" Alert if at least 100 events occur within two hours and less than a fifth of that number occurred in the previous two hours. "
timeframe: hours: 2
spike_height: 5
spike_type: up
threshold_cur: 100

hour1: 1000 events (ref: 0, cur: 1000) - No alert because ref window not filled

hour1: 2 events (ref: 0, cur: 2) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour2: 1 events (ref: 0, cur: 3) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour3: 20 events (ref: 2, cur: 21) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour4: 81 events (ref: 3, cur: 101) - Alert because (a) spike_height met, (b) threshold_cur met, (c) ref window filled

hour1: 10 events (ref: 0, cur: 10) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour2: 20 events (ref: 0, cur: 30) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour3: 40 events (ref: 10, cur: 60) - No alert because (a) threshold_cur not met, (b) ref window not filled
hour4: 80 events (ref: 30, cur: 120) - No alert because spike_height not met
hour5: 200 events (ref: 60, cur: 280) - No alert because spike_height not met
```

--------------------------------

### Blacklist rule configuration example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Example configuration for the blacklist rule type, showing how to define a list of blacklisted values and file paths using the '!file' syntax. The 'compare_key' field must match one of these values to trigger an alert.

```yaml
blacklist:
    - value1
    - value2
    - "!file /tmp/blacklist1.txt"
    - "!file /tmp/blacklist2.txt"
```

--------------------------------

### Build HTML documentation locally

Source: https://github.com/yelparchive/elastalert/blob/master/README.md

Install the required Sphinx theme and Sphinx, then build the HTML documentation. The output will be available at build/html/index.html.

```bash
pip install sphinx_rtd_theme sphinx
cd docs
make html
```

--------------------------------

### range filter example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/writing_filters.rst

Example of using the range filter for ranges on fields.

```yaml
filter:
- range:
    status_code:
      from: 500
      to: 599
```

--------------------------------

### Install elasticsearch-py for Elasticsearch 5.0+

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/running_elastalert.rst

Manually install the correct version of elasticsearch-py for Elasticsearch 5.0 and above.

```default
$ pip install "elasticsearch>=5.0.0"
```

--------------------------------

### Install elasticsearch-py for Elasticsearch 2.X

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/running_elastalert.rst

Manually install the correct version of elasticsearch-py for Elasticsearch 2.X.

```default
$ pip install "elasticsearch<3.0.0"
```

--------------------------------

### spike_aggregation metric_agg_script configuration example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Shows how to configure a scripted metric for the spike_aggregation rule using the metric_agg_script parameter with a Painless script. The example defines a scripted field named myScriptedMetric that multiplies two document fields.

```yaml
metric_agg_key: myScriptedMetric
metric_agg_script:
    script: doc['field1'].value * doc['field2'].value

```

--------------------------------

### theHive alert configuration example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Example configuration for the theHive alert type. Required keys: hive_host, hive_port, hive_apikey. Optional: hive_proxies, hive_observable_data_mapping. The hive_alert_config.title defaults to {rule[index]_rule[name]} if not provided.

```default
alert: hivealerter

 hive_connection:
   hive_host: http://localhost
   hive_port: <hive_port>
   hive_apikey: <hive_apikey>
   hive_proxies:
     http: ''
     https: ''

  hive_alert_config:
    title: 'Title'  ## This will default to {rule[index]_rule[name]} if not provided
    type: 'external'
    source: 'elastalert'
    description: '{match[field1]} {rule[name]} Sample description'
    severity: 2
    tags: ['tag1', 'tag2 {rule[name]}']
    tlp: 3
    status: 'New'
    follow: True

hive_observable_data_mapping:
    - domain: "{match[field1]}_{rule[name]}"
    - domain: "{match[field]}"
    - ip: "{match[ip_field]}"

```

--------------------------------

### Kibana 4 start timedelta configuration

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Specifies the start time offset for a generated Kibana 4 dashboard. The value is added in front of the event time.

```yaml
kibana4_start_timedelta: minutes: 2
```

--------------------------------

### HTTP POST alerter example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Example usage of the HTTP POST alerter. It sends results to a JSON endpoint using HTTP POST. The payload maps the clientip field to the JSON key ip, includes a static apikey, and sets an authorization header.

```default
alert: post
http_post_url: "http://example.com/api"
http_post_payload:
  ip: clientip
http_post_static_payload:
  apikey: abc123
http_post_headers:
  authorization: Basic 123dr3234
```

--------------------------------

### Sample data for aggregation_key example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Sample data points showing events with different usernames and timestamps. With aggregation_key set to 'my_data.username' and a 10-minute window, this results in two alerts: one for alice's two events and one for bob's single event.

```python
{'my_data': {'username': 'alice', 'event_type': 'login'}, '@timestamp': '2016-09-20T00:00:00'}
{'my_data': {'username': 'bob', 'event_type': 'something'}, '@timestamp': '2016-09-20T00:05:00'}
{'my_data': {'username': 'alice', 'event_type': 'something else'}, '@timestamp': '2016-09-20T00:06:00'}
```

--------------------------------

### terms filter examples

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/writing_filters.rst

Examples of using the terms filter to combine multiple term filters on one or multiple fields.

```yaml
filter:
- terms:
    field: ["value1", "value2"] # value1 OR value2
```

```yaml
- terms:
    fieldX: ["value1", "value2"]
    fieldY: ["something", "something_else"]
    fieldZ: ["foo", "bar", "baz"]
```

--------------------------------

### Create a modules folder for the enhancement

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/adding_enhancements.rst

Commands to create a `elastalert_modules` folder in the ElastAlert directory, move into it, and create an `__init__.py` file so the folder is a Python module.

```bash
$ mkdir elastalert_modules
$ cd elastalert_modules
$ touch __init__.py

```

--------------------------------

### Example rule configuration in example_frequency.yaml

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/running_elastalert.rst

This YAML file defines a frequency rule that sends an email when more than 50 documents match a term filter within a 4-hour timeframe. It requires a valid es_host and es_port pointing to the Elasticsearch cluster, and the email alert requires an SMTP server (defaults to localhost, configurable with smtp_host). The rule uses a unique 'name' and a timestamp field (default @timestamp).

```yaml
# From example_rules/example_frequency.yaml
es_host: elasticsearch.example.com
es_port: 14900
name: Example rule
type: frequency
index: logstash-*
num_events: 50
timeframe:
    hours: 4
filter:
- term:
    some_field: "some_value"
alert:
- "email"
email:
- "elastalert@example.com"

```

--------------------------------

### Example alert email body

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/running_elastalert.rst

Example email body generated by an alert, showing the rule name, the alert message, and the timestamp of the alert.

```default
Example rule

At least 50 events occurred between 1-15 11:12 PST and 1-15 15:12 PST

@timestamp: 2015-01-15T15:12:00-08:00
```

--------------------------------

### query_string filter examples

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/writing_filters.rst

Examples of using the query_string filter for partial or full matches across multiple fields, following Lucene query syntax.

```yaml
filter:
- query:
    query_string:
      query: "username: bob"
- query:
    query_string:
      query: "_type: login_logs"
- query:
    query_string:
      query: "field: value OR otherfield: othervalue"
- query:
    query_string:
       query: "this: that AND these: those"
```

--------------------------------

### Summary table output for aggregation

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Example summary table output for the sample data when summary_table_fields is set. Shows the specified fields for each event in the alert medium.

```text
+------------------+--------------------+
| my_data.username | my_data.event_type |
+------------------+--------------------+
|      alice       |       login        |
|       bob        |     something      |
|      alice       |   something else   |
+------------------+--------------------+
```

--------------------------------

### Rule configuration example_login_rule.yaml

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/adding_rules.rst

YAML configuration for the custom rule, specifying the rule type as elastalert_modules.my_rules.AwesomeRule, the time range, usernames, and the include field. It alerts via debug when a matching login occurs.

```yaml
name: "Example login rule"
es_host: elasticsearch.example.com
es_port: 14900
type: "elastalert_modules.my_rules.AwesomeRule"
# Alert if admin, userXYZ or foobaz log in between 8 PM and midnight
time_start: "20:00"
time_end: "24:00"
usernames:
- "admin"
- "userXYZ"
- "foobaz"
# We require the username field from documents
include:
- "username"
alert:
- debug
```

--------------------------------

### term filter examples

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/recipes/writing_filters.rst

Examples of using the term filter for exact field matches. Note that term queries may not behave as expected on analyzed fields; use query_string for full text matching.

```yaml
filter:
- term:
    name_field: "bob"
- term:
    _type: "login_logs"
```

--------------------------------

### Alerter property reference example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Example of referencing top-level rule properties in Alerter fields by surrounding the property name with dollar signs. This is useful when you have rule-level properties that you want to reference many times in your alert.

```default
jira_priority: $priority$
jira_alert_owner: $owner$
```

--------------------------------

### Kibana Discover index pattern saved object export example

Source: https://github.com/yelparchive/elastalert/blob/master/docs/source/ruletypes.rst

Example export of an index pattern's saved object, showing the _id field that can be modified and re-imported to change the index pattern ID.

```text
[
    {
        "_id": "4e97d188-8a45-4418-8a37-07ed69b4d34c",
        "_type": "index-pattern",
        "_source": { ... }
    }
]
```
Tokens:
Raw