### Build Release Output Example

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Example output from the `build-release` command indicating rule changes, exclusions, and a reminder to update version locks.

```bash
+ python -m detection_rules dev build-release --update-version-lock
[+] Building package 8.3
- 5 rules excluded from package
Rule changes detected!
- 10 changed rules
- 11 new rules
- 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Rule changes detected!
```

--------------------------------

### Install Detection Rules CLI Prerequisites (Shell)

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This command installs the necessary prerequisites for the detection-rules Python CLI. It assumes you have navigated to the 'detection-rules' directory in your terminal.

```shell
cd detection-rules
make
```

--------------------------------

### Explore Custom Rules CLI Setup Configuration

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Shows the help message for the 'setup-config' command within the 'custom-rules' subcommand. This command is used to initialize configuration files for managing custom rules.

```bash
python -m detection_rules custom-rules setup-config -h
```

--------------------------------

### Setup Custom Rules Directory and Config

Source: https://github.com/elastic/dac-reference/blob/main/docs/faq.md

Generates a base set of configuration files and a custom rules directory for detection rules. This is a prerequisite for running unit tests on custom rule sets.

```python
python -m detection_rules custom-rules setup-config <new_directory_name>
```

--------------------------------

### Clone and Install DaC Project Dependencies

Source: https://github.com/elastic/dac-reference/blob/main/README.md

This snippet demonstrates how to clone the Elastic DaC Reference repository and install its dependencies using Poetry. It assumes you have Git and Poetry installed.

```bash
git clone https://github.com/elastic/DaC-Reference
cd DaC-Reference
poetry install
```

--------------------------------

### Install Detection Rules CLI Prerequisites (Python Pip)

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This command installs the detection-rules Python CLI prerequisites directly using pip. It is an alternative to the 'make' command and requires Python to be installed.

```python
pip install "[.dev]"
```

--------------------------------

### Example Custom Rules Directory Structure

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Illustrates the standard directory structure for custom detection rules, including subdirectories for rules, actions, exceptions, and configuration files.

```yaml
#   example structure:
#     dac_custom_rules_dir
├── _config.yaml
├── actions
    ├── action_1.toml
    ├── action_2.toml
├── action_connectors
    ├── action_connector_1.toml
    ├── action_connector_2.toml
├── etc
│	   ├── deprecated_rules.json
│	   ├── packages.yaml
│	   ├── stack-schema-map.yaml
│	   ├── test_config.yaml
│	   └── version.lock.json
├── exceptions
    ├── exception_1.toml
    └── exception_2.toml
└── rules
    ├── example_rule_1.toml
    └── example_rule_2.toml
```

--------------------------------

### Example Detection Rule with Custom Fields (EQL)

Source: https://github.com/elastic/dac-reference/blob/main/docs/faq.md

An example of a detection rule written in EQL that utilizes custom fields defined in a schema. This rule demonstrates how to reference and query these custom fields.

```toml
[metadata]
creation_date = "2023/08/23"
integration = ["endpoint"]
maturity = "production"
min_stack_comments = "New fields added: required_fields, related_integrations, setup"
min_stack_version = "8.3.0"
updated_date = "2024/03/08"

[rule]
author = ["Elastic"]
description = """
Test Rule
"""
from = "now-9m"
index = ["logs-endpoint.events.*", "custom-index*"]
language = "eql"
license = "Elastic License v2"
name = "TEST Potential Protocol Tunneling via Chisel Client"
note = "Test Note"
references = [
    "https://blog.bitsadmin.com/living-off-the-foreign-land-windows-as-offensive-platform",
    "https://book.hacktricks.xyz/generic-methodologies-and-resources/tunneling-and-port-forwarding"
    ]
risk_score = 47
rule_id = "e8e3af2a-11b8-4ab7-9ca1-c6db621ea89d"
setup = "Test Setup"
severity = "medium"
tags = [
        "Domain: Endpoint",
        "OS: Linux",
        "Use Case: Threat Detection",
        "Tactic: Command and Control",
        "Data Source: Elastic Defend"
        ]
type = "eql"
query = '''
process where host.os.type == "linux" and process.NewValue == "GoodValue"
'''
timestamp_override = "event.ingested"

[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1572"
name = "Protocol Tunneling"
reference = "https://attack.mitre.org/techniques/T1572/"

[rule.threat.tactic]
id = "TA0011"
name = "Command and Control"
reference = "https://attack.mitre.org/tactics/TA0011/"

```

--------------------------------

### Custom Rule Configuration (YAML)

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Example YAML configuration file for custom detection rules, specifying directories for rule building blocks, actions, exceptions, and other essential rule components.

```yaml
bbr_rules_dirs:
- rules_building_block
directories:
  action_connector_dir: action_connectors
  action_dir: actions
  exception_dir: exceptions
files:
  deprecated_rules: etc/deprecated_rules.json
  packages: etc/packages.yaml
  stack_schema_map: etc/stack-schema-map.yaml
  version_lock: etc/version.lock.json
rule_dirs:
- rules
testing:
  config: etc/test_config.yaml
```

--------------------------------

### Create Detection Rule via CLI

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Initiates the semi-automated creation of detection rules using the 'create-rule' command. Users can specify a path, configuration file, and rule type to guide the interactive CLI prompts for generating compliant rule files.

```bash
detection_rules create-rule PATH
detection_rules create-rule PATH -c CONFIG_FILE
detection_rules create-rule PATH -t RULE_TYPE
```

--------------------------------

### Example Deprecated Rule Entry

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

An example JSON object showing the structure of a deprecated rule entry, including its deprecation date, rule name, and the Elastic Security stack version it supported.

```json
{
    "041d4d41-9589-43e2-ba13-5680af75ebc2": {
        "deprecation_date": "2023/09/25",
        "rule_name": "Deprecated - Potential DNS Tunneling via Iodine",
        "stack_version": "8.3"
    }
}
```

--------------------------------

### Kibana Import Rules Command Example

Source: https://github.com/elastic/dac-reference/blob/main/docs/known_limitations.md

This command is part of a workaround for an intermittent Kibana import issue where exception list references might be stripped. It attempts to re-upload rules that may have failed due to this bug.

```bash
kibana import-rules
```

--------------------------------

### Example Version.lock.json Structure

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Demonstrates the structure of the version.lock.json file used for managing detection rule versions. It includes fields like minimum stack version, rule name, SHA256 hash, rule type, and an incrementing version number.

```json
{
    "00140285-b827-4aee-aa09-8113f58a08f3": {
        "min_stack_version": "8.3",
        "rule_name": "Potential Credential Access via Windows Utilities",
        "sha256": "456e5ed43e056841aea460851e9e496aa85a9828fcb4bebade3a4f8b1d2a637e",
        "type": "eql",
        "version": 110
    }
}
```

--------------------------------

### Configure Detection Rules CLI Authentication (Environment Variables)

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This example shows how to configure the detection-rules CLI using environment variables. Variables prefixed with 'DR_' are used to pass authentication and connection details, mirroring the configuration file options.

```bash
env:
        DR_KIBANA_URL: ${{ secrets.KIBANA_URL }}
        DR_ES_USER: ${{ secrets.ES_USER }}
        DR_ES_PASSWORD: ${{ secrets.ES_PASSWORD }}
```

--------------------------------

### CI/CD Workflow for Unit Testing Rules (YAML)

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md

This YAML configuration defines a GitHub Actions workflow for running custom unit tests on Elastic Security rules. It checks out code, sets up Python, installs dependencies, performs linting with Flake8, and executes unit tests using the 'detection_rules' command.

```yaml
name: Custom Unit Tests

on:
  push:
    branches: [ "main", "7.*", "8.*" ]
  pull_request:
    branches: [ "*" ]

jobs:
  tests:

    runs-on: ubuntu-latest
    env:
      CUSTOM_RULES_DIR: ${{ secrets.CUSTOM_RULES_DIR }}

    steps:
    - uses: actions/checkout@v2

    - name: Set up Python 3.12
      uses: actions/setup-python@v5
      with:
        python-version: '3.12'

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip cache purge
        pip install .[dev]

    - name: Python Lint
      run: |
        python -m flake8 tests detection_rules --ignore D203,N815 --max-line-length 120

    - name: Unit tests
      run: |
        python -m detection_rules test

```

--------------------------------

### Import single rule with optional output

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Imports a single detection rule from a specified TOML file to Kibana. The `-o` flag can be used for optional output configuration. This command is useful for adding individual rules to your security monitoring setup.

```bash
python -m detection_rules kibana import-rules -f test-export-rules/credential_access_EXISTING_RULE.toml -o
```

--------------------------------

### GitHub Actions Cron Job Schedule Configuration

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Configures a scheduled trigger for GitHub Actions workflows using cron syntax, enabling automated deployments at predefined intervals. The example shows a daily run at 2 AM UTC.

```yaml
# Example GitHub Action Cron Job
on:
  schedule:
    - cron: '0 2 * * *' # Runs at 2 AM UTC daily
```

--------------------------------

### Import single rule with existing rule ID collision

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Attempts to import a single detection rule from a TOML file. This example demonstrates the expected failure when a rule with an identical rule ID already exists in Kibana. The output indicates an `AssertionError` due to the collision.

```bash
python -m detection_rules kibana import-rules -f test-export-rules/credential_access_NEW_RULE.toml
```

--------------------------------

### Import directory with rule ID collision

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Imports all detection rules from a specified directory. This command is designed to import multiple rules at once. The example shows an expected failure scenario where multiple rules within the directory have conflicting rule IDs, leading to a batch of import errors.

```bash
python -m detection_rules kibana import-rules -d test-export-rules
```

--------------------------------

### Alternative Methods to Run Unit Tests

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Provides alternative command-line methods for running unit tests, including using 'make test', 'pytest [tests]', and 'unittest [tests]'. These commands offer flexibility in how tests are invoked and managed within the project.

```bash
make test
```

```bash
pytest [tests]
```

```bash
unittest [tests]
```

--------------------------------

### Build Release Package using detection-rules CLI

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Builds a release-ready package for deployment, typically including all necessary components except manifest updates. Requires registry information in `packages.yaml`.

```bash
python -m detection_rules dev build-release
```

--------------------------------

### Detection Rules CLI Help

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This command displays the help information for the detection-rules CLI, including available options and sub-commands for managing detection rules within Elastic Security.

```bash
python -m detection_rules -h
```

--------------------------------

### Configure Local Detection Rules CLI

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

This JSON snippet shows the configuration for the `.detection-rules-cfg.json` file, used locally to set up authentication with Elastic Security. It includes `cloud_id` and `api_key` for connecting to the instance.

```json
{
    "cloud_id": "example:example",
    "api_key": "example"
}
```

--------------------------------

### Initialize Custom Rules Directory

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

Initializes a custom rules directory for managing detection rules. This command sets up the necessary file structure and configuration for custom rule management.

```python
python -m detection_rules custom-rules setup-config dac_custom_rules_dir
```

--------------------------------

### Build Documentation Locally with Sphinx

Source: https://github.com/elastic/dac-reference/blob/main/README.md

This command builds the project's documentation locally using Sphinx, invoked via Poetry. It generates HTML output from the source files in the 'docs' directory.

```bash
poetry run build-docs -b html docs docs/_build
```

--------------------------------

### Testing Custom Rules (Bash)

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Command-line instructions for validating and testing custom detection rules using 'make test' and Python module execution.

```bash
CUSTOM_RULES_DIR=dac_custom_rules_dir make test
```

```bash
CUSTOM_RULES_DIR=dac_custom_rules_dir python -m detection_rules test
```

--------------------------------

### Authentication Methods

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Details on how to authenticate with Elastic Security APIs, including configuration file, environment variables, and API key options.

```APIDOC
## Authentication

### Description
Authentication with Elastic Security APIs can be achieved through various methods, including explicit arguments, environment variables, configuration files, or API keys. The order of precedence for credentials is: explicit arguments, environment variables, configuration values, and finally prompts for certain values.

### Method
N/A (Authentication Configuration)

### Endpoint
N/A (Authentication Configuration)

### Parameters
#### Configuration File (`.detection-rules-cfg.json`)
- **elasticsearch_url** (string) - Required - The URL for Elasticsearch.
- **kibana_url** (string) - Required - The URL for Kibana.
- **cloud_id** (string) - Optional - The Cloud ID for Elastic Cloud.
- **es_username** (string) - Optional - Username for Elasticsearch authentication.
- **es_password** (string) - Optional - Password for Elasticsearch authentication.
- **api_key** (string) - Optional - API key for authentication.

#### Environment Variables
Environment variables can be used by prefixing with `DR_` (e.g., `DR_KIBANA_URL`, `DR_ES_USER`).

### Request Example
#### Using `.detection-rules-cfg.json`
```json
{
  "elasticsearch_url": "http://localhost:9200",
  "kibana_url": "http://localhost:5601",
  "es_username": "elastic",
  "es_password": "password"
}
```

#### Using API Key with Kibana URL
```json
{
  "kibana_url": "http://localhost:5601",
  "api_key": "key value"
}
```

### Response
N/A (Authentication details are for configuration)
```

--------------------------------

### detection-rules CLI Commands

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Overview of commands available through the detection-rules CLI for interacting with Elastic Security and Kibana.

```APIDOC
## detection-rules CLI Commands

### Description
The `detection-rules` CLI tool provides various commands for managing detection rules and interacting with Elastic Security and Kibana. It simplifies tasks such as exporting, generating indexes, and importing rules.

### Method
N/A (CLI Commands)

### Endpoint
N/A (CLI Commands)

### Parameters
#### Global Options
- **-D, --debug** - Enable debug logging.
- **-N, --no-debug** - Disable debug logging.
- **-h, --help** - Show help messages.

#### Available Commands
- **export-rules-from-repo**: Export rule(s) and exception(s) into an importable ndjson file.
- **generate-rules-index**: Generate enriched indexes of rules for Elasticsearch/Kibana.
- **import-rules-to-repo**: Import rules from various file formats into the repository.
- **kibana**: Commands for integrating with Kibana.

### Request Example
```bash
python -m detection_rules --help
python -m detection_rules export-rules-from-repo --rule-id 'some-rule-id'
```

### Response
N/A (CLI command output varies based on command executed)
```

--------------------------------

### Deploy Fleet Package to Elastic Security Environment

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Uploads a Fleet-compatible security detection rules package (as a zip file) to the Elastic Security environment via the Fleet API. Requires authentication and content type specification.

```bash
curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' https://my-kibana:9243/api/fleet/epm/packages -u elastic:<password> --data-binary @security_detection_engine-8.13.0-beta.0.zip
```

--------------------------------

### Run Detection Rules Unit Tests

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Executes the unit tests for the detection-rules repository, which includes built-in query validation. This command verifies query syntax and field validity.

```bash
python -m detection_rules test
```

--------------------------------

### Kibana Rule Import Help

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Displays the help message for the 'detection_rules kibana import-rules' command, outlining available options such as specifying rule files, directories, Kibana connection details, and overwrite behavior.

```bash
(venv312) ➜  detection-rules-fork git:(main) ✗ python -m detection_rules kibana import-rules -h

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Kibana client:
Options:
  --ignore-ssl-errors TEXT
  --space TEXT              Kibana space
  --api-key TEXT            [required]
  --cloud-id TEXT           ID of the cloud instance.
  --kibana-url TEXT

Usage: detection_rules kibana import-rules [OPTIONS]

  Import custom rules into Kibana.

Options:
  -f, --rule-file FILE
  -d, --directory DIRECTORY       Recursively load rules from a directory
  -id, --rule-id TEXT
  -o, --overwrite                 Overwrite existing rules
  -e, --overwrite-exceptions      Overwrite exceptions in existing rules
  -ac, --overwrite-action-connectors
                                  Overwrite action connectors in existing rules
  -h, --help                      Show this message and exit.

```

--------------------------------

### Configure _config file for Detection Rules

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

This YAML snippet shows essential parameters to add to the `_config` file within the custom rules directory for enhanced usability. These settings help manage version locking and schema generation.

```yaml
bypass_version_lock: True
normalize_kql_keywords: True
auto_gen_schema_file: "etc/schemas/auto_gen.json"
bypass_optional_elastic_validation: True
```

--------------------------------

### Run All Unit Tests (Python CLI)

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Executes all prebuilt unit tests for the detection rules using the Python CLI. This confirms that all rules pass their respective test cases, ensuring compatibility and correct formatting for Elastic Security.

```bash
(detection_dev) ➜  detection-rules git:(main) ✗ python -m detection_rules test
Loaded config file: /Users/stryker/workspace/ElasticGitHub/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

================================================== test session starts ==================================================
platform darwin -- Python 3.11.5, pytest-7.4.2, pluggy-1.3.0 -- /Users/stryker/.virtualenvs/detection_dev/bin/python
cachedir: .pytest_cache
rootdir: /Users/stryker/workspace/ElasticGitHub/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
```

--------------------------------

### Create a Webhook Action Connector in TOML

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This TOML snippet illustrates the creation of an action connector for a webhook. It includes metadata, associated rule IDs and names, connector configuration such as URL and method, and secrets.

```toml
[metadata]
creation_date = "2024/08/04"
action_connector_name = "Action Connector 478b2165-83fb-480d-8a4a-bb47cfcafd4c"
rule_ids = ["2e299dad-3a09-4c08-89cb-f08c4f85a18e"]
rule_names = ["TestActionRule"]
updated_date = "2024/08/04"

[[action_connectors]]
id = "478b2165-83fb-480d-8a4a-bb47cfcafd4c"
managed = false
type = "action"
references = []

[action_connectors.attributes]
actionTypeId = ".webhook"
isMissingSecrets = false
name = "test"

[action_connectors.attributes.config]
hasAuth = false
method = "post"
url = "https://best-website-ever.com"

[action_connectors.attributes.secrets]

```

--------------------------------

### Detection Rules CLI Kibana Integration Help

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This command displays help information specific to Kibana integration commands within the detection-rules CLI, including options for managing SSL errors and other Kibana-related functionalities.

```bash
detection_rules kibana [OPTIONS] COMMAND [ARGS]...
```

--------------------------------

### CLI Commands for Updating Schemas

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Provides bash commands to update integration manifests and download schemas using the detection_rules CLI. These commands are essential for maintaining up-to-date validation resources.

```bash
python -m detection_rules dev integrations build-manifests
python -m detection_rules dev integrations build-schemas
```

--------------------------------

### Import Detection Rules to Kibana

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

Imports detection rules and their associated exceptions and action connectors to a Kibana instance. This command overwrites existing rules and exceptions, so backups are recommended.

```bash
python -m detection_rules kibana import-rules \
--overwrite \
--overwrite-exceptions \
--overwrite-action-connectors
```

--------------------------------

### Configure Directories for DaC in _config.yaml

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This snippet shows how to specify directory names for action connectors, actions, and exception lists within the `_config.yaml` file for Detection as Code (DaC) management.

```yaml
directories:
   # action_connector_dir: action_connectors
   # action_dir: actions
   # exception_dir: exceptions
```

--------------------------------

### Sync/Deploy Exception and Action Lists with Rules

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

A comment indicating the final step in the deployment process, which involves syncing or deploying exception and action lists along with detection rules. It highlights that linked lists are updated automatically when rule updates are pushed.

```bash
# Import rules (ndjon) to Kibana
python -m detection_rules kibana import-rules ...
```

--------------------------------

### Running Detection Rules Tests (Bash)

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This section provides bash commands for executing detection rules tests. It covers running all default tests, using a custom test configuration file, and specifying a custom rules directory.

```bash
make test
python -m detection_rules test
DETECTION_RULES_TEST_CONFIG=path/to/test_config.yaml make test
CUSTOM_RULES_DIR=custom-rules make test
```

--------------------------------

### Creating and Validating Detection Rules (Python CLI)

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Demonstrates the use of the Elastic Security detection_rules Python CLI for creating and validating new detection rules. The create-rule command assists in generating a new rule file, while validate-rule checks its format compliance.

```python
python -m detection_rules create-rule /path/to/new_rule.toml
python -m detection_rules validate-rule /path/to/new_rule.toml
```

--------------------------------

### Deploying Rules Package via Fleet API

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This section details how to deploy a rules package to the Elastic Security environment using the Fleet API.

```APIDOC
## POST /api/fleet/epm/packages

### Description
Deploys a security detection rules package to the Elastic Security environment.

### Method
POST

### Endpoint
/api/fleet/epm/packages

### Parameters
#### Header Parameters
- **content-type** (string) - Required - Specifies the content type as application/zip.
- **kbn-xsrf** (string) - Required - Cross-Site Request Forgery protection token.

#### Request Body
- **file** (file) - Required - The rules package in zip format (e.g., security_detection_engine-8.13.0-beta.0.zip).

### Request Example
```bash
curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' https://my-kibana:5601/api/fleet/epm/packages -u elastic:<password> --data-binary @security_detection_engine-8.13.0-beta.0.zip
```

### Response
#### Success Response (200)
Indicates the package was successfully uploaded. The exact response body may vary.

#### Response Example
(No specific example provided in the source text)
```

--------------------------------

### GitHub Actions Manual Trigger Configuration

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Configures a manual trigger for GitHub Actions workflows, allowing for on-demand deployment of detection rules. This requires setting up the `workflow_dispatch` event.

```yaml
# Example GitHub Action Manual Trigger
on:
  workflow_dispatch:
```

--------------------------------

### Export Detection Rules from Kibana

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

Exports detection rules, exceptions, and action connectors from a Kibana instance using the CLI. This command overwrites existing files, strips version information, and skips errors during the process.

```bash
python -m detection_rules kibana export-rules \
--skip-errors \
--strip-version \
--export-exceptions \
--export-action-connectors \
--directory my-custom-rules
```

--------------------------------

### Configure Custom Schemas in stack-schema-map.yaml

Source: https://github.com/elastic/dac-reference/blob/main/docs/faq.md

Specifies custom schema files to be used for validating detection rules against a target stack version. This file maps stack versions to their corresponding ECS and custom schema files.

```yaml
8.14.0:
  beats: 8.12.2
  ecs: 8.11.0
  endgame: 8.4.0
  custom: schemas/custom-schema.json
```

--------------------------------

### Import Rules to Repository from ndjson

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

Imports detection rules from an ndjson file into the repository. This command supports importing rules, action connectors, and exceptions, with options to specify the default author and skip errors.

```bash
python -m detection_rules import-rules-to-repo <ndjson_file> \
--required-only \
--action-connector-import \
--exceptions-import \
--default-author DefaultAuthor \
--skip-errors
```

--------------------------------

### GitHub Action: Push Rules on Merge to Main

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This workflow deploys DAC rules to the Elastic Security production space when changes are merged into the main branch. It supports overwriting existing rules, exceptions, and action connectors, and allows specifying the Kibana space. Dependencies include Python and the detection_rules package.

```yaml
# Example GitHub Action Workflow
name: Push Latest Rules to Elastic Security Space

on:
  push:
    branches:
      - main
    paths:
      - '**/*.toml'
  workflow_dispatch:
    inputs:
      overwrite:
        description: 'Overwrite existing rules'
        required: false
        default: 'true'
      overwrite_exceptions:
        description: 'Overwrite existing exceptions'
        required: false
        default: 'true'
      overwrite_action_connectors:
        description: 'Overwrite existing action connectors'
        required: false
        default: 'true'
      space:
        description: 'Kibana space to use (dev or prod)'
        required: false
        default: 'prod'

jobs:
  sync-to-production:
    runs-on: ubuntu-latest
    env:
      CUSTOM_RULES_DIR: ${{ secrets.CUSTOM_RULES_DIR }}

    steps:
    - name: Checkout Repository
      uses: actions/checkout@v4

    - name: Set up Python 3.12
      uses: actions/setup-python@v2
      with:
        python-version: '3.12'

    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip cache purge
        pip install .[dev]

    - name: Import Rules to Kibana
      run: |
        FLAGS=""
        if [[ "${{ github.event_name }}" == "push" || "${{ inputs.overwrite }}" == "true" ]]; then
          FLAGS="$FLAGS --overwrite"
        fi
        if [[ "${{ github.event_name }}" == "push" || "${{ inputs.overwrite_exceptions }}" == "true" ]]; then
          FLAGS="$FLAGS --overwrite-exceptions"
        fi
        if [[ "${{ github.event_name }}" == "push" || "${{ inputs.overwrite_action_connectors }}" == "true" ]]; then
          FLAGS="$FLAGS --overwrite-action-connectors"
        fi
        if [[ "${{ github.event_name }}" == "push" ]]; then
          SPACE="prod"  # Default to production for push events
        elif [[ "${{ inputs.space }}" == "dev" || "${{ inputs.space }}" == "prod" ]]; then
          SPACE="${{ inputs.space }}"  # Use provided space if valid
        else
          echo "::error::Invalid space provided. Defaulting to 'dev'."
          SPACE="dev"
        fi
        SPACE_FLAG="--space $SPACE"
        python -m detection_rules kibana $SPACE_FLAG import-rules $FLAGS
      env:
        DR_CLOUD_ID: ${{ secrets.ELASTIC_CLOUD_ID }}
        DR_API_KEY: ${{ secrets.ELASTIC_API_KEY }}

```

--------------------------------

### Kibana Integration Commands

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Specific commands for integrating with Kibana via the detection-rules CLI, including rule uploading.

```APIDOC
## Kibana Integration Commands

### Description
This section details the commands available within the `detection_rules kibana` subcommand for direct integration with Kibana. It includes functionalities for uploading rules and managing Kibana-specific operations. Note that for stacks version 9.0 and later, `kibana import-rules` is recommended over `kibana upload-rule`.

### Method
N/A (Kibana CLI Commands)

### Endpoint
N/A (Kibana CLI Commands)

### Parameters
#### `kibana upload-rule` (Deprecated for v9.0+)
- **[OPTIONS]**: Various options for uploading rules.

#### `kibana import-rules` (Recommended for v9.0+)
- **[OPTIONS]**: Various options for importing rules into Kibana.

### Request Example
```bash
python -m detection_rules kibana upload-rule --rule-file 'path/to/rule.json'
python -m detection_rules kibana import-rules --rule-file 'path/to/rules.ndjson'
```

### Response
N/A (CLI command output varies based on command executed)
```

--------------------------------

### Detection-Rules _config.yaml Configuration

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

Defines the structure and content of the `_config.yaml` file used by detection-rules to manage custom rule directories, rule files, and testing configurations.

```yaml
# detection-rules _config.yaml file

bbr_rules_dirs:
- rules_building_block
files:
  deprecated_rules: etc/deprecated_rules.json
  packages: etc/packages.yaml
  stack_schema_map: etc/stack-schema-map.yaml
  version_lock: etc/version.lock.json
rule_dirs:
- rules
testing:
  config: etc/test_config.yaml
```

--------------------------------

### Set Custom Rules Directory Environment Variable

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

Sets the `CUSTOM_RULES_DIR` environment variable to specify the path to the custom rules directory. This allows the CLI tools to locate and use your custom configurations.

```bash
export CUSTOM_RULES_DIR="<full_path_to_dac_custom_rules_dir>"
```

--------------------------------

### Exporting Rules from Kibana

Source: https://github.com/elastic/dac-reference/blob/main/docs/faq.md

This command exports rules from Kibana to a specified directory, useful for backing up rules before overwriting them. The command includes flags to export specific rule details and configurations.

```bash
kibana export-rules -s -sv -e -ac -d <backup_directory_location>
```

--------------------------------

### Define Email and Slack Actions in TOML

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This TOML snippet demonstrates how to define multiple actions, including email and Slack notifications, for a rule. It includes metadata, action type, group, parameters, and optional fields like ID and frequency.

```toml
[metadata]
creation_date = "2024-02-21"
rule_id = "5d1e96c6-1ee8-4f19-9416-1d8d81428f59"
rule_name = "Example Rule Name"
updated_date = "2024-02-22"
deprecation_date = "2025-01-01"             # optional
comments = "This is an example action list" # optional
maturity = "beta"                           # optional

[[actions]]
action_type_id = ".email"
group = "default"
params.message = "Action triggered: Example Rule Name"
id = "action_001"                           # optional
frequency = { "throttle": "5m" }            # optional

[[actions]]
action_type_id = ".slack"
group = "default"
params.message = "Some other notification"
```

--------------------------------

### Define an Exception List with Items in TOML

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This TOML snippet shows how to define an exception list with multiple items, including metadata and specific entries for exception matching. It associates the list with several rule IDs and names.

```toml
[metadata]
creation_date = "2024/07/10"
list_name = "TestShareList"
rule_ids = ["7c22a9d2-5910-4da2-92af-7ff7481bd0f7", "521629d1-61e4-4203-8a16-a08d5dc20281", "222e1b03-fdc9-42a5-911e-2e3e0533549a"]
rule_names = ["Test Exception List", "Another Test Rule", "DaC Demo Rule"]
updated_date = "2024/07/10"

[[exceptions]]

[exceptions.container]
description = "TestShareList Desc"
list_id = "dbc9b368-5d39-41fa-9a16-bfcb995fc866"
name = "TestShareList"
namespace_type = "single"
tags = []
type = "detection"

[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "dbc9b368-5d39-41fa-9a16-bfcb995fc866"
item_id = "7c823cd0-ca30-46ba-af35-3633219eed1f"
name = "AllOSTestShare"
namespace_type = "single"
tags = []
type = "simple"

[[exceptions.items.entries]]
field = "Effective_process.name"
type = "match"
operator = "included"
value = "BadRooT"
```

--------------------------------

### Export Rules from Repository to ndjson

Source: https://github.com/elastic/dac-reference/blob/main/docs/etoe_reference_example.md

Exports detection rules from the repository to an ndjson file compatible with Kibana rule imports. This command includes action connectors and exceptions, and allows specifying an output file and directory.

```bash
python -m detection_rules export-rules-from-repo \
--directory my-custom-rules \
--outfile my-custom-rules-export.ndjson \
--include-action-connectors \
--include-exceptions
```

--------------------------------

### Kibana Import Rules to Repository Option

Source: https://github.com/elastic/dac-reference/blob/main/docs/known_limitations.md

This option, `--load-rule-loading` or `-lr`, is used with `kibana export-rules` and `import-rules-to-repo` commands. It supports using an existing folder structure when importing rules from a Kibana export, aiding in rule organization.

```bash
kibana export-rules --load-rule-loading / -lr
import-rules-to-repo --load-rule-loading / -lr
```

--------------------------------

### Configure Detection Rules CLI Authentication (API Key)

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This JSON configuration snippet demonstrates authentication using an API key for Kibana. This is the preferred method for securing access to the Elastic Security stack.

```json
{
  "kibana_url": "http://localhost:5601",
  "api_key": "key value"
}
```

--------------------------------

### Import and Validate Rule via Elastic Security API

Source: https://github.com/elastic/dac-reference/blob/main/docs/internals_of_the_detection_rules_repo.md

This method involves using the Detection Engine API to import a rule and then analyzing the API's response for validation feedback. It requires an active Elastic Security instance and allows for direct compatibility checks.

```bash
curl -X POST "<Elasticsearch_URL>:9243/_security/rule/" \
     -H "Content-Type: application/json" \
     -d '@<path_to_rule.json>'

# Analyze the response for errors or issues with the rule definition.
```

--------------------------------

### GitHub Action for Manual Rule Sync

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md

This GitHub Action workflow is designed to manually sync detection rules from Elastic Security to a GitHub repository. It supports exporting rules directly or creating a pull request for review, with configurable Kibana space and sync options. Dependencies include Python and specific GitHub actions for checkout and pull request creation.

```yaml
name: Manually Sync Rules from Elastic Security to GitHub

on:
  workflow_dispatch:
    inputs:
      pr_sync_rules_from_elastic:
        description: 'Sync rules from Elastic Security (direct commit (false) vs PR (true))'
        required: true
        default: 'true'
      space:
        description: 'Specify the Kibana space to export rules from'
        required: false
        default: 'dev'

jobs:
  manual-dispatch-sync:
    runs-on: ubuntu-latest
    env:
      CUSTOM_RULES_DIR: ${{ secrets.CUSTOM_RULES_DIR }}

    steps:
    - name: Checkout Repository
      uses: actions/checkout@v2

    - name: Set up Python 3.12
      uses: actions/setup-python@v2
      with:
        python-version: '3.12'

    - name: Install Dependencies
      run: |
        python -m pip --upgrade pip
        pip cache purge
        pip install .[dev]

    - name: Export Rules from Elastic Security and Import Rules to TOML
      run: |
        python -m detection_rules kibana --space "${{ github.event.inputs.space }}" export-rules --directory ${{ env.CUSTOM_RULES_DIR }}/rules/ -e -ac -s
      env:
        DR_CLOUD_ID: ${{ secrets.ELASTIC_CLOUD_ID }}
        DR_KIBANA_USER: ${{ secrets.ELASTIC_USERNAME }}
        DR_KIBANA_PASSWORD: ${{ secrets.ELASTIC_PASSWORD }}

    - name: Update Version Lock
      run: |
        python -m detection_rules dev update-lock-versions --force

    - name: Create Pull Request
      if: ${{ github.event.inputs.pr_sync_rules_from_elastic == 'true' }}
      uses: peter-evans/create-pull-request@v6
      with:
        token: ${{ secrets.WRITE_GITHUB_TOKEN }}
        commit-message: "Sync rules from Elastic Security"
        title: "Sync rules from Elastic Security"
        body: "This PR syncs rules from Elastic Security to the repository for review."
        branch: "sync-rules-${{ github.run_id }}"
        delete-branch: true
        labels: "auto-update"

    - name: Commit Directly to Main
      if: ${{ github.event.inputs.pr_sync_rules_from_elastic == 'false' }}
      uses: stefanzweifel/git-auto-commit-action@v5
      with:
        commit_message: "Manual dispatch sync"
        branch: main
        file_pattern: "."
        commit_user_name: "GitHub Action"
        commit_user_email: "action@github.com"
      env:
        GITHUB_TOKEN: ${{ secrets.WRITE_GITHUB_TOKEN }}

```

--------------------------------

### Import Detection Rules from Directory

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

Command to import detection rules from a specified directory into Kibana. The '-o' flag forces updates for existing rules. This process loads a configuration file and indicates the number of rules successfully imported.

```bash
python -m detection_rules kibana import-rules -d test-export-rules -o
```

--------------------------------

### List Exported Rule Files

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This command lists the contents of the 'test-export-rules' directory, which was created by the rule export process. It shows the various '.toml' files representing the exported detection rules, as well as the '_errors.txt' file containing details of any issues encountered during export.

```shell
ls test-export-rules

_errors.txt
collection_exchange_mailbox_export_via_powershell.toml.toml
credential_access_multiple_okta_user_auth_events_with_same_device_token_hash_behind_a_proxy.toml.toml
credential_access_potential_okta_mfa_bombing_via_push_notifications.toml.toml
defense_evasion_agent_spoofing_multiple_hosts_using_same_agent.toml.toml
defense_evasion_attempt_to_disable_syslog_service.toml.toml
defense_evasion_kernel_module_removal.toml.toml
discovery_enumeration_of_kernel_modules.toml.toml
execution_interactive_terminal_spawned_via_python.toml.toml
initial_access_multiple_okta_client_addresses_for_a_single_user_session.toml.toml
initial_access_new_okta_authentication_behavior_detected.toml.toml
initial_access_okta_fastpass_phishing_detection.toml.toml
initial_access_okta_sign_in_events_via_third_party_idp.toml.toml
initial_access_okta_user_sessions_started_from_different_geolocations.toml.toml
lateral_movement_multiple_okta_sessions_detected_for_a_single_user.toml.toml
my_first_alert.toml.toml
persistence_new_okta_identity_provider_idp_added_by_admin.toml.toml
test_data_view.toml.toml
test_noisy.toml.toml
test_suppress.toml.toml
web_application_suspicious_activity_post_request_declined.toml.toml
web_application_suspicious_activity_sqlmap_user_agent.toml.toml
web_application_suspicious_activity_unauthorized_method.toml.toml
```

--------------------------------

### Configure Detection Rules CLI Authentication (JSON)

Source: https://github.com/elastic/dac-reference/blob/main/docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

This JSON configuration file specifies connection details for the Elastic Security and Elasticsearch stack, including URLs and credentials. It supports direct credential entry or can be left empty if using API keys or other methods.

```json
{
  "elasticsearch_url": "http://localhost:9200",
  "kibana_url": "http://localhost:5601",
  "cloud_id": "",
  "es_username": "elastic",
  "es_password": "password"
}
```