### Start Dsiem and ELK Stack with Docker Compose

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Pulls the necessary Docker images and starts the Dsiem, ELK stack, and Suricata services. This command brings up the entire testing environment defined in docker-compose.yml.

```shell
docker-compose pull
docker-compose up
```

--------------------------------

### Install Dsiem Server

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Downloads and installs the Dsiem server binary for Linux x86_64 into a specified directory. Requires root privileges. It also sets the DSIEM_DIR environment variable and changes the current directory to the installation path.

```shell
[ "$EUID" -ne 0 ] && echo must be run as root! || ( \
    export DSIEM_DIR=/var/dsiem && \
    mkdir -p $DSIEM_DIR && \
    wget https://github.com/defenxor/dsiem-rs/releases/latest/download/dsiem-server_linux_x86_64.zip -O /tmp/dsiem.zip && \
    unzip /tmp/dsiem.zip -d $DSIEM_DIR && rm -rf /tmp/dsiem.zip  && \
    cd $DSIEM_DIR
    )
```

--------------------------------

### Download and Extract Dsiem Repository

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Downloads the Dsiem project archive from GitHub and navigates into the extracted directory. This is the initial step for setting up the local environment.

```shell
unzip dsiem-rs-master.zip && cd dsiem-rs-master
```

--------------------------------

### Configure Dsiem Backend Systemd Service

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Creates and configures a systemd service file to automatically start, enable, and manage the Dsiem backend. Requires root privileges. It sets environment variables for DSIEM_NODE, DSIEM_MSQ (NATS), and DSIEM_FRONTEND connections.

```shell
[ "$EUID" -ne 0 ] && echo must be run as root! || ( \
  cat <<EOF > /etc/systemd/system/dsiem-backend.service 
  [Unit]
  Description=Dsiem Backend
  After=network.target

  [Service]
  Type=simple
  WorkingDirectory=/var/dsiem
  Environment="DSIEM_NODE=dsiem-backend-0"
  Environment="DSIEM_MSQ=nats://localhost:4222"
  Environment="DSIEM_FRONTEND=nats://localhost:8080"
  ExecStart=/var/dsiem/dsiem-backend serve
  Restart=on-failure

  [Install]
  WantedBy=multi-user.target
  EOF
  systemctl daemon-reload && \
  systemctl enable dsiem-backend.service && \
  systemctl start dsiem-backend.service && \
  systemctl status dsiem-backend.service
  )
```

--------------------------------

### Install NATS Server

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Downloads and installs the NATS server binary for Linux x86_64 into a specified directory. Requires root privileges. It sets the NATS_DIR and NATS_VER environment variables.

```shell
[ "$EUID" -ne 0 ] && echo must be run as root! || ( \
  export NATS_DIR=/var/nats && \
  export NATS_VER=v2.10.10 && \
  mkdir -p $NATS_DIR && cd $NATS_DIR && \
  curl -L https://github.com/nats-io/nats-server/releases/download/${NATS_VER}/nats-server-${NATS_VER}-linux-amd64.zip -o nats-server.zip && \
  unzip nats-server.zip && \
  mv nats-server-${NATS_VER}-linux-amd64/nats-server ./ && \
  rm -rf nats-server.zip nats-server-${NATS_VER}-linux-amd64
  )
```

--------------------------------

### Configure NATS Systemd Service

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Creates and configures a systemd service file to automatically start, enable, and manage the NATS server. Requires root privileges. It specifies the working directory and the executable path for the NATS server.

```shell
[ "$EUID" -ne 0 ] && echo must be run as root! || ( \
  cat <<EOF > /etc/systemd/system/nats.service

  [Unit]
  Description=NATS
  After=network.target

  [Service]
  Type=simple
  WorkingDirectory=/var/nats
  ExecStart=/var/nats/nats-server
  Restart=on-failure

  [Install]
  WantedBy=multi-user.target
  EOF
  systemctl daemon-reload && \
  systemctl enable nats.service && \
  systemctl start nats.service && \
  systemctl status nats.service
  )
```

--------------------------------

### Configure Dsiem Frontend Systemd Service

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Creates and configures a systemd service file to automatically start, enable, and manage the Dsiem frontend. Requires root privileges. It sets environment variables for the DSIEM_NODE and DSIEM_MSQ (NATS) connection.

```shell
[ "$EUID" -ne 0 ] && echo must be run as root! || ( \
  cat <<EOF > /etc/systemd/system/dsiem-frontend.service

  [Unit]
  Description=Dsiem Frontend
  After=network.target

  [Service]
  Type=simple
  WorkingDirectory=/var/dsiem
  Environment="DSIEM_NODE=dsiem-frontend-0"
  Environment="DSIEM_MSQ=nats://localhost:4222"
  ExecStart=/var/dsiem/dsiem-frontend serve
  Restart=on-failure

  [Install]
  WantedBy=multi-user.target
  EOF
  systemctl daemon-reload && \
  systemctl enable dsiem-frontend.service && \
  systemctl start dsiem-frontend.service && \
  systemctl status dsiem-frontend.service
  )
```

--------------------------------

### Start Docker Compose Environment

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

Launches the example Docker environment using docker-compose. Ensure the PROMISC_INTERFACE environment variable is set correctly for your network setup.

```shell
cd deployments/docker && PROMISC_INTERFACE=eth0 docker-compose up
```

--------------------------------

### Serve Dsiem UI with Trunk

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Starts the Dsiem UI development server using Trunk. It serves the application at `/ui` on port `9000`, enabling live reloading for frontend changes.

```shell
cd ./web && trunk serve --public-url /ui --port 9000
```

--------------------------------

### Install Trunk for Rust WebAssembly

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Installs the Trunk build tool, essential for compiling Rust code to WebAssembly for frontend development. This command uses Cargo, Rust's package manager.

```shell
cargo install --locked trunk
```

--------------------------------

### Start Docker Compose Environment

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Launches the Dsiem development environment using Docker Compose. It requires the `PROMISC_INTERFACE` environment variable to be set, typically to a network interface like `eth0`.

```shell
cd ./deployments/docker && \
PROMISC_INTERFACE=eth0 docker-compose up
```

--------------------------------

### Configure Elasticsearch and Kibana URLs

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Configures the `esconfig.json` file to specify the URLs for Elasticsearch and Kibana. It also shows how to include basic authentication credentials for Elasticsearch if required.

```json
{
  "elasticsearch": "http://elasticsearch:9200",
  "kibana": "http://kibana:5601"
}
```

```json
{
  "elasticsearch": "http://username:password@elasticsearch:9200",
  "kibana": "http://kibana:5601"
}
```

--------------------------------

### Import Kibana Dashboard

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Imports a Kibana dashboard JSON file and associated index-patterns. Authentication can be provided via ES_USERNAME and ES_PASSWORD environment variables.

```shell
./scripts/kbndashboard-import.sh ${your-kibana-IP-or-hostname} ./deployments/kibana/dashboard-siem.json
```

```shell
export ES_USERNAME=elastic
export ES_PASSWORD=weak
./scripts/kbndashboard-import.sh ${your-kibana-IP-or-hostname} ./deployments/kibana/dashboard-siem.json
```

--------------------------------

### Import Kibana Dashboard

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Imports the Dsiem Kibana dashboard and its dependencies into Kibana. Ensure indices like 'siem_alarms' and 'siem_events-*' are created by triggering test alarms before importing.

```shell
./scripts/kbndashboard-import.sh localhost ./deployments/kibana/dashboard-siem.json
```

--------------------------------

### Set Promiscuous Network Interface

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Sets the PROMISC_INTERFACE environment variable to specify the network interface Suricata should monitor. Replace 'eth0' with your system's actual network interface name.

```shell
export PROMISC_INTERFACE=eth0
```

--------------------------------

### DSIEM Frontend Serve Command Help

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/commands.md

Details the usage and options for the 'serve' command of the dsiem-frontend executable. This command starts the DSIEM frontend server and allows configuration of its listening IP, port, node name, and other operational parameters. Environment variables can also be used for configuration.

```APIDOC
DSIEM Frontend Serve Command Usage:
  ./dsiem-frontend serve [OPTIONS]

Start Dsiem frontend server

Options:
  -a, --ip-address <ip>
          IP address for the HTTP server to listen on [env: DSIEM_ADDRESS=] [default: 0.0.0.0]
  -p, --tcp-port <tcp>
          TCP port for the HTTP server to listen on [env: DSIEM_PORT=] [default: 8080]
  -n, --node <string>
          Unique node name to use when deployed in cluster mode [env: DSIEM_NODE=]
  -e, --max_eps <number>
          Maximum events per second [env: MAX_EPS=] [default: 1000]
  -i, --min_eps <number>
          Minimum events per second [env: MIN_EPS=] [default: 100]
      --msq <string>
          Nats address to use for frontend - backend communication [env: DSIEM_MSQ=] [default: nats://dsiem-nats:4222]
  -q, --max_queue <events>
          Length of queue for unprocessed events, setting this to 0 will use 1,000,000 events to emulate unbounded queue [env: DSIEM_MAXQUEUE=] [default: 25000]
      --writable-config
          Whether to allow configuration file update through HTTP [env: DSIEM_WRITEABLECONFIG=]
  -s, --status <comma separated strings>
          Alarm status to use, the first one will be assigned to new alarms [env: DSIEM_STATUS=] [default: Open,In-Progress,Closed]
  -t, --tags <comma separated strings>
          Alarm tags to use, the first one will be assigned to new alarms [env: DSIEM_TAGS=] [default: "Identified Threat,False Positive,Valid Threat,Security Incident"]
  -h, --help
          Print help (see more with '--help')

Example Usage:
  ./dsiem-frontend serve --node=dsiem-0
  export DSIEM_NODE=dsiem-0; ./dsiem-frontend serve

Environment Variable Configuration:
All flags can be configured through environment variables. For example, to set the node name, use export DSIEM_NODE=dsiem-0. The prefix DSIEM_ is used for most parameters, with exceptions like MAX_EPS and MIN_EPS.
```

--------------------------------

### Set Filebeat Config Ownership to Root

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Changes the ownership of Filebeat configuration files to root. This is a security requirement for Beats components to operate correctly.

```shell
cd deployments/docker && \
  sudo chown root $(find conf/filebeat/ conf/filebeat-es/ -name "*.yml")
```

--------------------------------

### Develop dsiem-rs Plugin: Setup and Build

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

Steps to set up a Rust development environment, clone the dsiem-rs repository, and build the project. This is the initial phase for developing custom plugins.

```Shell
git clone https://github.com/defenxor/dsiem-rs
cd dsiem-rs
cargo build
```

--------------------------------

### JSON Configuration Example for Intel Plugin

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

An example JSON configuration file for an IP-based threat intelligence plugin. It specifies the plugin name, type, and configuration details, including a URL for the threat intelligence source.

```JSON
{
  "intel_sources": [
    {
      "name": "Wise",
      "plugin": "Wise",
      "type": "IP",
      "enabled": true,
      "config": "{ \"url\" : \"http://wise:8081/ip/${ip}\" }"
    }
  ]
}
```

--------------------------------

### Uninstall Dsiem (Docker Compose)

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/installation.md

Uninstalls Dsiem when installed via Docker Compose by stopping and removing containers and volumes.

```shell
cd dsiem/deployments/docker && \
docker-compose down -v
```

--------------------------------

### Initial dpluger Configuration Template

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

An example of the initial JSON configuration file generated by the `dpluger create` command. This file serves as a template that needs to be customized with specific field mappings and identifiers.

```json
{
  "name": "suricata",
  "type": "SID",
  "output_file": "70_siem-plugin-suricata.conf",
  "index_pattern": "suricata-*",
  "elasticsearch_address": "http://elasticsearch:9200",
  "identifier_field": "INSERT_LOGSTASH_IDENTIFYING_FIELD_HERE (example: [application] or [fields][log_type] etc)",
  "identifier_value": "INSERT_IDENTIFYING_FIELD_VALUE_HERE (example: suricata)",
  "identifier_filter": "INSERT_ADDITIONAL_FILTER_HERE_HERE (example: and [alert])",
  "field_mapping": {
    "title": "es:INSERT_ES_FIELDNAME_HERE",
    "timestamp": "es:INSERT_ES_FIELDNAME_HERE",
    "timestamp_format": "INSERT_TIMESTAMP_FORMAT_HERE (example: ISO8601)",
    "sensor": "es:INSERT_ES_FIELDNAME_HERE",
    "plugin_id": "INSERT_PLUGIN_NUMBER_HERE",
    "plugin_sid": "es:INSERT_ES_FIELDNAME_HERE or collect:INSERT_ES_FIELDNAME_HERE",
    "product": "INSERT_PRODUCT_NAME_HERE",
    "src_ip": "es:INSERT_ES_FIELDNAME_HERE",
    "src_port": "es:INSERT_ES_FIELDNAME_HERE",
    "dst_ip": "es:INSERT_ES_FIELDNAME_HERE",
    "dst_port": "es:INSERT_ES_FIELDNAME_HERE",
    "protocol": "es:INSERT_ES_FIELDNAME_HERE or INSERT_PROTOCOL_NAME_HERE"
  }
}
```

--------------------------------

### Prepare and Import Kibana Dashboard

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Copies Kibana dashboard configuration files, modifies the port number from `8080` to `9000` to match the Trunk server, and then imports the updated dashboard into Kibana.

```shell
cp ./deployments/kibana/* /tmp/
sed -i 's/8080/9000/' /tmp/dashboard-siem.json
./scripts/kbndashboard-import.sh localhost /tmp/dashboard-siem.json
```

--------------------------------

### Example filter.conf Content

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

Demonstrates the YAML content for a filter configuration file (`filter.conf`) that can be embedded into the dpluger configuration. It includes conditional logic and mutate operations.

```yaml
if [@metadata][log_type] == "sshd" and [sshd_result]
{
  mutate {
    id => "tag normalizedEvent 10008"
    add_field => {
      "[@metadata][siem_plugin_type]" => "sshd"
      "[@metadata][siem_data_type]" => "normalizedEvent"
    }
  }
}
```

--------------------------------

### Setup Configuration Directory for Filter Comparison

Source: https://github.com/defenxor/dsiem-rs/blob/master/scripts/filter-comparison/Readme.md

Creates the target debug configuration directory and symlinks the directive.json file. This is a prerequisite for running the performance comparison tool.

```shell
mkdir -p ./target/debug/configs && ln -s /absolute/path/to/directive.json ./target/debug/configs/directive.json
```

--------------------------------

### dpluger Execution Output

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

Example output from the `dpluger run` command, showing the process of checking field existence in Elasticsearch and confirming the successful creation of the Logstash configuration file.

```shell
Creating plugin (logstash config) for suricata, using ES: http://elasticsearch:9200 and index pattern: suricata-*
2018-11-24T22:52:32.686+0700    INFO    Found ES version 6.4.2
Checking existence of field alert.signature... OK
Checking existence of field timestamp... OK
Checking existence of field host.name... OK
Checking existence of field alert.signature_id... OK
Checking existence of field alert.category... OK
Checking existence of field src_ip... OK
Checking existence of field src_port... OK
Checking existence of field dest_ip... OK
Checking existence of field dest_port... OK
Checking existence of field proto... OK
Logstash conf file created.
```

--------------------------------

### Run Tailwind CSS in Watch Mode

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Starts the Tailwind CSS build process in development mode, watching for changes in source files and automatically recompiling CSS. This script is located in the project's scripts directory.

```shell
./scripts/css.sh dev
```

--------------------------------

### Verify Build Tools

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/building.md

Checks if the necessary build tools, Cargo (Rust's package manager) and npm (Node.js package manager), are installed and accessible in the system's PATH.

```shell
cargo version
npm -v
```

--------------------------------

### Dsiem Backend Healthy Log Example

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/managing_performance.md

Example log output from a Dsiem backend node that is performing well, characterized by a low queue length and fast average processing times. This indicates the node is efficiently handling the ingestion rate.

```json
{"message":"watchdog report","eps":1615.38,"queue_length":5,"avg_proc_time_ms":0.023,"ttl_directives":5001,"active_directives":70,"backlogs":552,"timedout_backlogs":0}
{"message":"watchdog report","eps":1565.22,"queue_length":11,"avg_proc_time_ms":0.025,"ttl_directives":5001,"active_directives":70,"backlogs":580,"timedout_backlogs":0}
{"message":"watchdog report","eps":1545.45,"queue_length":1,"avg_proc_time_ms":0.028,"ttl_directives":5001,"active_directives":70,"backlogs":601,"timedout_backlogs":0}
{"message":"watchdog report","eps":1535.07,"queue_length":1,"avg_proc_time_ms":0.03,"ttl_directives":5001,"active_directives":70,"backlogs":615,"timedout_backlogs":0}
{"message":"watchdog report","eps":1531.58,"queue_length":0,"avg_proc_time_ms":0.031,"ttl_directives":5001,"active_directives":70,"backlogs":621,"timedout_backlogs":0}
```

--------------------------------

### DSIEM Directive Structure Example

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/directive_and_alarm.md

Illustrates a dsiem directive with multiple PluginRule definitions, showcasing stages, occurrence counts, and timeout configurations for event processing.

```json
{
  "name": "Ping Flood from SRC_IP",
  "kingdom": "Reconnaissance & Probing",
  "category": "Misc Activity",
  "id": 1,
  "priority": 3,
  "rules" : [
    { "name": "ICMP Ping", "type": "PluginRule", "stage": 1, "plugin_id": 1001, 
      "plugin_sid": [ 2100384 ], "occurrence": 1, "from": "HOME_NET", "to": "ANY",
      "port_from": "ANY", "port_to": "ANY", "protocol": "ICMP", 
      "reliability": 1, "timeout": 0 
    },
    { "name": "ICMP Ping", "type": "PluginRule", "stage": 2, "plugin_id": 1001,
      "plugin_sid": [ 2100384 ], "occurrence": 5, "from": ":1", "to": "ANY",
      "port_from": "ANY", "port_to": "ANY", "protocol": "ICMP",
      "reliability": 5, "timeout": 600 
    },
    { "name": "ICMP Ping", "type": "PluginRule", "stage": 3, "plugin_id": 1001,
      "plugin_sid": [ 2100384 ], "occurrence": 500, "from": ":1", "to": "ANY", 
      "port_from": "ANY", "port_to": "ANY", "protocol": "ICMP",
      "reliability": 10, "timeout": 3600
    }
  ]
}
```

--------------------------------

### Run dsiem Backend and Filter Logs

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

Starts the dsiem-backend service and filters its output for 'intel' related entries to verify that the plugin is loaded. The output shows the path of the loaded configuration file.

```shell
cargo run --bin dsiem-backend -- serve -n dsiem-backend-0 -f http://localhost:8080 2>&1 | grep intel
```

--------------------------------

### Add Rust WASM Target

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Adds the `wasm32-unknown-unknown` target to the Rust toolchain. This target is required for compiling Rust code to WebAssembly.

```shell
rustup target add wasm32-unknown-unknown
```

--------------------------------

### Dsiem Correlation Rule Structure Example

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/differences_from_ossim.md

Demonstrates the linear structure of Dsiem correlation rules and the use of the 'all_rules_always_active' flag to maintain state across rule levels, mimicking the behavior of multiple paths in OSSIM rules.

```text
"all_rules_always_active": true
level 1: SSH failed login (occurrence: 1)
level 2: SSH failed login (occurrence: 4, timeout=300s)
level 3: SSH succesful login (occurrence: 1, timeout=300s)
```

--------------------------------

### Dsiem Backend Lagging Log Example

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/managing_performance.md

Example log output from a Dsiem backend node experiencing performance issues, indicated by a consistently rising queue length and skipped events. This suggests the node is not keeping up with the ingestion rate.

```json
{"message":"watchdog report","eps":1636.36,"queue_length":8733,"avg_proc_time_ms":0.035,"ttl_directives":5001,"active_directives":70,"backlogs":284,"timedout_backlogs":0}
{"message":"watchdog report","eps":1571.43,"queue_length":18530,"avg_proc_time_ms":0.037,"ttl_directives":5001,"active_directives":70,"backlogs":290,"timedout_backlogs":0}
{"message":"watchdog report","eps":1548.39,"queue_length":28382,"avg_proc_time_ms":0.039,"ttl_directives":5001,"active_directives":70,"backlogs":295,"timedout_backlogs":0}
{"message":"filtering lagged and skipped 2 events"}
{"message":"filtering lagged and skipped 1 events"}
{"message":"filtering lagged and skipped 1 events"}
{"message":"filtering lagged and skipped 2 events"}
```

--------------------------------

### Customized Suricata SIEM Plugin Configuration

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

An example of a customized JSON configuration file for a Suricata SIEM plugin, tailored for Suricata Eve JSON format data in Elasticsearch. This includes specific field mappings and identifiers.

```json
{
  "name": "suricata",
  "type": "SID",
  "output_file": "70_siem-plugin-suricata.conf",
  "index_pattern": "suricata-*",
  "elasticsearch_address": "http://elasticsearch:9200",
  "identifier_field": "[application]",
  "identifier_value": "suricata",
  "identifier_filter": "and [alert]",
  "field_mapping": {
    "title": "es:alert.signature",
    "timestamp": "es:timestamp",
    "timestamp_format": "ISO8601",
    "sensor": "es:host.name",
    "plugin_id": "1001",
    "plugin_sid": "es:alert.signature_id",
    "product": "Intrusion Detection System",
    "category": "es:alert.category",
    "src_ip": "es:src_ip",
    "src_port": "es:src_port",
    "dst_ip": "es:dest_ip",
    "dst_port": "es:dest_port",
    "protocol": "es:proto"
  }
}
```

--------------------------------

### OSSIM Correlation Rule Structure Example

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/differences_from_ossim.md

Illustrates the hierarchical and branching structure of correlation rules in OSSIM, showing how multiple paths (e.g., 3a and 3b) can be defined within a single directive to capture different event sequences or additional data.

```text
level 1: SSH failed login (occurrence: 1)
  level 2: SSH failed login (occurrence: 4, timeout: 300s)
    level 3a: SSH succesful login (occurrence: 1, timeout: 300s)
    level 3b: SSH Failed login (occurrence: 1000, timeout: 300s)
```

--------------------------------

### Trigger Alarm by Pinging Interface

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Triggers an alarm in the Dsiem system by pinging an IP address associated with a specified network interface (e.g., `eth0`). This is used to test the alarm detection mechanism.

```shell
ip a | grep inet | grep eth0
ping [the ip address]
```

--------------------------------

### Filebeat Configuration: Application Identifier

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/event_processing.md

Example of a Filebeat configuration snippet that adds an application identifier field to events, used by Logstash for routing.

```yaml
{application: suricata}
```

--------------------------------

### Monitor Dsiem Backend Memory Usage

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/managing_performance.md

Example shell command to display resident memory usage for dsiem-backend processes. This helps compare memory footprints between different configurations, such as with or without preloading directives.

```shell
ps -axo rss,cmd | grep -v grep | grep dsiem-backend
# Example Output:
# 400736 /dsiem/dsiem-backend serve
# 1196992 /dsiem/dsiem-backend serve --preload-all-directives true
```

--------------------------------

### Set Root Directory Variable

Source: https://github.com/defenxor/dsiem-rs/blob/master/web/readme.md

Sets an environment variable `ROOT_DIR` to the absolute path of the Git repository's top-level directory. This is useful for referencing files and directories relative to the project root.

```shell
export ROOT_DIR=$(git rev-parse --show-toplevel)
echo $ROOT_DIR
```

--------------------------------

### Rust VulnChecker Trait Definition

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

Defines the `VulnChecker` trait for vulnerability lookup plugins. It features an `initialize` method for setup and a `check_ip_port` method to query IP and port combinations against vulnerability databases.

```Rust
pub trait VulnChecker: Send + Sync {
    async fn check_ip_port(&self, ip: IpAddr, port: u16) -> Result<HashSet<VulnResult>>;
    fn initialize(&mut self, config: String) -> Result<();
}
```

--------------------------------

### Stream and Filter Dsiem Watchdog Logs

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/telemetry.md

This example shows how to stream logs from a Dsiem backend container and filter them using `jq` to display only messages related to 'dsiem::watchdog'. This is useful for real-time monitoring of Dsiem's operational metrics.

```shell
docker logs -f dsiem-backend-0 --since=10s | jq -c --unbuffered '. | select(.target=="dsiem::watchdog")'
```

```json
{"timestamp":"2024-03-26T04:44:50.978559Z","level":"INFO","fields":{"message":"watchdog report","eps":7530.67,"queue_length":0,"avg_proc_time_ms":0.006,"directives_ttl":23691,"directives_active":42,"events_rcvd":45495,"events_matched":755,"backlogs":42,"backlogs_timedout":0},"target":"dsiem::watchdog"}
```

--------------------------------

### Generate Dsiem Directives using dpluger CLI

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/directive_and_alarm.md

This command uses the dpluger tool to generate Dsiem directives from a TSV file. The `-i` flag specifies the starting directive ID, and `-f` points to the input TSV file containing plugin and SID information.

```console
./dpluger directive -i 3001 -f sangfor_plugin-sids.tsv
```

--------------------------------

### Dsiem Backend High Processing Time Log Example

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/managing_performance.md

Log output from a Dsiem backend node where average processing time is high, potentially indicating it will not sustain the target ingestion rate if it increases. This might be due to configuration or environmental errors.

```json
{"message":"watchdog report","eps":578.84,"queue_length":132,"avg_proc_time_ms":12.222,"ttl_directives":12434,"active_directives":432,"backlogs":463,"timedout_backlogs":431}
{"message":"avg. processing time maybe too long to sustain the target 1000 event/sec (or 1.000 ms/event)","avg_proc_time_ms":12.222}
{"message":"watchdog report","eps":581.19,"queue_length":11,"avg_proc_time_ms":25.3,"ttl_directives":12434,"active_directives":438,"backlogs":469,"timedout_backlogs":437}
{"message":"avg. processing time maybe too long to sustain the target 1000 event/sec (or 1.000 ms/event)","avg_proc_time_ms":25.3}
```

--------------------------------

### DSIEM Frontend CLI General Help

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/commands.md

Provides general help and usage information for the dsiem-frontend executable. It lists available commands and global options. Configuration can be managed via command-line flags or environment variables prefixed with DSIEM_.

```APIDOC
DSIEM Frontend CLI Usage:
  ./dsiem-frontend [COMMAND]

Commands:
  serve  Start Dsiem frontend server
  help   Print this message or the help of the given subcommand(s)

Options:
  -v, --verbosity...  Increase logging verbosity
      --debug         Enable debug output, for compatibility purpose [env: DSIEM_DEBUG=]
      --trace         Enable trace output, for compatibility purpose [env: DSIEM_TRACE=]
  -j, --json          Enable json-lines log output [env: DSIEM_JSON=]
      --test-env      Testing environment flag
  -h, --help          Print help (see more with '--help')
  -V, --version       Print version

Environment Variable Configuration:
Flags can be configured via environment variables. For example, to set the debug flag, use export DSIEM_DEBUG=true. The prefix DSIEM_ is common for most parameters.
```

--------------------------------

### Build with Environment Variables

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/building.md

Demonstrates how to use environment variables to customize the build process, such as enabling dynamic linking with libgcc or using a nightly Rust toolchain with specific features like sanitizers.

```shell
DSIEM_DYNAMIC_LIBGCC=1 DSIEM_NIGHTLY_RUST=1 ./scripts/build-glibc.sh
```

```shell
DSIEM_NIGHTLY_RUST=1 ./scripts/dockerbuild.sh defenxor/dsiem-rs alpine
```

--------------------------------

### Build Web UI

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/building.md

Compiles the web user interface for the project in production mode. The built static assets are placed in the `./web/dist` directory.

```shell
./scripts/build-web.sh prod
```

--------------------------------

### Build Docker Image

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/building.md

Uses a helper script to build a Docker image for the dsiem-rs project. It allows specifying a base image and tags the resulting image appropriately.

```shell
./scripts/dockerbuild.sh defenxor/dsiem-rs [base_image]
```

--------------------------------

### Build Debug Binaries

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/building.md

Compiles both the dsiem-backend and dsiem-frontend components in debug mode using Cargo. The resulting executables are placed in the `./target/debug/` directory.

```shell
cargo build
```

--------------------------------

### Build Release Binaries (Glibc)

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/building.md

Builds the release version of the dsiem-backend and dsiem-frontend binaries, specifically targeting glibc for broader Linux compatibility. The output is located in the cross-compilation release directory.

```shell
./scripts/build-glibc.sh
```

--------------------------------

### Build dsiem Binary

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

Compiles the dsiem project using Cargo to include the latest plugin changes. This is the initial step before running the service.

```shell
cargo build
```

--------------------------------

### Create dpluger Configuration File

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

Command to generate an initial dpluger configuration file for a Suricata SIEM plugin. It specifies Elasticsearch connection details, index pattern, and plugin name.

```shell
./dpluger create -a http://elasticsearch:9200 -i "suricata-*" -n "suricata" -c dpluger_suricata.json
```

--------------------------------

### Run Filter Comparison Application

Source: https://github.com/defenxor/dsiem-rs/blob/master/scripts/filter-comparison/Readme.md

Builds and executes the filter-comparison application. Using the release build is recommended for more accurate performance measurements.

```shell
cargo run -p filter-comparison
```

```shell
cargo build --release -p filter-comparison && ./target/release/filter-comparison
```

--------------------------------

### dpluger Command for Taxonomy Plugin Template

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

Shell command to initiate the creation of a taxonomy-based SIEM plugin template using dpluger. The '-t Taxonomy' flag specifies the plugin type.

```shell
./dpluger create -i firewall-* -t Taxonomy
```

--------------------------------

### dpluger Taxonomy Plugin Template Config

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

JSON configuration template generated by dpluger for a taxonomy-based plugin. It includes placeholders for essential fields like product, category, and subcategory mapping.

```json
{
  "name": "suricata",
  "type": "Taxonomy",
  "output_file": "70_siem-plugin-suricata.conf",
  "index_pattern": "firewall-*",
  "elasticsearch_address": "http://elasticsearch:9200",
  "identifier_field": "INSERT_LOGSTASH_IDENTIFYING_FIELD_HERE (example: [application] or [fields][log_type] etc)",
  "identifier_value": "INSERT_IDENTIFYING_FIELD_VALUE_HERE (example: suricata)",
  "identifier_filter": "INSERT_ADDITIONAL_FILTER_HERE_HERE (example: and [alert])",
  "field_mapping": {
    "title": "es:INSERT_ES_FIELDNAME_HERE",
    "timestamp": "es:INSERT_ES_FIELDNAME_HERE",
    "timestamp_format": "INSERT_TIMESTAMP_FORMAT_HERE (example: ISO8601)",
    "sensor": "es:INSERT_ES_FIELDNAME_HERE",
    "product": "INSERT_PRODUCT_NAME_HERE",
    "category": "es:INSERT_ES_FIELDNAME_HERE",
    "subcategory": "es:INSERT_ES_FIELDNAME_HERE",
    "src_ip": "es:INSERT_ES_FIELDNAME_HERE",
    "src_port": "es:INSERT_ES_FIELDNAME_HERE",
    "dst_ip": "es:INSERT_ES_FIELDNAME_HERE",
    "dst_port": "es:INSERT_ES_FIELDNAME_HERE",
    "protocol": "es:INSERT_ES_FIELDNAME_HERE or INSERT_PROTOCOL_NAME_HERE"
  }
}
```

--------------------------------

### Web UI Elasticsearch/Kibana Configuration

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/faq.md

Defines the structure for the `esconfig.json` file, which specifies the Elasticsearch and Kibana endpoints for the DSIEM-RS web UI. It includes placeholders for user credentials, essential for connecting to secured Elasticsearch clusters.

```json
{
  "elasticsearch": "http(s)://${UID}:${PASSWD}@address-reachable-from-the-browser:9200/",
  "kibana": "http(s)://address-reachable-from-the-browser:5601/"
}
```

--------------------------------

### Run dpluger to Generate Logstash Config

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

Command to execute the dpluger tool with a specified configuration file. This process verifies field existence in Elasticsearch and generates a Logstash configuration file for the SIEM plugin.

```shell
./dpluger run -c dpluger_suricata.json
```

--------------------------------

### Configure Directive Loading

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/managing_performance.md

Manage how Dsiem loads backlog managers for directives. Options include dynamic loading (default) or pre-loading all directives at startup using the --preload-all-directives flag, affecting memory usage and reaction time.

```APIDOC
--idle-directives-timeout: Timeout in seconds for unloading idle directives (default behavior).
--preload-all-directives [true|false]: Flag to preload all directives at startup. Setting to 'true' increases memory usage but ensures stable reaction times.
```

--------------------------------

### Dsiem Architecture Diagram

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/adv_deployment.md

Visual representation of the Dsiem architecture, illustrating the flow of normalized events from Logstash through frontend nodes, NATS messaging, to backend nodes, and their interaction with vulnerability and intelligence sources.

```mermaid
flowchart TB

l1 --> |Normalized Event| f1
l1 --> |Normalized Event| f2
l1 --> |Normalized Event| f3

f1 --> |Normalized Event| m1

m1 --> |Normalized Event| b1
m1 --> |Normalized Event| b2
m1 --> |Normalized Event| b3

b1 --> |Query+Cache| v1
b1 --> |Query+Cache| v2

b1 --> |Alarm|fb

subgraph Logstash
  l1[Logstash] 
end

subgraph Dsiem
  subgraph Frontend
    f1[fe1]
    f2[fe2]
    f3[fe..N]
  end
  subgraph Message Queue
    m1[nats1]
    m2[nats2]
    m3[nats..N]
  end
  subgraph Backend
    b1[be1]
    b2[be2]
    b3[be..N]
  end
end

subgraph Vuln and Intel Sources
  v1[Moloch-Wise]
  v2[Nessus-CSV]
  v3[Others]
end

subgraph Filebeat
  fb[Filebeat]
end

```

--------------------------------

### Using Dsiem without Elastic Stack (Vector & Proxy)

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/faq.md

Provides guidance on how to use Dsiem as a correlation engine without relying on the full Elastic stack. It suggests using Vector for log normalization and HTTP sinks, and a special-purpose proxy for handling Elasticsearch's `scripted upsert` requirements.

```text
So how to use Dsiem without Elastic stack?

First use something else other than Logstash for normalizing your logs in accordance to Dsiem [normalized event specification](https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md#normalized-event). For instance, you can use [Vector](https://vector.dev/) for this purpose.

Next, send those normalized events to Dsiem through HTTP. For Vector, that means using those events as input to an [HTTP sink](https://vector.dev/docs/reference/configuration/sinks/http/).

Finally, substitute Filebeat with something else to read Dsiem output (`siem_alarms.json`), and send it to the final storage or notification destination. The HTTP and Elasticsearch sinks in Vector do not support [`scripted upsert`](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#scripted_upsert) in a way that Logstash does, so as a workaround there is a special-purpose [proxy](https://github.com/mmta/dsiem-esproxy) for this.

A working example of Dsiem deployment without Logstash is provided [here](https://github.com/mmta/dsiem-esproxy/tree/master/examples/elasticsearch). There's also [another one](https://github.com/mmta/dsiem-esproxy/tree/master/examples/surrealdb) that also replaces Elasticseach with Surrealdb.
```

--------------------------------

### Embedding Identifier Block Source Configuration

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

Shows how to configure `identifier_block_source` in a dpluger configuration file to embed content from an external file. This bypasses `identifier_field`, `identifier_value`, and `identifier_filter`.

```json
...
"identifier_block_source": "filter.conf",
...
```

--------------------------------

### dpluger SID-based Plugin Config (McAfee NSP)

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/dsiem_plugin.md

JSON configuration for a SID-based SIEM plugin. It maps event fields and uses 'collect:signature' to dynamically generate plugin_sid values from unique signature entries.

```json
{
  "name": "mcafee-nsp",
  "type": "SID",
  "output_file": "70_siem-plugin-mcafee-nsp.conf",
  "index_pattern": "mcafee-nsp-*",
  "elasticsearch_address": "http://elasticsearch:9200",
  "identifier_field": "[fields][log_type]",
  "identifier_value": "mcafee-nsp",
  "identifier_filter": "",
  "field_mapping": {
    "title": "es:signature",
    "timestamp": "es:syslog_timestamp",
    "timestamp_format": "ISO8601",
    "sensor": "es:device_hostname",
    "plugin_id": "1832",
    "plugin_sid": "collect:signature",
    "product": "Intrusion Prevention System",
    "src_ip": "es:src_ip",
    "src_port": "es:src_port",
    "dst_ip": "es:dst_ip",
    "dst_port": "es:dst_port",
    "protocol": ""
  }
}
```

--------------------------------

### Copy Plugin Configuration

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

Copies the new plugin configuration file to the expected location within the target debug directory. This makes the plugin discoverable by the dsiem backend.

```shell
mkdir -p target/debug/configs && cp configs/intel_myintel.json target/debug/configs/
```

--------------------------------

### Develop dsiem-rs Plugin: Create New Intel Plugin

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/ti_vuln_plugins.md

Commands to create a new threat intelligence plugin by copying an existing one (Wise) and modifying its name and references. This includes updating source files and configuration.

```Shell
# prepare the new plugin files based on wise
cp server/src/intel/plugins/wise.rs server/src/intel/plugins/myintel.rs

# replace wise -> myintel and Wise -> Myintel in the code
sed -i 's/wise/myintel/g; s/Wise/Myintel/g' server/src/intel/plugins/myintel.rs

# do the same for config file
cp configs/intel_wise.json configs/intel_myintel.json
sed -i 's/Wise/Myintel/g; s/wise/myintel/g' configs/intel_myintel.json

# insert entry in intel plugins
export MODFILE=server/src/intel/plugins/mod.rs
sed -i '4 i mod myintel;' ${MODFILE}
sed -i '24 i let myintel = Checker::new(Box::<myintel::Myintel>::default(), "Myintel");' ${MODFILE}
sed -i 's/vec![wise]/vec![wise, myintel]/' ${MODFILE}
```

--------------------------------

### Run Dsiem Plugin with Configuration

Source: https://github.com/defenxor/dsiem-rs/blob/master/docs/managing_performance.md

Command to run a Dsiem plugin, potentially applying configurations from a dplugin_config.json file. This is used for selectively ingesting logs by filtering events before they reach Dsiem.

```shell
dpluger run
```