parsedmarc

https://domainaware.github.io/parsedmarc/
Admin
parsedmarc is an open source Python module and CLI utility for parsing DMARC reports, with support...
Tokens:89,591
Snippets:490
Trust Score:9.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Enable and Start Elasticsearch and Kibana Services

Source: https://domainaware.github.io/parsedmarc/elasticsearch.html

Reloads systemd, enables, and starts the Elasticsearch and Kibana services. This should be run after installation and configuration.

```bash
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl enable kibana.service
sudo systemctl start elasticsearch.service
sudo systemctl start kibana.service

```

--------------------------------

### Install Java JRE

Source: https://domainaware.github.io/parsedmarc/_sources/davmail.md.txt

Install the default headless Java Runtime Environment required for running DavMail.

```bash
sudo apt-get install default-jre-headless
```

--------------------------------

### Install Elasticsearch and Kibana on Debian/Ubuntu

Source: https://domainaware.github.io/parsedmarc/_sources/elasticsearch.md.txt

Commands to add the Elastic repository and install the required packages on Debian or Ubuntu systems.

```bash
sudo apt-get install -y apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update
sudo apt-get install -y elasticsearch kibana
```

--------------------------------

### Install msgconvert on Debian or Ubuntu

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Use this command to install the libemail-outlook-message-perl package, enabling support for OLE .msg files.

```bash
sudo apt-get install libemail-outlook-message-perl
```

--------------------------------

### Install geoipupdate on Ubuntu with PPA

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Installs the geoipupdate package on Ubuntu systems by adding the MaxMind PPA. This ensures you get the latest version.

```bash
sudo add-apt-repository ppa:maxmind/ppa
sudo apt update
sudo apt install -y geoipupdate
```

--------------------------------

### Finalize Kibana Security Setup

Source: https://domainaware.github.io/parsedmarc/elasticsearch.html

Set passwords, generate encryption keys, and update the configuration file.

```bash
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
sudo /usr/share/kibana/bin/kibana-encryption-keys generate
sudo vim /etc/kibana/kibana.yml
```

--------------------------------

### Install Elasticsearch and Kibana on Debian/Ubuntu

Source: https://domainaware.github.io/parsedmarc/elasticsearch.html

Installs Elasticsearch and Kibana on Debian/Ubuntu systems. Ensure Elasticsearch and Kibana 6 or later are required.

```bash
sudo apt-get install -y apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update
sudo apt-get install -y elasticsearch kibana

```

--------------------------------

### Manage Elasticsearch and Kibana Services

Source: https://domainaware.github.io/parsedmarc/_sources/elasticsearch.md.txt

Commands to reload systemd, enable services at boot, and start the Elasticsearch and Kibana services.

```bash
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl enable kibana.service
sudo systemctl start elasticsearch.service
sudo systemctl start kibana.service
```

--------------------------------

### Install Java for DavMail

Source: https://domainaware.github.io/parsedmarc/davmail.html

Install the default Java Runtime Environment headless version on Debian-based systems. This is a prerequisite for running DavMail.

```bash
sudo apt-get install default-jre-headless

```

--------------------------------

### Install geoipupdate on CentOS/RHEL

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Installs the geoipupdate package on CentOS or RHEL systems using dnf.

```bash
sudo dnf install -y geoipupdate
```

--------------------------------

### Install Python 3 and Development Packages on Debian/Ubuntu

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Installs Python 3, pip, virtualenv, and necessary development libraries for building Python packages on Debian or Ubuntu systems.

```bash
sudo apt-get install -y python3-pip python3-virtualenv python3-dev libxml2-dev libxslt-dev
```

--------------------------------

### Configure Syslog Output

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

Examples for configuring Syslog output using different protocols and authentication methods in an INI file.

```ini
[syslog]
server = syslog.example.com
port = 514
```

```ini
[syslog]
server = syslog.example.com
port = 6514
protocol = tcp
timeout = 10.0
retry_attempts = 5
```

```ini
[syslog]
server = syslog.example.com
port = 6514
protocol = tls
cafile_path = /path/to/ca-cert.pem
timeout = 10.0
```

```ini
[syslog]
server = syslog.example.com
port = 6514
protocol = tls
cafile_path = /path/to/ca-cert.pem
certfile_path = /path/to/client-cert.pem
keyfile_path = /path/to/client-key.pem
timeout = 10.0
retry_attempts = 3
retry_delay = 5
```

--------------------------------

### Install geoipupdate on Debian/Ubuntu

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Installs the geoipupdate package on Debian or Ubuntu systems. Ensure the 'contrib' component is enabled in apt sources.

```bash
sudo apt-get install -y geoipupdate
```

--------------------------------

### Multi-tenant Configuration Example

Source: https://domainaware.github.io/parsedmarc/usage.html

Define tenant names and associated domains in a YAML file for multi-tenant support. This mapping is used by parsedmarc to prefix index names with tenant identifiers.

```yaml
example:
  - example.com
  - example.net
  - example.org

whalensolutions:
  - whalensolutions.com

```

--------------------------------

### DavMail Configuration Properties

Source: https://domainaware.github.io/parsedmarc/_sources/davmail.md.txt

Example configuration file for DavMail to act as an EWS/OWA to IMAP gateway.

```properties
# DavMail settings, see http://davmail.sourceforge.net/ for documentation

#############################################################
# Basic settings

# Server or workstation mode
davmail.server=true

# connection mode auto, EWS or WebDav
davmail.enableEws=auto

# base Exchange OWA or EWS url
davmail.url=https://outlook.office365.com/EWS/Exchange.asmx

# Listener ports
davmail.imapPort=1143

#############################################################
# Network settings

# Network proxy settings
davmail.enableProxy=false
davmail.useSystemProxies=false
davmail.proxyHost=
davmail.proxyPort=
davmail.proxyUser=
davmail.proxyPassword=

# proxy exclude list
davmail.noProxyFor=

# block remote connection to DavMail
davmail.allowRemote=false

# bind server sockets to the loopback address
davmail.bindAddress=127.0.0.1

# disable SSL for specified listeners
davmail.ssl.nosecureimap=true

# Send keepalive character during large folder and messages download
davmail.enableKeepalive=true

# Message count limit on folder retrieval
davmail.folderSizeLimit=0

#############################################################
# IMAP settings

# Delete messages immediately on IMAP STORE \Deleted flag
davmail.imapAutoExpunge=true

# Enable IDLE support, set polling delay in minutes
davmail.imapIdleDelay=1

# Always reply to IMAP RFC822.SIZE requests with Exchange approximate
# message size for performance reasons
davmail.imapAlwaysApproxMsgSize=true

# Client connection timeout in seconds - default 300, 0 to disable
davmail.clientSoTimeout=0

#############################################################
```

--------------------------------

### Install Python 3 and Development Packages on CentOS/RHEL

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Installs Python 3 and development libraries required for parsedmarc on CentOS or RHEL systems.

```bash
sudo dnf install -y python39 python3-virtualenv python3-setuptools python3-devel libxml2-devel libxslt-devel
```

--------------------------------

### Check geoipupdate Version

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Command to check the installed version of the geoipupdate utility.

```bash
geoipupdate -V
```

--------------------------------

### Specify Configuration File

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

Demonstrates how to use the `-c` option to specify a configuration file for parsedmarc. This is the recommended way to configure the tool starting from version 6.0.0.

```bash
parsedmarc -c /etc/parsedmarc.ini
```

--------------------------------

### ParseDMARC Multi-tenant Configuration Example

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

Define tenant-specific domain mappings in a YAML file for multi-tenant support. This allows ParseDMARC to prefix data with tenant names for index segregation.

```yaml
example:
  - example.com
  - example.net
  - example.org

whalensolutions:
  - whalensolutions.com
```

--------------------------------

### Enable and Restart DavMail Service

Source: https://domainaware.github.io/parsedmarc/davmail.html

Commands to reload systemd, enable the DavMail service to start on boot, and restart the service to apply changes. These commands are necessary after creating or modifying the service file.

```bash
sudo systemctl daemon-reload
sudo systemctl enable parsedmarc.service
sudo service davmail restart

```

--------------------------------

### Set Up Passwords and Generate Encryption Keys

Source: https://domainaware.github.io/parsedmarc/_sources/elasticsearch.md.txt

Set up initial passwords for Elasticsearch users and generate encryption keys for Kibana.

```bash
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
sudo /usr/share/kibana/bin/kibana-encryption-keys generate
```

--------------------------------

### Create OpenSearch Indexes

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/opensearch.html

Initializes a list of indexes with specified settings if they do not already exist.

```python
def create_indexes(names: list[str], settings: Optional[dict[str, Any]] = None):
    """
    Create OpenSearch indexes

    Args:
        names (list): A list of index names
        settings (dict): Index settings

    """
    for name in names:
        index = Index(name)
        try:
            if not index.exists():
                logger.debug("Creating OpenSearch index: {0}".format(name))
                if settings is None:
                    index.settings(number_of_shards=1, number_of_replicas=0)
                else:
                    index.settings(**settings)
                index.create()
        except Exception as e:
            raise OpenSearchError("OpenSearch error: {0}".format(e.__str__()))
```

--------------------------------

### Configure Web Proxy Settings

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Set system-wide proxy configurations for HTTP, HTTPS, and FTP. Ensure to use correct credentials if required.

```bash
http_proxy=http://user:password@prox-server:3128
https_proxy=https://user:password@prox-server:3128
ftp_proxy=http://user:password@prox-server:3128
```

```bash
http_proxy=http://prox-server:3128
https_proxy=https://prox-server:3128
ftp_proxy=http://prox-server:3128
```

--------------------------------

### Initialize virtualenv

Source: https://domainaware.github.io/parsedmarc/installation.html

Create a virtual environment for parsedmarc.

```bash
sudo -u parsedmarc virtualenv /opt/parsedmarc/venv
```

```bash
sudo -u parsedmarc virtualenv -p python3.10  /opt/parsedmarc/venv
```

--------------------------------

### Configure OpenSearch Connection

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/opensearch.html

Sets up the connection parameters for OpenSearch authentication.

```python
    conn_params["api_key"] = api_key
    else:
        raise OpenSearchError(
            f"Unsupported OpenSearch auth_type '{auth_type}'. "
            "Expected 'basic' or 'awssigv4'."
        )
    connections.create_connection(**conn_params)
```

--------------------------------

### Install or Upgrade Parsedmarc

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Installs or upgrades the parsedmarc package within the activated virtual environment using pip. This command must be run as the dedicated parsedmarc user.

```bash
sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip install -U parsedmarc
```

--------------------------------

### Enroll Kibana in Elasticsearch

Source: https://domainaware.github.io/parsedmarc/elasticsearch.html

Generate an enrollment token and verify the connection.

```bash
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
```

```bash
sudo /usr/share/kibana/bin/kibana-verification-code
```

--------------------------------

### GET /ip-info

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/utils.html

Retrieves reverse DNS and country information for a specified IP address.

```APIDOC
## GET /ip-info

### Description
Returns reverse DNS and country information for the given IP address.

### Method
GET

### Endpoint
/ip-info

### Parameters
#### Query Parameters
- **ip_address** (str) - Required - The IP address to check
- **ip_db_path** (str) - Optional - Path to a MMDB file from MaxMind or DBIP
- **reverse_dns_map_path** (str) - Optional - Path to a reverse DNS map file
- **reverse_dns_map_url** (str) - Optional - URL to the reverse DNS map file
- **always_use_local_files** (bool) - Optional - Do not download files
- **offline** (bool) - Optional - Do not make online queries for geolocation or DNS
- **timeout** (float) - Optional - Sets the DNS timeout in seconds

### Response
#### Success Response (200)
- **ip_address** (str) - The IP address
- **reverse_dns** (str) - The reverse DNS record
- **country** (str) - The country associated with the IP
- **base_domain** (str) - The base domain extracted from reverse DNS
- **name** (str) - The service name
- **type** (str) - The service type
```

--------------------------------

### GET /_cluster/health

Source: https://domainaware.github.io/parsedmarc/usage.html

Retrieves the current health status of the Elasticsearch cluster, including shard counts.

```APIDOC
## GET /_cluster/health

### Description
Retrieves the current health status of the Elasticsearch cluster.

### Method
GET

### Endpoint
/_cluster/health

### Query Parameters
- **pretty** (boolean) - Optional - Formats the JSON output for readability

### Response
#### Success Response (200)
- **active_primary_shards** (int) - Number of active primary shards
- **active_shards** (int) - Total number of active shards

#### Response Example
{
  "active_primary_shards": 932,
  "active_shards": 932
}
```

--------------------------------

### GET /reports/zip

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc.html

Generates a zip archive containing all parsed aggregate, forensic, and SMTP TLS reports.

```APIDOC
## GET /reports/zip

### Description
Creates a zip file containing the parsed report output, including JSON and CSV files for aggregate, forensic, and SMTP TLS reports, along with forensic samples.

### Method
GET

### Endpoint
/reports/zip

### Response
#### Success Response (200)
- **body** (bytes) - The binary content of the generated zip file.
```

--------------------------------

### Create Systemd Service File

Source: https://domainaware.github.io/parsedmarc/_sources/davmail.md.txt

Commands and configuration content to register DavMail as a systemd service.

```bash
sudo nano /etc/systemd/system/davmail.service
```

```ini
[Unit]
Description=DavMail gateway service
Documentation=https://sourceforge.net/projects/davmail/
Wants=network-online.target
After=syslog.target network.target

[Service]
ExecStart=/opt/davmail/davmail /opt/davmail/davmail.properties
User=davmail
Group=davmail
Restart=always
RestartSec=5m

[Install]
WantedBy=multi-user.target
```

--------------------------------

### Create Kibana Enrollment Token

Source: https://domainaware.github.io/parsedmarc/_sources/elasticsearch.md.txt

Generate an enrollment token for Kibana to enroll with Elasticsearch.

```bash
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
```

--------------------------------

### Generate Kibana Verification Code

Source: https://domainaware.github.io/parsedmarc/_sources/elasticsearch.md.txt

Generate a verification code for Kibana, typically used during initial setup or enrollment.

```bash
sudo /usr/share/kibana/bin/kibana-verification-code
```

--------------------------------

### Enable and Restart parsedmarc Service

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

Commands to reload systemd, enable the parsedmarc service to start on boot, and restart the service.

```bash
sudo systemctl daemon-reload
sudo systemctl enable parsedmarc.service
sudo service parsedmarc restart
```

--------------------------------

### OpenSearch Configuration

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

Configure connection and indexing settings for OpenSearch.

```APIDOC
## OpenSearch Configuration

### Description
Configure connection details and indexing behavior for OpenSearch.

### Parameters
#### Connection Parameters
- **hosts** (str) - Required - A comma-separated list of hostnames and ports or URLs (e.g., `127.0.0.1:9200` or `https://user:secret@localhost`).
- **user** (str) - Optional - Basic auth username.
- **password** (str) - Optional - Basic auth password. Special characters must be URL encoded.
- **api_key** (str) - Optional - API key for authentication.
- **auth_type** (str) - Optional - Authentication type: `basic` (default) or `awssigv4`. `authentication_type` is an accepted alias.
- **aws_region** (str) - Required when `auth_type` is `awssigv4` - AWS region for SigV4 authentication.
- **aws_service** (str) - Optional - AWS service for SigV4 signing. Defaults to `es`.
- **ssl** (bool) - Optional - Use an encrypted SSL/TLS connection. Defaults to `True`.
- **timeout** (float) - Optional - Timeout in seconds. Defaults to 60.
- **cert_path** (str) - Optional - Path to trusted certificates.
- **skip_certificate_verification** (bool) - Optional - Skip certificate verification (not recommended).

#### Indexing Parameters
- **index_suffix** (str) - Optional - A suffix to apply to the index names.
- **index_prefix** (str) - Optional - A prefix to apply to the index names.
- **monthly_indexes** (bool) - Optional - Use monthly indexes instead of daily indexes.
- **number_of_shards** (int) - Optional - The number of shards to use when creating the index. Defaults to `1`.
- **number_of_replicas** (int) - Optional - The number of replicas to use when creating the index. Defaults to `0`.
```

--------------------------------

### Get Service from Reverse DNS

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/utils.html

Retrieves the service name associated with a base domain derived from a reverse DNS lookup.

```python
def get_service_from_reverse_dns_base_domain(
    base_domain,
    *,
    always_use_local_file: bool = False,
    local_file_path: Optional[str] = None,
    url: Optional[str] = None,
    offline: bool = False,
    reverse_dns_map: Optional[ReverseDNSMap] = None,
) -> ReverseDNSService:
    """
    Returns the service name of a given base domain name from reverse DNS.

    Args:
        base_domain (str): The base domain of the reverse DNS lookup
        always_use_local_file (bool): Always use a local map file
        local_file_path (str): Path to a local map file
        url (str): URL ro a reverse DNS map
        offline (bool): Use the built-in copy of the reverse DNS map
        reverse_dns_map (dict): A reverse DNS map
```

--------------------------------

### Create Virtual Environment for Parsedmarc

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Sets up a Python virtual environment for parsedmarc in its designated directory.

```bash
sudo -u parsedmarc virtualenv /opt/parsedmarc/venv
```

--------------------------------

### Run geoipupdate to Download Databases

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Executes the geoipupdate command to download the necessary GeoIP databases. This command requires root privileges.

```bash
sudo geoipupdate
```

--------------------------------

### Get Base Domain

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/utils.html

Extracts the base domain from a given domain string using public suffix lists and local overrides.

```python
def get_base_domain(domain: str) -> Optional[str]:
    """
    Gets the base domain name for the given domain

    .. note::
        Results are based on a list of public domain suffixes at
        https://publicsuffix.org/list/public_suffix_list.dat and overrides included in
        parsedmarc.resources.maps.psl_overrides.txt

    Args:
        domain (str): A domain or subdomain

    Returns:
        str: The base domain of the given domain

    """
    domain = domain.lower()
    publicsuffix = psl.privatesuffix(domain)
    for override in psl_overrides:
        if domain.endswith(override):
            return override.strip(".").strip("-")
    return publicsuffix
```

--------------------------------

### Restart Services

Source: https://domainaware.github.io/parsedmarc/elasticsearch.html

Restart both Kibana and Elasticsearch services.

```bash
sudo systemctl restart kibana
sudo systemctl restart elasticsearch
```

--------------------------------

### parsedmarc CLI Help

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

Displays all available command-line options for the parsedmarc tool. Use this to understand the various parameters for parsing DMARC reports.

```text
usage: parsedmarc [-h] [-c CONFIG_FILE] [--strip-attachment-payloads] [-o OUTPUT]
                  [--aggregate-json-filename AGGREGATE_JSON_FILENAME] [--forensic-json-filename FORENSIC_JSON_FILENAME]
                  [--smtp-tls-json-filename SMTP_TLS_JSON_FILENAME] [--aggregate-csv-filename AGGREGATE_CSV_FILENAME]
                  [--forensic-csv-filename FORENSIC_CSV_FILENAME] [--smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME]
                  [-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--offline] [-s] [-w] [--verbose] [--debug] 
                  [--log-file LOG_FILE] [--no-prettify-json] [-v]
                  [file_path ...]

Parses DMARC reports

positional arguments:
  file_path             one or more paths to aggregate or forensic report files, emails, or mbox files'

options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        a path to a configuration file (--silent implied)
  --strip-attachment-payloads
                        remove attachment payloads from forensic report output
  -o OUTPUT, --output OUTPUT
                        write output files to the given directory
  --aggregate-json-filename AGGREGATE_JSON_FILENAME
                        filename for the aggregate JSON output file
  --forensic-json-filename FORENSIC_JSON_FILENAME
                        filename for the forensic JSON output file
  --smtp-tls-json-filename SMTP_TLS_JSON_FILENAME
                        filename for the SMTP TLS JSON output file
  --aggregate-csv-filename AGGREGATE_CSV_FILENAME
                        filename for the aggregate CSV output file
  --forensic-csv-filename FORENSIC_CSV_FILENAME
                        filename for the forensic CSV output file
  --smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME
                        filename for the SMTP TLS CSV output file
  -n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
                        nameservers to query
  -t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
                        number of seconds to wait for an answer from DNS (default: 2.0)
  --offline             do not make online queries for geolocation or DNS
  -s, --silent          only print errors
  -w, --warnings        print warnings in addition to errors
  --verbose             more verbose output
  --debug               print debugging information
  --log-file LOG_FILE   output logging to a file
  --no-prettify-json    output JSON in a single line without indentation
  -v, --version         show program's version number and exit
```

--------------------------------

### Docker Compose Example for ParseDMARC

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

This Docker Compose configuration sets up ParseDMARC with various environment variables for IMAP, Elasticsearch, and data saving preferences. It assumes IMAP_PASSWORD is set in the environment.

```yaml
services:
  parsedmarc:
    image: parsedmarc:latest
    environment:
      PARSEDMARC_IMAP_HOST: imap.example.com
      PARSEDMARC_IMAP_USER: dmarc@example.com
      PARSEDMARC_IMAP_PASSWORD: ${IMAP_PASSWORD}
      PARSEDMARC_MAILBOX_WATCH: "true"
      PARSEDMARC_ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      PARSEDMARC_GENERAL_SAVE_AGGREGATE: "true"
      PARSEDMARC_GENERAL_SAVE_FORENSIC: "true"
```

--------------------------------

### Convert Outlook MSG to RFC 822

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/utils.html

Converts an Outlook MSG file to RFC 822 format using the msgconvert Perl utility. Requires the msgconvert tool to be installed in the system path.

```python
def convert_outlook_msg(msg_bytes: bytes) -> bytes:
    """
    Uses the ``msgconvert`` Perl utility to convert an Outlook MS file to
    standard RFC 822 format

    Args:
        msg_bytes (bytes): the content of the .msg file

    Returns:
        A RFC 822 bytes payload
    """
    if not is_outlook_msg(msg_bytes):
        raise ValueError("The supplied bytes are not an Outlook MSG file")
    orig_dir = os.getcwd()
    tmp_dir = tempfile.mkdtemp()
    os.chdir(tmp_dir)
    with open("sample.msg", "wb") as msg_file:
        msg_file.write(msg_bytes)
    try:
        subprocess.check_call(
            ["msgconvert", "sample.msg"], stdout=null_file, stderr=null_file
        )
        eml_path = "sample.eml"
        with open(eml_path, "rb") as eml_file:
            rfc822 = eml_file.read()
    except FileNotFoundError:
        raise EmailParserError(
            "Failed to convert Outlook MSG: msgconvert utility not found"
        )
    finally:
        os.chdir(orig_dir)
        shutil.rmtree(tmp_dir)

    return rfc822
```

--------------------------------

### Create Virtual Environment with Specific Python Version

Source: https://domainaware.github.io/parsedmarc/_sources/installation.md.txt

Creates a Python virtual environment using a specific Python version (e.g., python3.10) on systems where the default Python 3 might be older.

```bash
sudo -u parsedmarc virtualenv -p python3.10  /opt/parsedmarc/venv
```

--------------------------------

### Elasticsearch Configuration

Source: https://domainaware.github.io/parsedmarc/_sources/usage.md.txt

Configure connection and indexing settings for Elasticsearch.

```APIDOC
## Elasticsearch Configuration

### Description
Configure connection details and indexing behavior for Elasticsearch.

### Parameters
#### Connection Parameters
- **hosts** (str) - Required - A comma-separated list of hostnames and ports or URLs (e.g., `127.0.0.1:9200` or `https://user:secret@localhost`).
- **user** (str) - Optional - Basic auth username.
- **password** (str) - Optional - Basic auth password. Special characters must be URL encoded.
- **api_key** (str) - Optional - API key for authentication.
- **ssl** (bool) - Optional - Use an encrypted SSL/TLS connection. Defaults to `True`.
- **timeout** (float) - Optional - Timeout in seconds. Defaults to 60.
- **cert_path** (str) - Optional - Path to trusted certificates.
- **skip_certificate_verification** (bool) - Optional - Skip certificate verification (not recommended).

#### Indexing Parameters
- **index_suffix** (str) - Optional - A suffix to apply to the index names.
- **index_prefix** (str) - Optional - A prefix to apply to the index names.
- **monthly_indexes** (bool) - Optional - Use monthly indexes instead of daily indexes.
- **number_of_shards** (int) - Optional - The number of shards to use when creating the index. Defaults to `1`.
- **number_of_replicas** (int) - Optional - The number of replicas to use when creating the index. Defaults to `0`.
```

--------------------------------

### Get DMARC Reports from Mailbox Connection

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc.html

Retrieves and parses DMARC reports from a specified mailbox folder using an active MailboxConnection. Supports options for archiving, deleting, and offline processing. Requires a MailboxConnection object.

```python
def get_dmarc_reports_from_mailbox(
    connection: MailboxConnection,
    *,
    reports_folder: str = "INBOX",
    archive_folder: str = "Archive",
    delete: bool = False,
    test: bool = False,
    ip_db_path: Optional[str] = None,
    always_use_local_files: bool = False,
    reverse_dns_map_path: Optional[str] = None,
    reverse_dns_map_url: Optional[str] = None,
    offline: bool = False,

```

--------------------------------

### Create Blank Mailman 3 Footer Template

Source: https://domainaware.github.io/parsedmarc/dmarc.html

Use this command to create an empty footer template for a Mailman 3 list, which is necessary to remove the default message footer. Replace 'list.example.com' with your list ID and 'en' with the appropriate language code.

```bash
touch var/templates/lists/list.example.com/en/list:member:regular:footer
```

--------------------------------

### Initialize HECClient

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/splunk.html

Initializes the HECClient with Splunk HEC URL, access token, and index. Supports SSL verification and custom timeouts.

```python
def __init__(
        self,
        url: str,
        access_token: str,
        index: str,
        source: str = "parsedmarc",
        verify=True,
        timeout=60,
    ):
        """
        Initializes the HECClient

        Args:
            url (str): The URL of the HEC
            access_token (str): The HEC access token
            index (str): The name of the index
            source (str): The source name
            verify (bool): Verify SSL certificates
            timeout (float): Number of seconds to wait for the server to send
                data before giving up
        """
        parsed_url = urlparse(url)
        self.url = "{0}://{1}/services/collector/event/1.0".format(
            parsed_url.scheme, parsed_url.netloc
        )
        self.access_token = access_token.lstrip("Splunk ")
        self.index = index
        self.host = socket.getfqdn()
        self.source = source
        self.session = requests.Session()
        self.timeout = timeout
        self.verify = verify
        self._common_data: dict[str, Union[str, int, float, dict]] = dict(
            host=self.host, source=self.source, index=self.index
        )

        self.session.headers = {
            "User-Agent": USER_AGENT,
            "Authorization": "Splunk {0}".format(self.access_token),
        }
```

--------------------------------

### Configure JVM Heap Size

Source: https://domainaware.github.io/parsedmarc/_sources/elasticsearch.md.txt

Settings to adjust the JVM heap size in /etc/elasticsearch/jvm.options to prevent crashes under load.

```bash
-Xms4g
-Xmx4g
```

--------------------------------

### Move and Secure Kibana Keys

Source: https://domainaware.github.io/parsedmarc/_sources/elasticsearch.md.txt

Move the generated certificate and key files to the Kibana configuration directory and set appropriate permissions.

```bash
sudo mv kibana.*
/etc/kibana
sudo chmod 660 /etc/kibana/kibana.key
```

--------------------------------

### Enable and Manage DavMail Service

Source: https://domainaware.github.io/parsedmarc/_sources/davmail.md.txt

Commands to reload systemd, enable the service, and restart it.

```bash
sudo systemctl daemon-reload
sudo systemctl enable parsedmarc.service
sudo service davmail restart
```

--------------------------------

### Create Elasticsearch Indexes

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/elastic.html

Creates specified Elasticsearch indexes with optional custom settings. Useful for initializing the Elasticsearch environment for Parsedmarc.

```python
def create_indexes(names: list[str], settings: Optional[dict[str, Any]] = None):
    """
    Create Elasticsearch indexes

    Args:
        names (list): A list of index names
        settings (dict): Index settings

    """
    for name in names:
        index = Index(name)
        try:
            if not index.exists():
                logger.debug("Creating Elasticsearch index: {0}".format(name))
                if settings is None:
                    index.settings(number_of_shards=1, number_of_replicas=0)
                else:
                    index.settings(**settings)
                index.create()
        except Exception as e:
```

--------------------------------

### Load IP Database

Source: https://domainaware.github.io/parsedmarc/_modules/parsedmarc/utils.html

Downloads or loads a local IP-to-country MMDB database for geolocation lookups.

```python
def load_ip_db(
    *,
    always_use_local_file: bool = False,
    local_file_path: Optional[str] = None,
    url: Optional[str] = None,
    offline: bool = False,
) -> None:
    """
    Downloads the IP-to-country MMDB database from a URL and caches it
    locally. Falls back to the bundled copy on failure or when offline.

    Args:
        always_use_local_file: Always use a local/bundled database file
        local_file_path: Path to a local MMDB file
        url: URL to the MMDB database file
        offline: Do not make online requests
    """
    global _IP_DB_PATH

    if url is None:
        url = (
            "https://github.com/domainaware/parsedmarc/raw/"
            "refs/heads/master/parsedmarc/resources/dbip/"
            "dbip-country-lite.mmdb"
        )

    if local_file_path is not None and os.path.isfile(local_file_path):
        _IP_DB_PATH = local_file_path
        logger.info(f"Using local IP database at {local_file_path}")
        return

    cache_dir = os.path.join(tempfile.gettempdir(), "parsedmarc")
    cached_path = os.path.join(cache_dir, "dbip-country-lite.mmdb")

    if not (offline or always_use_local_file):
        try:
            logger.debug(f"Trying to fetch IP database from {url}...")
            headers = {"User-Agent": USER_AGENT}
            response = requests.get(url, headers=headers, timeout=60)
            response.raise_for_status()
            os.makedirs(cache_dir, exist_ok=True)
            tmp_path = cached_path + ".tmp"
            with open(tmp_path, "wb") as f:
                f.write(response.content)
            shutil.move(tmp_path, cached_path)
            _IP_DB_PATH = cached_path
            logger.info("IP database updated successfully")
            return
        except requests.exceptions.RequestException as e:
            logger.warning(f"Failed to fetch IP database: {e}")
        except Exception as e:
            logger.warning(f"Failed to save IP database: {e}")

    # Fall back to a previously cached copy if available
    if os.path.isfile(cached_path):
        _IP_DB_PATH = cached_path
        logger.info("Using cached IP database")
        return

    # Final fallback: bundled copy
    _IP_DB_PATH = str(
        files(parsedmarc.resources.dbip).joinpath("dbip-country-lite.mmdb")
    )
    logger.info("Using bundled IP database")
```
Tokens:
Raw