### Run Fluent Bit Setup Script on Linux Source: https://github.com/idaholab/malcolm/blob/main/docs/third-party-logs.md This example demonstrates the interactive process of using the `fluent-bit-setup.sh` script on a Linux system to install Fluent Bit, configure it to collect CPU metrics, and set up a systemd service. ```bash $ ~/Malcolm/scripts/third-party-logs/fluent-bit-setup.sh 0 ALL 1 InstallFluentBit 2 GetMalcolmConnInfo 3 GetFluentBitFormatInfo 4 CreateFluentbitService Operation: 0 Install fluent-bit via GitHub/fluent install script [Y/n]? y ================================ Fluent Bit Installation Script ================================ This script requires superuser access to install packages. You will be prompted for your password by sudo. … Installation completed. Happy Logging! Choose input plugin and enter parameters. Leave parameters blank for defaults. see https://docs.fluentbit.io/manual/pipeline/inputs 1 collectd 2 cpu 3 disk 4 docker 5 docker_events 6 dummy 7 dummy_thread 8 exec 9 fluentbit_metrics 10 forward 11 head 12 health 13 http 14 kmsg 15 mem 16 mqtt 17 netif 18 nginx_metrics 19 node_exporter_metrics 20 opentelemetry 21 proc 22 prometheus_scrape 23 random 24 serial 25 statsd 26 stdin 27 syslog 28 systemd 29 tail 30 tcp 31 thermal Input plugin: 2 cpu Interval_Sec: 10 cpu Interval_NSec: cpu PID: Enter Malcolm host or IP address (172.16.0.20): 172.16.0.20 Enter Malcolm Filebeat TCP port (5045): 5045 Enter agent hostname (hostname): hostname Enter fluent-bit output format (json_lines): json_lines Nest values under field: cpu Add "module" value: cpu /usr/local/bin/fluent-bit -R /etc/fluent-bit/parsers.conf -i cpu -p Interval_Sec=10 -o tcp://172.16.0.20:5045 -p tls=on -p tls.verify=off -p tls.ca_file=/home/user/Malcolm/filebeat/certs/ca.crt -p tls.crt_file=/home/user/Malcolm/filebeat/certs/client.crt -p tls.key_file=/home/user/Malcolm/filebeat/certs/client.key -p format=json_lines -F nest -p Operation=nest -p Nested_under=cpu -p WildCard='*' -m '*' -F record_modifier -p 'Record=module cpu' -m '*' -f 1 Configure service to run fluent-bit [y/N]? y Enter .service file prefix: fluentbit_cpu Configure systemd service as user "user" [Y/n]? y [sudo] password for user: Created symlink /home/user/.config/systemd/user/default.target.wants/fluentbit_cpu.service → /home/user/.config/systemd/user/fluentbit_cpu.service. ● fluentbit_cpu.service Loaded: loaded (/home/user/.config/systemd/user/fluentbit_cpu.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2022-08-09 09:19:43 MDT; 5s ago Main PID: 105521 (fluent-bit) Tasks: 5 (limit: 76711) Memory: 24.7M CPU: 8ms CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/fluentbit_cpu.service └─105521 /usr/local/bin/fluent-bit -R /etc/fluent-bit/parsers.conf -i cpu -p Interval_Sec=10 -o tcp://172.16.0.20:5045 -p tls=on -p tls.verify=off -p tls.ca_fil… Aug 09 09:19:43 localhost fluent-bit[105521]: Fluent Bit v1.9.6 … Aug 09 09:19:43 localhost fluent-bit[105521]: [2022/08/09 09:19:43] [ info] [output:tcp:tcp.0] worker #0 started Aug 09 09:19:43 localhost fluent-bit[105521]: [2022/08/09 09:19:43] [ info] [output:tcp:tcp.0] worker #1 started ``` -------------------------------- ### Install Podman on Ubuntu 24.04 LTS using alvistack repository Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md Example process for installing and configuring Podman on Ubuntu 24.04 LTS using a third-party repository. This example uses the alvistack repository for up-to-date packages. Podman v5.6.0 or higher is recommended. ```bash # Initial Podman installation and system tweaks ``` -------------------------------- ### Podman Installation Example Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md This snippet shows a typical output from Podman when listing running containers. It is useful for verifying that Malcolm components are running correctly. ```bash malcolm-pipeline-suricata-live-1 ghcr.io/idaholab/malcolm/suricata:{{ site.malcolm.version }} "/usr/bin/tini -- /u…" suricata-live 8 minutes ago Up 8 minutes (healthy) malcolm-pipeline-upload-1 ghcr.io/idaholab/malcolm/file-upload:{{ site.malcolm.version }} "/usr/bin/tini -- /u…" upload 8 minutes ago Up 8 minutes (healthy) 22/tcp, 80/tcp malcolm-pipeline-zeek-1 ghcr.io/idaholab/malcolm/zeek:{{ site.malcolm.version }} "/usr/bin/tini -- /u…" zeek 8 minutes ago Up 8 minutes (healthy) malcolm-pipeline-zeek-live-1 ghcr.io/idaholab/malcolm/zeek:{{ site.malcolm.version }} "/usr/bin/tini -- /u…" zeek-live 8 minutes ago Up 8 minutes (healthy) ``` -------------------------------- ### Set Up Malcolm Authentication Source: https://github.com/idaholab/malcolm/blob/main/docs/aws.md Navigate to the Malcolm installation directory and run the authentication setup script. This script guides through configuring authentication settings. ```bash $ cd ~/Malcolm $ ./scripts/auth_setup all Configure all authentication-related settings ``` -------------------------------- ### Execute Docker Install Script Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-windows.md Run the downloaded script with sudo to install Docker. ```bash sudo sh get-docker.sh ``` -------------------------------- ### Test Docker Installation Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md Verify your Docker installation by running 'docker info' or by pulling and running the 'hello-world' image. ```bash docker info ``` ```bash docker run --rm hello-world ``` -------------------------------- ### Install Docker Desktop and Docker Compose Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-macos.md Use Homebrew Cask to install Docker Desktop, which includes the Docker Compose plugin. This command installs the latest version. ```bash $ brew install --cask docker-desktop ``` -------------------------------- ### Install WSL Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-windows.md Run this command in an administrator PowerShell or Command Prompt to install the Windows Subsystem for Linux. ```bash wsl --install ``` -------------------------------- ### Run Authentication Setup Script Source: https://github.com/idaholab/malcolm/blob/main/docs/malcolm-upgrade.md Execute the authentication setup script to verify the authentication method and ensure existing authentication material is preserved. Avoid regenerating certificates or overwriting passwords unless intended. ```bash ./scripts/auth_setup ``` -------------------------------- ### Start Malcolm Services Source: https://github.com/idaholab/malcolm/blob/main/docs/aws.md Execute this script to initiate Malcolm. It may take a few minutes for all services to synchronize. Look for 'Started Malcolm' to confirm successful startup. ```bash $ cd ~/Malcolm $ ./scripts/start logstash | [2026-01-16T17:58:33,274][INFO ][logstash.agent ] Pipelines running {:count=>7, :running_pipelines=>[:"malcolm-output", :"malcolm-input", :"malcolm-filescan", :"malcolm-suricata", :"malcolm-enrichment", :"malcolm-beats", :"malcolm-zeek"], :non_running_pipelines=>[]} Started Malcolm Malcolm services can be accessed at https:/// ------------------------------------------------------------------------------ ``` -------------------------------- ### Install Dialog Interface for Malcolm Installer Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md Installs the 'dialog' utility and the 'python3-dialog' library to enable a dialog-driven user interface for the Malcolm installer. ```bash $ sudo apt-get -y install python3-dialog dialog Reading package lists... Done … Setting up dialog (1.3-20240101-1) ... Setting up python3-dialog (3.5.1-4) ... … ``` -------------------------------- ### Start Malcolm Instance Source: https://github.com/idaholab/malcolm/blob/main/docs/running.md Use this script to initiate Malcolm containers. It creates the network and starts the containers in the background. Allow several minutes for full startup. ```bash $ ./scripts/start ``` -------------------------------- ### Add Podman Repository and Install Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md Installs Podman and related packages by adding a custom OpenSUSE repository. This involves updating package lists, configuring APT sources and preferences, and then performing the installation. ```bash $ sudo apt-get update … Fetched 38.6 MB in 8s (4901 kB/s) $ apt-cache policy podman podman: Installed: (none) Candidate: 4.9.3+ds1-1ubuntu0.2 … $ echo 'deb [signed-by=/etc/apt/trusted.gpg.d/home_alvistack.gpg] http://download.opensuse.org/repositories/home:/alvistack/xUbuntu_24.04/ / | sudo tee /etc/apt/sources.list.d/home:alvistack.list >/dev/null $ curl -fsSL https://download.opensuse.org/repositories/home:alvistack/xUbuntu_24.04/Release.key \ | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/home_alvistack.gpg >/dev/null $ sudo tee /etc/apt/preferences.d/99-home_alvistack >/dev/null <<'EOT' Package: * Pin: origin download.opensuse.org Pin-Priority: 1 Package: buildah catatonit conmon containernetworking containernetworking-plugins containers-common cri-o-runc crun libcharon-standard-plugins libslirp0 passt podman podman-aardvark-dns podman-netavark python3-podman-compose slirp4netns Pin: origin download.opensuse.org Pin-Priority: 1001 EOT $ sudo apt-get update … Get:5 http://download.opensuse.org/repositories/home:/alvistack/xUbuntu_24.04 InRelease [1551 B] Get:6 http://downloadcontentcdn.opensuse.org/repositories/home:/alvistack/xUbuntu_24.04 Packages [170 kB] Fetched 172 kB in 3s (61.8 kB/s) Reading package lists... Done $ apt-cache policy podman podman: Installed: (none) Candidate: 100:5.6.2-1 … $ sudo apt-get install -y \ buildah \ catatonit \ crun \ fuse-overlayfs \ passt \ podman \ podman-aardvark-dns \ podman-netavark \ slirp4netns \ uidmap … Setting up podman (100:5.6.2-1) ... … ``` -------------------------------- ### Start Malcolm Kubernetes Deployment Source: https://github.com/idaholab/malcolm/blob/main/docs/kubernetes.md Use the start script to create and initialize Malcolm's Kubernetes deployment. Provide the path to your kubeconfig file using the -f or --file argument. ```bash ./scripts/start -f /path/to/kubeconfig.yml ``` -------------------------------- ### Install Malcolm using install.py Source: https://github.com/idaholab/malcolm/blob/main/docs/development.md Installs Malcolm by running the `install.py` script with the configuration tarball as an argument. This script can also handle the installation of Docker and Docker Compose if they are not already present. ```bash install.py malcolm_XXXXXXXX_XXXXXX_XXXXXXX.tar.gz ``` -------------------------------- ### Install kubectl Source: https://github.com/idaholab/malcolm/blob/main/docs/aws.md Installs kubectl, the Kubernetes command-line tool, on a Linux system. Requires curl and chmod. ```bash $ curl -fsSL \ -o /tmp/kubectl \ "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/$(uname -m | sed 's/^x86_64$/amd64/' | sed 's/^aarch64$/arm64/')/kubectl" $ chmod 755 /tmp/kubectl $ sudo mv /tmp/kubectl /usr/local/bin/ $ kubectl version Client Version: v1.32.3 ``` -------------------------------- ### Malcolm Installation Configuration Summary Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md This snippet displays the final configuration summary after the Malcolm installation, detailing various settings including the container runtime. ```bash ============================================================ FINAL CONFIGURATION SUMMARY ============================================================ Configuration Only : No Auto Apply System Tweaks : Yes Configuration Directory : /home/johndoe/Malcolm/config Container Runtime : podman Run Profile : malcolm Process UID/GID : 1000/1000 Container Restart Policy : No Container Network : default Default Storage Locations : Yes HTTPS/SSL : Yes Node Name : ubuntu-noble-191 ============================================================ Proceed with Malcolm installation using the above configuration? (y / N): y … [2025-11-05 18:17:34] (SUCCESS) [INSTALLER]: Installation completed successfully ``` -------------------------------- ### Install eksctl Source: https://github.com/idaholab/malcolm/blob/main/docs/aws.md Installs eksctl, the official CLI for Amazon EKS, on a Linux system. Requires curl and tar. ```bash $ curl -fsSL \ -o /tmp/eksctl.tar.gz \ "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_$(uname -m | sed 's/^x86_64$/amd64/').tar.gz" $ tar -xzf /tmp/eksctl.tar.gz -C /tmp && rm /tmp/eksctl.tar.gz $ sudo mv /tmp/eksctl /usr/local/bin/ $ eksctl version 0.207.0 ``` -------------------------------- ### Install usrmerge Package Source: https://github.com/idaholab/malcolm/blob/main/docs/malcolm-upgrade.md Install the 'usrmerge' package to convert the system to the merged /usr layout if it has not already been converted. ```bash apt -y install --no-install-recommends usrmerge ``` -------------------------------- ### Download Docker Install Script Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-windows.md Use curl to download the official Docker installation script for Linux. ```bash curl -fsSL https://get.docker.com -o get-docker.sh ``` -------------------------------- ### Install Homebrew and Cask Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-macos.md Execute these commands in a terminal to install Homebrew and the cask package manager, which simplifies Docker installation on macOS. ```bash $ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" $ brew install cask ``` -------------------------------- ### Malcolm Configuration Menu Example Source: https://github.com/idaholab/malcolm/blob/main/docs/malcolm-hedgehog-e2e-iso-install.md Example of configuration options within the Malcolm Configuration Menu, showing settings for Logstash, Reachback ACL, Document Store, and Arkime WISE URL. ```text --- Malcolm Configuration Menu --- ... ├── 18. Logstash Host (current: malcolm.home.arpa:5044) ├── 19. Malcolm Reachback ACL (current: ['192.168.122.1', '192.168.122.5', '10.0.0.130']) └── 20. Primary Document Store (current: opensearch-remote) ├── 21. Primary OpenSearch/Elasticsearch URL (current: https://malcolm.home.arpa:9200) ... ├── 30. Enable Arkime Analysis (current: Yes) └── 31. Arkime WISE URL (current: https://malcolm.home.arpa/wise/) ... ├── 55. NetBox Mode (current: Remote) └── 56. NetBox Site Name (current: Remote Site ABC) ... ``` -------------------------------- ### Build Malcolm Installer ISO with Vagrant Source: https://github.com/idaholab/malcolm/blob/main/docs/malcolm-iso.md Use this command to initiate a clean build of the Malcolm installer ISO. Ensure you are in the local Malcolm working copy. ```bash $ ./malcolm-iso/build_via_vagrant.sh -f -i malcolm ``` -------------------------------- ### Example Container Log Output Source: https://github.com/idaholab/malcolm/blob/main/docs/kubernetes.md This is an example of the log output you might see from various Malcolm deployment containers. ```text api | [2023-04-24 20:55:59 +0000] [7] [INFO] Booting worker with pid: 7 dashboards | log [20:59:28.784] [info][server][OpenSearchDashboards][http] http server running at http://0.0.0.0:5601/dashboards freq | 2023-04-24 20:57:09,481 INFO success: freq entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) htadmin | 2023-04-24 20:58:04,724 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs) opensearch | [2023-04-24T21:00:18,442][WARN ][o.o.c.m.MetadataIndexTemplateService] [opensearch-deployment-75498799f6-5v92w] index template [malcolm_template] has index patterns [arkime_sessions3-*] matching patterns from existing older templates [arkime_sessions3_ecs_template,arkime_sessions3_template] with patterns (arkime_sessions3_ecs_template => [arkime_sessions3-*],arkime_sessions3_template => [arkime_sessions3-*]); this template [malcolm_template] will take precedence during new index creation pcap-capture | 8:57PM INF Listening at http://0.0.0.0:80 /... pcap-monitor | 2023-04-24 20:59:53 INFO: ۞ started [1] upload | 2023-04-24 20:59:27,496 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs) zeek-live | 8:59PM INF Listening at http://0.0.0.0:80 /... zeek-offline | 2023-04-24 20:58:16,072 INFO success: pcap-zeek entered RUNNING state, process has stayed up for > than 15 seconds (startsecs) suricata-live | 8:57PM INF Listening at http://0.0.0.0:80 /... logstash | [2023-04-24T21:00:34,470][INFO ][logstash.agent ] Pipelines running {:count=>6, :running_pipelines=>[:"malcolm-input", :"malcolm-output", :"malcolm-suricata", :"malcolm-beats", :"malcolm-enrichment", :"malcolm-zeek"], :non_running_pipelines=>[]}... ``` -------------------------------- ### Start Malcolm with Kubeconfig Source: https://github.com/idaholab/malcolm/blob/main/docs/kubernetes.md Use the start script with a kubeconfig file to launch Malcolm in a Kubernetes cluster. The script adapts based on the provided configuration. ```bash ./scripts/start -f kubeconfig.yaml ``` -------------------------------- ### Run Malcolm Install Script Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md Execute the install.py script with sudo to configure the system. This command initiates the Malcolm installation process, requiring elevated privileges. ```bash user@host:~/Malcolm$ sudo ./scripts/install.py ``` -------------------------------- ### Run All Malcolm Installer Tests Source: https://github.com/idaholab/malcolm/blob/main/scripts/installer/tests/README.md Execute all unit and mock tests for the Malcolm installer. Assumes execution from the repository root. ```bash python scripts/installer/tests/run_tests.py ``` -------------------------------- ### Malcolm Installation Options Menu Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md This is the main interactive menu for configuring Malcolm's installation. Users select options by number or action to customize settings. ```bash --- Malcolm Installation Options --- Select an item number to configure, or an action: ├── 1. Automatically Apply System Tweaks (current: Yes) ├── 2. Docker Compose Installation Method (current: github) ├── 3. Docker Installation Method (current: repository) ├── 4. Docker Users (current: ['ubuntu']) ├── 5. Install Docker if Missing (current: Yes) ├── 6. Pull Malcolm Images (current: No) ├── 7. Try Docker Convenience Script (current: No) └── 8. Try Docker Repository Installation (current: Yes) --- Actions --- s. Save and Continue x. Exit Installer Enter item number or action: ``` -------------------------------- ### Install Docker Compose CLI Plugin Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md Downloads and installs the Docker Compose CLI plugin for Linux x86_64 architecture into the user's Docker configuration directory. This makes the `docker-compose` command available. ```bash $ mkdir -p ~/.docker/cli-plugins && \ curl -fsSL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 \ -o ~/.docker/cli-plugins/docker-compose && \ chmod 755 ~/.docker/cli-plugins/docker-compose ``` -------------------------------- ### Display auth_setup Script Help Source: https://github.com/idaholab/malcolm/blob/main/docs/authsetup.md Run the `auth_setup` script with the `--help` option to view all available command-line arguments and their descriptions. This is useful for understanding the non-interactive setup options. ```bash $ ./scripts/auth_setup --help usage: auth_setup Malcolm control script options: -v [DEBUG], --verbose [DEBUG] Verbose output -f , --file Docker compose or kubeconfig YML file -e , --environment-dir Directory containing Malcolm's .env files … Authentication Setup: --auth [CMDAUTHSETUP] Configure Malcolm authentication --auth-noninteractive [CMDAUTHSETUPNONINTERACTIVE] Configure Malcolm authentication (noninteractive using arguments provided) --auth-method Authentication method (for --auth-noninteractive) --auth-ldap-mode LDAP server compatibility type (for --auth-noninteractive when --auth-method is ldap) --auth-ldap-start-tls [true|false] Use StartTLS (rather than LDAPS) for LDAP connection security (for --auth-noninteractive when --auth-method is ldap) --auth-admin-username Administrator username (for --auth-noninteractive) --auth-admin-password-openssl Administrator password hash from "openssl -passwd -6" (for --auth-noninteractive) --auth-admin-password-htpasswd Administrator password hash from "htpasswd -n -B username | cut -d: -f2" (for --auth-noninteractive) --auth-arkime-password Password hash secret for Arkime viewer cluster (for --auth-noninteractive) --auth-generate-webcerts [AUTHGENWEBCERTS] (Re)generate self-signed certificates for HTTPS access (for --auth-noninteractive) --auth-generate-fwcerts [AUTHGENFWCERTS] (Re)generate self-signed certificates for a remote log forwarder --auth-netbox-token API token for remote NetBox instance (for --auth-noninteractive when NETBOX_MODE=remote in netbox-common.env) --auth-generate-netbox-passwords [AUTHGENNETBOXPASSWORDS] (Re)generate internal passwords for NetBox --auth-generate-valkey-password [AUTHGENVALKEYPASSWORD] (Re)generate internal passwords for Valkey --auth-generate-postgres-password [AUTHGENPOSTGRESPASSWORD] (Re)generate internal superuser passwords for PostgreSQL --auth-generate-opensearch-internal-creds [AUTHGENOPENSEARCHCREDS] (Re)generate internal credentials for embedded OpenSearch instance --auth-generate-keycloak-db-password [AUTHGENKEYCLOAKDBPASSWORD] (Re)generate internal passwords for Keycloak's PostgreSQL database --auth-keycloak-realm Keycloak realm --auth-keycloak-redirect-uri Keycloak redirect URI --auth-keycloak-url Keycloak URL --auth-keycloak-client-id Keycloak client ID --auth-keycloak-client-secret Keycloak client secret --auth-keycloak-bootstrap-user Temporary Keycloak admin bootstrap username --auth-keycloak-bootstrap-password Temporary Keycloak admin bootstrap password --auth-require-group Required group(s) to which users must belong (--auth-method is keycloak|keycloak_remote) --auth-require-role Required role(s) which users must be assigned (--auth-method is keycloak|keycloak_remote) --auth-role-based-access-control [AUTHRBACENABLED] Enable Role-Based Access Control (--auth-method is keycloak|keycloak_remote) … ``` -------------------------------- ### Clone Malcolm and Install Dependencies Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md Clones the Malcolm project repository from GitHub and installs essential Python packages required for its setup script. ```bash $ git clone https://github.com/idaholab/Malcolm Cloning into 'Malcolm'... remote: Enumerating objects: 57191, done. remote: Counting objects: 872/872, done. remote: Compressing objects: 96/96, done. remote: Total 57191 (delta 813), reused 790 (delta 772), pack-reused 56319 (from 2) Receiving objects: 100% (57191/57191), 237.41 MiB | 4.05 MiB/s, done. Resolving deltas: 100% (42544/42544), done. $ sudo apt-get install -y \ python3-pip \ python3-ruamel.yaml \ python3-dotenv \ python3-dialog \ dialog … ``` -------------------------------- ### Run Malcolm Installation Script Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-linux.md Executes the Malcolm installation script, skipping the splash screen. This script guides the user through system configuration, including container runtime selection. ```bash $ sudo ./scripts/install.py --skip-splash ``` -------------------------------- ### Run Malcolm Authentication Setup Script Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md Execute the `auth_setup` script to configure authentication, generate certificates, and set up user accounts. Follow the prompts to select authentication methods, create administrator credentials, and generate necessary TLS certificates for HTTPS and log forwarding. ```bash user@host:~/Malcolm$ ./scripts/auth_setup 1: all - Configure all authentication-related settings 2: method - Select authentication method (currently "basic") 3: admin - Store administrator username/password for basic HTTP authentication 4: webcerts - (Re)generate self-signed certificates for HTTPS access 5: fwcerts - (Re)generate self-signed certificates for a remote log forwarder 6: keycloak - Configure Keycloak 7: remoteos - Configure remote primary or secondary OpenSearch/Elasticsearch instance 8: email - Store username/password for OpenSearch Alerting email sender account 9: netbox - (Re)generate internal passwords for NetBox 10: keycloakdb - (Re)generate internal passwords for Keycloak's PostgreSQL database 11: postgres - (Re)generate internal superuser passwords for PostgreSQL 12: valkey - (Re)generate internal passwords for Valkey 13: arkime - Store password hash secret for Arkime viewer cluster 14: txfwcerts - Transfer self-signed client certificates to a remote log forwarder Configure Authentication (all): 1 Select authentication method (currently "basic")? (Y / n): y 1: basic - Use basic HTTP authentication 2: ldap - Use Lightweight Directory Access Protocol (LDAP) for authentication 3: keycloak - Use embedded Keycloak for authentication 4: keycloak_remote - Use remote Keycloak for authentication 5: no_authentication - Disable authentication Select authentication method (basic): 1 Store administrator username/password for basic HTTP authentication? (Y / n): y Administrator username (between 4 and 32 characters; alphanumeric, _, -, and . allowed) (): analyst analyst password (between 8 and 128 characters): : analyst password (again): : Additional local accounts can be created at https://localhost/auth/ when Malcolm is running (Re)generate self-signed certificates for HTTPS access? (Y / n): y (Re)generate self-signed certificates for a remote log forwarder? (Y / n): y Configure Keycloak? (Y / n): n Configure remote primary or secondary OpenSearch/Elasticsearch instance? (y / N): n Store username/password for OpenSearch Alerting email sender account? (y / N): n (Re)generate internal passwords for NetBox? (Y / n): y (Re)generate internal passwords for Keycloak's PostgreSQL database? (Y / n): y (Re)generate internal superuser passwords for PostgreSQL? (Y / n): y (Re)generate internal passwords for Valkey? (Y / n): y Store password hash secret for Arkime viewer cluster? (y / N): n Transfer self-signed client certificates to a remote log forwarder? (y / N): n ``` -------------------------------- ### Configure Malcolm Source: https://github.com/idaholab/malcolm/blob/main/docs/aws.md Initializes Malcolm configuration using a provided kubeconfig file. The script guides users through setup prompts. ```bash ./configure -f "${KUBECONFIG:-$HOME/.kube/config}" ``` -------------------------------- ### Malcolm Installation Prompts Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md When installing from release artifacts, the script prompts for confirmation to extract the tarball and to specify the installation path. This allows customization of the installation location. ```bash Found Malcolm tarball: malcolm_20251029_140727_d22a504f.tar.gz. Use this file? (Y / n): Enter installation path for Malcolm [/home/user/malcolm] (/home/user/Malcolm): ``` -------------------------------- ### Install AWS CLI Source: https://github.com/idaholab/malcolm/blob/main/docs/aws.md Installs the AWS Command Line Interface (CLI) on a Linux system. Ensure you have curl and unzip installed. ```bash $ curl -fsSL \ -o /tmp/awscli.zip \ "https://awscli.amazonaws.com/awscli-exe-linux-$(uname -m).zip" $ unzip -d /tmp /tmp/awscli.zip $ sudo /tmp/aws/install You can now run: /usr/local/bin/aws --version $ aws --version aws-cli/2.26.2 Python/3.13.2 Linux/6.1.0-32-amd64 exe/x86_64.ubuntu.24 ``` -------------------------------- ### Install New Malcolm Packages Source: https://github.com/idaholab/malcolm/blob/main/docs/malcolm-upgrade.md Installs new packages required by the Malcolm ISO configuration. This script fetches package lists from GitHub, filters, combines, and installs them using apt. ```bash curl -s "https://api.github.com/repos/idaholab/Malcolm/contents/malcolm-iso/config/package-lists?ref=v{{ site.malcolm.version }}" \ | grep "download_url" \ | cut -d '"' -f 4 \ | grep -Ev ".binary$|live\.|virtualguest." \ | while read -r url; do curl -sSL "$url"; echo; done \ | sort -u \ | grep -Ev "^\s*#|^\s*$" \ | xargs -r apt install -y ``` -------------------------------- ### List initial release files Source: https://github.com/idaholab/malcolm/blob/main/docs/contributing-release-prep.md Lists the files present before splitting ISOs. This includes build logs and the original ISO images. ```bash $ ls -l total 8,502,263,808 -rw-r--r-- 1 romeogdetlevjr romeogdetlevjr 1,209,240 Oct 22 09:50 hedgehog-{{ site.malcolm.version }}-build.log -rw-r--r-- 1 romeogdetlevjr romeogdetlevjr 2,664,972,288 Oct 22 09:50 hedgehog-{{ site.malcolm.version }}.iso -rw-r--r-- 1 romeogdetlevjr romeogdetlevjr 963,775 Oct 22 09:49 malcolm-{{ site.malcolm.version }}-build.log -rw-r--r-- 1 romeogdetlevjr romeogdetlevjr 5,835,110,400 Oct 22 09:49 malcolm-{{ site.malcolm.version }}.iso ``` -------------------------------- ### Setting up Kubernetes Cluster and Persistent Volumes with Vagrant Source: https://github.com/idaholab/malcolm/blob/main/docs/kubernetes.md This snippet demonstrates the initial steps to bring up a virtualized Kubernetes cluster using Vagrant and configure NFS persistent volumes. It includes commands to retrieve the kubeconfig file and modify volume definitions for the Vagrant environment. ```bash $ cd kubernetes/vagrant $ vagrant up Bringing machine 'Malcolm-K8S' up with 'libvirt' provider... ... Malcolm-K8S: Waiting for Traefik ingress controller... Malcolm-K8S: Traefik ingress controller is ready Malcolm-K8S: Malcolm-K8S: To SSH into the virtual machine, run: vagrant ssh Malcolm-K8S: To retrieve kubeconfig file, run: vagrant ssh -c 'cat /home/vagrant/.kube/config' >./kubeconfig && sed -i 's/127.0.0.1/192.168.121.239/g' ./kubeconfig Malcolm-K8S: Malcolm-K8S: To populate NFS volumes manifest, run: sed -e 's/^\]([[:space:]]*server:\]([[:space:]]*\]).*/\1192.168.121.239/' -e 's|/malcolm/|/mnt/vdb/nfs-export/malcolm/|g' ../01-volumes-nfs.yml.example >../01-volumes-nfs-vagrant.yml Malcolm-K8S: Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/config 192.168.121.239/24 Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/filescan-logs 192.168.121.239/24 Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/opensearch 192.168.121.239/24 Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/opensearch-backup 192.168.121.239/24 Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/pcap 192.168.121.239/24 Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/runtime-logs 192.168.121.239/24 Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/suricata-logs 192.168.121.239/24 Malcolm-K8S: /mnt/vdb/nfs-export/malcolm/zeek-logs 192.168.121.239/24 $ vagrant ssh -c 'cat /home/vagrant/.kube/config' >./kubeconfig && sed -i 's/127.0.0.1/192.168.121.239/g' ./kubeconfig $ sed -e 's/^\]([[:space:]]*server:\]([[:space:]]*\]).*/\1192.168.121.239/' \ -e 's|/malcolm/|/mnt/vdb/nfs-export/malcolm/|g' \ ../01-volumes-nfs.yml.example > ../01-volumes-nfs-vagrant.yml $ cp ../99-ingress-traefik.yml.example ../99-ingress.yml $ KUBECONFIG=kubeconfig kubectl get nodes NAME STATUS ROLES AGE VERSION debian-13 Ready control-plane,etcd 6m58s v1.35.3+rke2r3 $ cd ../../ ``` -------------------------------- ### Verify Docker Installation Source: https://github.com/idaholab/malcolm/blob/main/docs/host-config-windows.md Check if Docker and Docker Compose are installed correctly by verifying their versions. ```bash docker --version ``` ```bash docker compose version ``` -------------------------------- ### Malcolm Configure Script Help Source: https://github.com/idaholab/malcolm/blob/main/docs/malcolm-config.md Displays all available command-line arguments for the Malcolm configuration script. Use this to understand the various options for setup and customization. ```bash usage: configure [-h] [--debug [true|false]] [--quiet] [--configure [true|false]] [--dry-run] [--log-to-file [filename]] [--skip-splash] [--tui | --dui | --gui | --non-interactive] [--compose-file ] [--environment-dir-input ] [--environment-dir-output ] [--export-malcolm-config-file []] [--import-malcolm-config-file | --load-existing-env [true|false] | --defaults] [--malcolm-file ] [--image-file ] [--extra [EXTRASETTINGS ...]] Malcolm Installer options: -h, --help show this help message and exit Installer Options: --debug, --verbose [true|false] Enable debug output including tracebacks and debug utilities --quiet, --silent Suppress console logging output during installation --configure, -c [true|false] Only write configuration and ancillary files; skip installation steps --dry-run Log planned actions without writing files or making system changes --log-to-file [filename] Log output to file. If no filename provided, creates timestamped log file. --skip-splash Skip the splash screen prompt on startup Interface Mode (mutually exclusive): --tui Run in command-line text-based interface mode (default) --dui Run in python dialogs text-based user interface mode (if available - requires python dialogs) --gui Run in graphical user interface mode (if available - requires customtkinter) --non-interactive Run in non-interactive mode for unattended installations (suppresses all user prompts) Configuration File Options: --compose-file, --configure-file, --kube-file, -f Path to docker-compose.yml (for compose) or kubeconfig (for Kubernetes) Environment Config Options: --environment-dir-input Input directory containing Malcolm's .env and .env.example files --environment-dir-output, -e Target directory for writing Malcolm's .env files --export-malcolm-config-file, --export-mc-file [] Export configuration to JSON/YAML settings file (auto-generates filename if not specified) --import-malcolm-config-file, --import-mc-file Import configuration from JSON/YAML settings file --load-existing-env, -l [true|false] Automatically load provided config/ .env files from the input directory when present. Can be used in conjunction with --environment-dir-input --defaults, -d Use built-in default configuration values and skip loading from the config directory Installation Files: --malcolm-file, -m Malcolm .tar.gz file for installation --image-file, -i Malcolm container images .tar.xz file for installation Additional Configuration Options: --extra [EXTRASETTINGS ...] Extra environment variables to set (e.g., foobar.env:VARIABLE_NAME=value) … ``` -------------------------------- ### Run Malcolm Installer Mock Tests with Unittest Discovery Source: https://github.com/idaholab/malcolm/blob/main/scripts/installer/tests/README.md Discover and run mock tests using Python's unittest module. Run from the repository root. ```bash python -m unittest discover -s scripts/installer/tests/mock -p "test_*.py" ``` -------------------------------- ### Create Dummy Certificate Files Source: https://github.com/idaholab/malcolm/blob/main/docs/hedgehog-standalone.md Create empty certificate files for CA, client certificate, and client key. These files are required for Malcolm to start, even if not actively used in standalone mode. ```bash touch ~/Malcolm/filebeat/certs/{ca.crt,client.crt,client.key} ``` -------------------------------- ### Install Helm Source: https://github.com/idaholab/malcolm/blob/main/docs/aws.md Installs Helm, the package manager for Kubernetes, on a Linux system. Requires curl and chmod. ```bash $ curl -fsSL \ -o /tmp/get_helm.sh \ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 $ chmod 700 /tmp/get_helm.sh $ /tmp/get_helm.sh $ helm version version.BuildInfo{Version:"v3.17.3", GitCommit:"e4da49785aa6e6ee2b86efd5dd9e43400318262b", GitTreeState:"clean", GoVersion:"go1.23.7"} ``` -------------------------------- ### Malcolm Configuration Menu - Save and Continue Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md After selecting desired configuration options in the Malcolm Configuration Menu, use the 's' action to save the settings and proceed to the next step of the installation. ```bash … --- Actions --- s. Save and Continue w. Where Is...? (search for settings) x. Exit Installer --------------------------------- Enter item number or action: s ``` -------------------------------- ### Package Malcolm for Standalone Docker Installation Source: https://github.com/idaholab/malcolm/blob/main/docs/contributing-release-prep.md This script packages Malcolm for a standalone Docker installation. It prompts for Kubernetes manifest inclusion and indicates the output tarball location. It also provides instructions for installing, configuring, and managing Malcolm services. ```bash $ mkdir releases $ cd releases $ ~/Malcolm/scripts/malcolm_appliance_packager.sh … mkdir: created directory … Package Kubernetes manifests in addition to docker-compose.yml [y/N]? y … Packaged Malcolm to "/home/romeogdetlevjr/Malcolm/releases/malcolm_20241008_215936_deadbeef.tar.gz" Do you need to package container images also [y/N]? n To install and configure Malcolm, run install.py To start, stop, restart, etc. Malcolm: Use the control scripts in the "scripts/" directory: - start (start Malcolm) - stop (stop Malcolm) - restart (restart Malcolm) - logs (monitor Malcolm logs) - wipe (stop Malcolm and clear its database) - auth_setup (change authentication-related settings) Malcolm services can be accessed at https:/// $ ls -l total 749,568 drwxrwxr-x 10 romeogdetlevjr romeogdetlevjr 156 Oct 29 14:15 installer -rwxrwxr-x 1 romeogdetlevjr romeogdetlevjr 44,201 Oct 29 14:15 install.py -rw-rw-r-- 1 romeogdetlevjr romeogdetlevjr 460 Oct 29 14:15 malcolm_20251029_140727_d22a504f.README.txt -rw-rw-r-- 1 romeogdetlevjr romeogdetlevjr 275,657 Oct 29 14:15 malcolm_20251029_140727_d22a504f.tar.gz -rw-rw-r-- 1 romeogdetlevjr romeogdetlevjr 75,769 Oct 29 14:15 malcolm_common.py -rw-rw-r-- 1 romeogdetlevjr romeogdetlevjr 5,685 Oct 29 14:15 malcolm_constants.py -rw-rw-r-- 1 romeogdetlevjr romeogdetlevjr 50,329 Oct 29 14:15 malcolm_kubernetes.py -rw-rw-r-- 1 romeogdetlevjr romeogdetlevjr 36,952 Oct 29 14:15 malcolm_utils.py ``` -------------------------------- ### Install Build Dependencies Source: https://github.com/idaholab/malcolm/blob/main/hedgehog-raspi/README.md Install required packages for building the Raspberry Pi image on Debian Bullseye or higher. ```shell apt install -y vmdb2 dosfstools qemu-utils qemu-user-static debootstrap binfmt-support time kpartx bmap-tools python3 zerofree apt install -y fakemachine ``` -------------------------------- ### Start Malcolm Deployment on Kubernetes Source: https://github.com/idaholab/malcolm/blob/main/docs/kubernetes.md This command initiates the Malcolm services on Kubernetes using a specified kubeconfig file. It shows the typical output during startup, including the status of running pipelines. ```bash $ ./scripts/start -f /path/to/kubeconfig.yaml ... logstash | [2026-01-16T17:58:33,274][INFO ][logstash.agent ] Pipelines running {:count=>7, :running_pipelines=>[:"malcolm-output", :"malcolm-input", :"malcolm-filescan", :"malcolm-suricata", :"malcolm-enrichment", :"malcolm-beats", :"malcolm-zeek"], :non_running_pipelines=>[]} Started Malcolm Malcolm services can be accessed at https://192.168.121.239/ ------------------------------------------------------------------------------ ``` -------------------------------- ### Configure Malcolm Installation Source: https://github.com/idaholab/malcolm/blob/main/docs/malcolm-hedgehog-e2e-iso-install.md This prompt appears during the Malcolm installation process. It confirms the network and security configuration before proceeding. ```bash HTTPS/SSL : Yes Node Name : host ============================================================ Proceed with Malcolm installation using the above configuration? (y / N): y ``` -------------------------------- ### Malcolm Final Configuration Summary Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md This summary displays the selected configuration settings before the final installation confirmation. Users can review and confirm the settings. ```text ============================================================ FINAL CONFIGURATION SUMMARY ============================================================ Configuration Only : No Auto Apply System Tweaks : Yes Configuration Directory : /home/user/Malcolm/config Container Runtime : docker Run Profile : malcolm Process UID/GID : 1000/1000 Container Restart Policy : unless-stopped Container Network : default Default Storage Locations : Yes HTTPS/SSL : Yes Node Name : host ============================================================ Proceed with Malcolm installation using the above configuration? (y / N): y ``` -------------------------------- ### Install Python Libraries for Malcolm Source: https://github.com/idaholab/malcolm/blob/main/docs/ubuntu-install-example.md Installs required Python libraries for Malcolm using apt-get. This includes 'ruamel.yaml' and 'python-dotenv'. ```bash $ sudo apt-get -y -qq update $ sudo apt-get -y install python3-ruamel.yaml python3-dotenv Reading package lists... Done … The following NEW packages will be installed: python3-dotenv python3-ruamel.yaml python3-ruamel.yaml.clib … Setting up python3-dotenv (1.0.1-1) ... Setting up python3-ruamel.yaml.clib:amd64 (0.2.8-1build1) ... Setting up python3-ruamel.yaml (0.17.21-1) ... … ``` -------------------------------- ### Configure Local OpenSearch Instance Source: https://github.com/idaholab/malcolm/blob/main/docs/opensearch-instances.md Select 'opensearch-local' for the Primary Document Store to use Malcolm's default standalone configuration with a local OpenSearch instance. ```bash ... ├── 6. Run Profile (current: malcolm) ... │ ├── 22. OpenSearch Memory (current: 31g) │ └── 23. Primary Document Store (current: opensearch-local) ... ```