### API Key Configuration File Example Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/README.md This snippet shows the expected format for the `~/.shadowserver.api` configuration file. It contains the API key, secret, and the API URI required for the `call-api` programs to authenticate and connect to the Shadowserver API. ```INI [api] key = <> secret = <> uri = https://transform.shadowserver.org/api2/ ``` -------------------------------- ### Example Configuration File (config.ini) - INI Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/cef/README.md This example `config.ini` file demonstrates the structure for configuring the Shadowserver CEF logger. It includes a `[general]` section for global settings like state directory, API credentials, and auto-update, and a `[device_id]` section for specific report directives such as syslog facility and report types. ```INI [general] state_directory=/var/lib/cef/state apikey = secret = auto_update=true [device_id] facility=user types=device_id ``` -------------------------------- ### Report Manager Crontab Schedule Example Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/README.md This snippet provides an example crontab entry to schedule `report-manager.py` to run hourly. It shows how to configure the cron job to execute the script with its configuration file, ensuring regular checks for new report downloads. ```Crontab 15 * * * * /opt/shadowserver/report-manager.py /opt/shadowserver/reports.ini ``` -------------------------------- ### Report Manager Configuration File Example Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/README.md This snippet provides a sample configuration file for `report-manager.py`, detailing settings for report storage, disk space management, and various notification queue options like STOMP, Redis, and Kafka. It illustrates how to configure the `[reports]` section and specific notifier sections. ```INI [reports] directory = /var/tmp/reports min_disk_free = 512 notifier = none url_prefix = http://myserver/reports/ [stomp] server = 127.0.0.1 port = 61613 user = guest password = guest queue = /queue/mytest [redis] server = 127.0.0.1 port = 6379 ;password = guest queue = mytest [kafka] server = 127.0.0.1 port = 9092 queue = mytest ``` -------------------------------- ### Example Configuration for Shadowserver ECS Logger Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/elasticsearch/README.md This `config.ini` file defines settings for the `shadowserver_ecs_logger.py` script, including the state directory, Shadowserver API credentials, and an optional `auto_update` flag. It also specifies report types and the log output path for Filebeat consumption. ```INI [general] state_directory=/var/lib/ecs/state apikey = secret = auto_update=true [device_id] types=device_id log=/var/lib/ecs/filebeat ``` -------------------------------- ### Example Filebeat Configuration for Shadowserver Logs Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/elasticsearch/README.md This `filebeat.yml` configuration sets up Filebeat to ingest JSON logs generated by the Shadowserver ECS logger. It defines input paths, JSON processing rules (e.g., `keys_under_root`, `overwrite_keys`), and processors to drop unnecessary fields before sending data to Elasticsearch. ```YAML filebeat.inputs: - type: log id: shadowserver-feeds enabled: true paths: - /var/lib/ecs/filebeat/*.json json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true publisher_pipeline.disable_host: true harvester_limit: 8 scan_frequency: 1m close_inactive: 5m ignore_older: 24h clean_inactive: 25h processors: - drop_fields: when: equals: input.type: "log" fields: - "agent.ephemeral_id" - "agent.hostname" - "agent.name" - "agent.id" - "agent.type" - "agent.version" - "ecs.version" - "input.type" - "process.name" - "process.pid" - "process.thread.id" - "process.thread.name" - "log.original" - "log.offset" - "log.level" - "log.origin.function" - "log.origin.file.name" - "log.origin.file.line" - "log.logger" - "log.file.path" setup.template.settings: index.number_of_shards: 1 setup.template: name: "filebeat" pattern: "filebeat" setup.kibana: host: "http://127.0.0.1:5601" ssl.verification_mode: none output.elasticsearch: # Array of hosts to connect to. hosts: ["https://127.0.0.1:9200"] ssl.certificate_authorities: ["/opt/elasticsearch/config/certs/http_ca.crt"] # Index by report type and month index: "shadowserver-%{[data_stream.dataset]}-%{+yyyy-MM}" # Authentication credentials - either API key or username/password. #api_key: "beats:OXlsZmQ0b0JnLUTPwjbKCtrtRG06R2tXaUdxdmdURTJLa0Ytdk1Ya1pXdw==" username: "elastic" password: "YvInisCyhKtwpCkFY2F+" ``` -------------------------------- ### Python API Call Timeout Error Example Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/README.md This snippet demonstrates a common timeout error encountered when using `call-api.py` for large data queries. It shows the command execution and the resulting `API Exception: The read operation timed out` message, indicating that the program's timeout setting needs to be increased. ```Shell $ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"city":"ashburn"}, "limit":3}' pretty API Exception: The read operation timed out ``` -------------------------------- ### Custom Logs Integration: Drop Fields Processor Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/elasticsearch/README.md This YAML snippet configures a `drop_fields` processor for a Custom Logs integration in Elastic, mirroring the Filebeat setup. It specifies a condition (`input.type: "log"`) to remove a comprehensive list of agent, ECS, process, and log-related fields, optimizing data storage and relevance. ```YAML - drop_fields: when: equals: input.type: "log" fields: - "agent.ephemeral_id" - "agent.hostname" - "agent.name" - "agent.id" - "agent.type" - "agent.version" - "ecs.version" - "input.type" - "process.name" - "process.pid" - "process.thread.id" - "process.thread.name" - "log.original" - "log.offset" - "log.level" - "log.origin.function" - "log.origin.file.name" - "log.origin.file.line" - "log.logger" - "log.file.path" ``` -------------------------------- ### Manually Running Shadowserver Reports Add-on in Splunk (Shell) Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/splunk/README.md This shell command allows for a manual execution of the Shadowserver Reports add-on for a specific instance. It navigates to the Splunk home directory, prints the modular input configuration for the specified instance, and pipes it to the add-on's Python script, effectively simulating an input run and sending data to stdout for debugging or verification. ```Shell (cd $SPLUNK_HOME;bin/splunk cmd splunkd print-modinput-config shadowserver_reports shadowserver_reports://device_id | bin/splunk cmd python etc/apps/shadowserver_reports/bin/shadowserver_reports.py) ``` -------------------------------- ### Configuring Shadowserver Reports Input in Splunk (INI) Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/splunk/README.md This configuration snippet demonstrates how to set up a Shadowserver Reports data input in Splunk's inputs.conf file. It specifies the API key, secret, and the type of report to import (e.g., 'device_id'), along with enabling the input. This allows Splunk to automatically fetch and index the specified Shadowserver intelligence. ```INI [shadowserver_reports://device_id] api_key = ........-....-....-....-........... secret = .......... types = device_id disabled = 0 ``` -------------------------------- ### Running Shadowserver CEF Logger Script - Shell Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/cef/README.md This command executes the `shadowserver_cef_logger.py` script using Python 3, providing the path to the configuration file (`config.ini`) as an argument. This is the standard way to run the script for processing Shadowserver intelligence reports. ```Shell $ python3 shadowserver_cef_logger.py config.ini ``` -------------------------------- ### Running Shadowserver ECS Logger Script Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/elasticsearch/README.md This command executes the `shadowserver_ecs_logger.py` script, which processes Shadowserver intelligence reports and logs them in ECS format. It requires a configuration file (e.g., `config.ini`) to specify API keys, state directories, and report types. ```Bash $ python3 shadowserver_ecs_logger.py config.ini ``` -------------------------------- ### Updating Shadowserver CEF Mapping - Shell Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/cef/README.md This command executes the `shadowserver_cef_logger.py` script with the `update` argument, along with the configuration file. This specifically triggers an update of the dynamic report field mapping to the Common Event Format (CEF), ensuring the latest mapping is used. ```Shell $ python3 shadowserver_cef_logger.py config.ini update ``` -------------------------------- ### Custom Logs Integration: JSON Processing Settings Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/elasticsearch/README.md This YAML snippet provides JSON processing settings for a Custom Logs integration in Elastic, equivalent to those found in `filebeat.yml`. These settings ensure that JSON fields are parsed correctly, placed at the root of the document, and that existing keys can be overwritten, facilitating proper ECS mapping. ```YAML json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true ``` -------------------------------- ### Report Manager Notification JSON Structure Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/README.md This snippet illustrates the JSON object structure used for notifications sent by `report-manager.py`. It includes fields for `timestamp`, `report_date`, `report_type`, and `uri`, providing details about the downloaded report. ```JSON { "timestamp" : "2022-09-01 11:32:45", "report_date" : "2022-08-31", "report_type" : "scan_stun", "uri" : "http://myserver/reports/2022/08/31/2022-08-31-scan_stun_example_com-asn.csv" } ``` -------------------------------- ### Updating Shadowserver ECS Logger Mapping Source: https://github.com/the-shadowserver-foundation/api_utils/blob/main/elasticsearch/README.md This command runs the `shadowserver_ecs_logger.py` script with the 'update' argument, forcing it to download and apply the latest dynamic mapping for Shadowserver reports to align with the Elastic Common Schema (ECS). This is useful for manual updates or when `auto_update` is disabled. ```Bash $ python3 shadowserver_ecs_logger.py config.ini update ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.