### Install Dependencies Source: https://github.com/prometheus/alertmanager/blob/main/ui/app/CONTRIBUTING.md Run this command to install all project dependencies. Ensure you have Node.js and npm installed. ```sh npm ci ``` -------------------------------- ### Run New Mantine UI Dev Server Source: https://github.com/prometheus/alertmanager/blob/main/AGENTS.md Install dependencies and start the development server for the new Mantine UI. This command is used for frontend development. ```bash cd ui/mantine-ui && npm install && npm run dev ``` -------------------------------- ### Install amtool using Go Source: https://github.com/prometheus/alertmanager/blob/main/README.md Install the amtool CLI using the Go build tool. Ensure Go is installed and configured on your system. ```bash $ go install github.com/prometheus/alertmanager/cmd/amtool@latest ``` -------------------------------- ### Run Legacy Elm UI Dev Server Source: https://github.com/prometheus/alertmanager/blob/main/AGENTS.md Start the development server for the legacy Elm UI. Ensure you are in the 'ui/app' directory and refer to 'ui/app/CONTRIBUTING.md' for setup details. ```bash cd ui/app && make build ``` -------------------------------- ### Start Development Server Source: https://github.com/prometheus/alertmanager/blob/main/ui/app/CONTRIBUTING.md Navigate to the UI app directory and run this command to start the development server. Changes to Elm files will trigger recompilation. ```sh # cd ui/app # make dev-server ``` -------------------------------- ### Email TLS Configuration Examples Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Examples demonstrating how to configure TLS for email notifications, including forcing implicit TLS and using default auto-detection based on port. ```yaml # Example 1: Force implicit TLS on any port (recommended for security) receivers: - name: email-implicit-tls email_configs: - to: alerts@example.com smarthost: smtp.example.com:8465 force_implicit_tls: true # Use direct TLS connection on port 8465 # Example 2: Backward compatible (no force_implicit_tls specified) receivers: - name: email-default email_configs: - to: alerts@example.com smarthost: smtp.example.com:465 # Auto-detects implicit TLS - to: alerts@example.com smarthost: smtp.example.com:587 # Auto-detects explicit TLS ``` -------------------------------- ### Annotation Example Source: https://github.com/prometheus/alertmanager/blob/main/docs/notifications.md An example of an annotation map containing summary and description. ```go { summary: "alert summary", description: "alert description", } ``` -------------------------------- ### Run Multi-Node HA Alertmanager with Goreman Source: https://github.com/prometheus/alertmanager/blob/main/AGENTS.md Start a multi-node High Availability setup for Alertmanager using Goreman. This requires Goreman to be installed. ```bash goreman start ``` -------------------------------- ### Tracing Configuration Example Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure the tracing client type, endpoint, sampling fraction, and TLS settings. ```yaml # The tracing client type, supported values are `http` and `grpc`. [ client_type: | default = "grpc" ] # The tracing endpoint. [ endpoint: | default = "" ] # The sampling fraction. [ sampling_fraction: | default = 0.0 ] # Whether to disable TLS. [ insecure: | default = false ] # The HTTP client's configuration. [ tls_config: ] # Custom HTTP headers. [ http_headers: [ ] ] # The tracing compression. [ compression: | default = "gzip" ] # The tracing timeout. [ timeout: | default = 0s ] ``` -------------------------------- ### Alertmanager Configuration File Example Source: https://github.com/prometheus/alertmanager/blob/main/README.md An example YAML configuration file for amtool, specifying Alertmanager URL, author, comment requirements, output format, and default receiver. ```yaml # Define the path that `amtool` can find your `alertmanager` instance alertmanager.url: "http://localhost:9093" # Override the default author. (unset defaults to your username) author: me@example.com # Force amtool to give you an error if you don't include a comment on a silence comment_required: true # Set a default output format. (unset defaults to simple) output: extended # Set a default receiver receiver: team-X-pager ``` -------------------------------- ### Alertmanager Routing Configuration Example Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md This example demonstrates a root route with nested child routes, including matching conditions, grouping, and time interval muting/activation. ```yaml # The root route with all parameters, which are inherited by the child # routes if they are not overwritten. route: receiver: 'default-receiver' group_wait: 30s group_interval: 5m repeat_interval: 4h group_by: [cluster, alertname] # All alerts that do not match the following child routes # will remain at the root node and be dispatched to 'default-receiver'. routes: # All alerts with service=mysql or service=cassandra # are dispatched to the database pager. - receiver: 'database-pager' group_wait: 10s matchers: - service=~"mysql|cassandra" # All alerts with the team=frontend label match this sub-route. # They are grouped by product and environment rather than cluster # and alertname. - receiver: 'frontend-pager' group_by: [product, environment] matchers: - team="frontend" # All alerts with the service=inhouse-service label match this sub-route. # the route will be muted during offhours and holidays time intervals. # even if it matches, it will continue to the next sub-route - receiver: 'dev-pager' matchers: - service="inhouse-service" mute_time_intervals: - offhours - holidays continue: true # All alerts with the service=inhouse-service label match this sub-route # the route will be active only during offhours and holidays time intervals. - receiver: 'on-call-pager' matchers: - service="inhouse-service" active_time_intervals: - offhours - holidays ``` -------------------------------- ### Basic Configuration Check with Warnings Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Use `amtool check-config` to validate a configuration file. This example shows output with warnings for UTF-8 incompatibility and label disagreement. ```bash amtool check-config config.yml Checking 'config.yml' level=warn msg="Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible. Alertmanager has instead parsed the input using the classic matchers parser as a fallback. To make this input compatible with the UTF-8 matchers parser please make sure all regular expressions and values are double-quoted and backslashes are escaped. If you are still seeing this message please open an issue." input="foo=" origin=config err="end of input: expected label value" suggestion="foo=\"\"" level=warn msg="Matchers input has disagreement" input="qux=\"\\xf0\\x9f\\x99\\x82\"\n" origin=config SUCCESS Found: - global config - route - 2 inhibit rules - 2 receivers - 0 templates ``` -------------------------------- ### Run Alertmanager Locally with Configuration Source: https://github.com/prometheus/alertmanager/blob/main/AGENTS.md Start a local instance of Alertmanager using a specified configuration file. This is useful for testing configurations. ```bash ./alertmanager --config.file=doc/examples/simple.yml ``` -------------------------------- ### Alertmanager Configuration Example Source: https://github.com/prometheus/alertmanager/blob/main/README.md This YAML configuration covers global settings, routing rules with matchers and sub-routes, inhibition rules, and receiver definitions for email and PagerDuty notifications. It is intended for users who need a detailed example of Alertmanager's configuration capabilities. ```yaml global: # The smarthost and SMTP sender used for mail notifications. smtp_smarthost: 'localhost:25' smtp_from: 'alertmanager@example.org' # The root route on which each incoming alert enters. route: # The root route must not have any matchers as it is the entry point for # all alerts. It needs to have a receiver configured so alerts that do not # match any of the sub-routes are sent to someone. receiver: 'team-X-mails' # The labels by which incoming alerts are grouped together. For example, # multiple alerts coming in for cluster=A and alertname=LatencyHigh would # be batched into a single group. # # To aggregate by all possible labels use '...' as the sole label name. # This effectively disables aggregation entirely, passing through all # alerts as-is. This is unlikely to be what you want, unless you have # a very low alert volume or your upstream notification system performs # its own grouping. Example: group_by: [...] group_by: ['alertname', 'cluster'] # When a new group of alerts is created by an incoming alert, wait at # least 'group_wait' to send the initial notification. # This way ensures that you get multiple alerts for the same group that start # firing shortly after another are batched together on the first # notification. group_wait: 30s # When the first notification was sent, wait 'group_interval' to send a batch # of new alerts that started firing for that group. group_interval: 5m # If an alert has successfully been sent, wait 'repeat_interval' to # resend them. repeat_interval: 3h # All the above attributes are inherited by all child routes and can # overwritten on each. # The child route trees. routes: # This route performs a regular expression match on alert labels to # catch alerts that are related to a list of services. - matchers: - service=~"^(foo1|foo2|baz)$" receiver: team-X-mails # The service has a sub-route for critical alerts, any alerts # that do not match, i.e. severity != critical, fall-back to the # parent node and are sent to 'team-X-mails' routes: - matchers: - severity="critical" receiver: team-X-pager - matchers: - service="files" receiver: team-Y-mails routes: - matchers: - severity="critical" receiver: team-Y-pager # This route handles all alerts coming from a database service. If there's # no team to handle it, it defaults to the DB team. - matchers: - service="database" receiver: team-DB-pager # Also group alerts by affected database. group_by: [alertname, cluster, database] routes: - matchers: - owner="team-X" receiver: team-X-pager - matchers: - owner="team-Y" receiver: team-Y-pager # Inhibition rules allow to mute a set of alerts given that another alert is # firing. # We use this to mute any warning-level notifications if the same alert is # already critical. inhibit_rules: - source_matchers: - severity="critical" target_matchers: - severity="warning" # Apply inhibition if the alertname is the same. # CAUTION: # If all label names listed in `equal` are missing # from both the source and target alerts, # the inhibition rule will apply! equal: ['alertname'] receivers: - name: 'team-X-mails' email_configs: - to: 'team-X+alerts@example.org, team-Y+alerts@example.org' - name: 'team-X-pager' email_configs: - to: 'team-X+alerts-critical@example.org' pagerduty_configs: - routing_key: '' - name: 'team-Y-mails' email_configs: - to: 'team-Y+alerts@example.org' - name: 'team-Y-pager' pagerduty_configs: - routing_key: '' - name: 'team-DB-pager' pagerduty_configs: - routing_key: '' ``` -------------------------------- ### Time Interval Example with Location Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md An example of a time interval definition that includes specific times, weekdays, and a time zone location. ```yaml times: - start_time: 09:00 end_time: 17:00 weekdays: ['monday:friday'] location: 'Australia/Sydney' ``` -------------------------------- ### Render Template with Example Data Source: https://github.com/prometheus/alertmanager/blob/main/README.md Tests how a specific Alertmanager template would render with provided data. This command is useful for template development and debugging. ```bash amtool template render --template.glob='/foo/bar/*.tmpl' --template.text='{{ template "slack.default.markdown.v1" . }}' ``` -------------------------------- ### Alertmanager Cluster Configuration Example Source: https://github.com/prometheus/alertmanager/blob/main/docs/high_availability.md Configure Alertmanager cluster communication using command-line flags. Ensure all peers are listed for effective gossip and state synchronization. ```bash alertmanager \ --cluster.listen-address=0.0.0.0:9094 \ --cluster.peer=am-1.example.com:9094 \ --cluster.peer=am-2.example.com:9094 \ --cluster.peer=am-3.example.com:9094 \ --cluster.advertise-address=192.0.2.1:9094 \ --cluster.peer-timeout=15s \ --cluster.gossip-interval=200ms \ --cluster.pushpull-interval=60s ``` -------------------------------- ### Jira Labels Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Example of configuring labels for a Jira issue, supporting template expressions for dynamic label generation. ```yaml labels: - 'alertmanager' - '{{ .CommonLabels.severity }}' ``` -------------------------------- ### Prometheus Configuration for Alertmanager Cluster Source: https://github.com/prometheus/alertmanager/blob/main/README.md Example prometheus.yml configuration to point to multiple Alertmanager instances for high availability. ```yaml alerting: alertmanagers: - static_configs: - targets: - alertmanager1:9093 - alertmanager2:9093 - alertmanager3:9093 ``` -------------------------------- ### Alertmanager Route Configuration Example Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md This snippet shows the structure and available parameters for configuring a route in Alertmanager. It includes settings for receivers, grouping, matching, notification timing, and nested routes. ```yaml route: receiver: "default-receiver" group_by: ["alertname", "cluster", "service"] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: - receiver: "critical-receiver" matchers: - severity == "critical" continue: true - receiver: "warning-receiver" matchers: - severity == "warning" routes: - receiver: "database-receiver" matchers: - service == "database" ``` -------------------------------- ### Enable Classic Mode in Alertmanager Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Use this command to start Alertmanager in classic mode, which is equivalent to versions 0.26.0 and older. This mode can be used for troubleshooting if issues are suspected with fallback or UTF-8 strict modes. ```bash alertmanager --config.file=config.yml --enable-feature="classic-mode" ``` -------------------------------- ### Jira Custom Fields Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Example of configuring various Jira custom fields, including components, text fields, select lists, and multi-select lists. ```yaml fields: # Components components: { name: "Monitoring" } # Custom Field TextField customfield_10001: "Random text" # Custom Field SelectList customfield_10002: {"value": "red"} # Custom Field MultiSelect customfield_10003: [{"value": "red"}, {"value": "blue"}, {"value": "green"}] ``` -------------------------------- ### Enable UTF-8 Strict Mode in Alertmanager Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Use this command to start Alertmanager in UTF-8 strict mode. This mode disables support for classic matchers and requires all matchers to be UTF-8 compatible. Ensure all compatibility warnings are resolved before enabling. ```bash alertmanager --config.file=config.yml --enable-feature="utf8-strict-mode" ``` -------------------------------- ### Alertmanager Configuration Error: UTF-8 Strict Mode Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md This error message appears when Alertmanager fails to start in UTF-8 strict mode because of unresolved incompatible matchers in the configuration file. All warnings related to matcher compatibility must be addressed before UTF-8 strict mode can be enabled. ```log ts=2024-02-11T10:00:00Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=config.yml err="end of input: expected label value" ``` -------------------------------- ### Build Alertmanager from Source Source: https://github.com/prometheus/alertmanager/blob/main/README.md Clones the Alertmanager repository and builds the binary. Requires Go and Node.js. The --config.file flag specifies the configuration file path. ```bash git clone https://github.com/prometheus/alertmanager.git cd alertmanager make build ./alertmanager --config.file= ``` -------------------------------- ### Build All Assets Source: https://github.com/prometheus/alertmanager/blob/main/ui/app/CONTRIBUTING.md Run this command at the root level of the repository before committing changes to ensure all assets are built. ```sh make build-all ``` -------------------------------- ### Whitespace Insensitive Matcher Example Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Whitespace, including spaces, tabs, and newlines, is permitted outside of double quotes and does not affect the matching logic. This example shows equivalent matchers with different whitespace formatting. ```yaml { alertname = "Watchdog", severity =~ "warning|critical", } ``` ```yaml {alertname="Watchdog",severity=~"warning|critical"} ``` -------------------------------- ### Time Range Definition Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Specifies a time range within a time interval, inclusive of the start time and exclusive of the end time. ```yaml times: - start_time: HH:MM end_time: HH:MM ``` -------------------------------- ### Run Common Development Tasks Source: https://github.com/prometheus/alertmanager/blob/main/AGENTS.md Execute common formatting, linting, and testing commands before pushing code. Ensure your code adheres to project standards. ```bash make common-format make lint make test ``` -------------------------------- ### Time and Duration Functions Source: https://github.com/prometheus/alertmanager/blob/main/docs/notifications.md Functions for working with time and durations, including getting the current time, formatting time, and calculating time differences. ```APIDOC ## date ### Description Returns the text representation of the time in the specified format. For documentation on formats refer to [pkg.go.dev/time](https://pkg.go.dev/time#pkg-constants). ### Arguments - format (string) - The time format string. - t (time.Time) - The time value. ``` ```APIDOC ## humanizeDuration ### Description Returns a human-readable string representing the duration, and the error if it happened. ### Arguments - duration (number or string) - The duration to humanize. ``` ```APIDOC ## now ### Description Returns the current local time. Equivalent to [time.Now](https://pkg.go.dev/time#Now). ### Arguments - None. ``` ```APIDOC ## since ### Description Returns the duration of how much time passed from the provided time till the current system time. Equivalent to [time.Since](https://pkg.go.dev/time#Since). ### Arguments - t (time.Time) - The reference time. ``` ```APIDOC ## tz ### Description Returns the time in the specified timezone. For example, 'Europe/Paris'. ### Arguments - timezone (string) - The IANA timezone name. - t (time.Time) - The time value. ``` -------------------------------- ### Run Tests Source: https://github.com/prometheus/alertmanager/blob/main/ui/app/CONTRIBUTING.md Execute this command to run all project tests. This is a standard make target for testing. ```sh make test ``` -------------------------------- ### Specify Configuration File Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Use the `--config.file` flag to specify the Alertmanager configuration file to load. This is essential for defining custom routing and receiver settings. ```bash ./alertmanager --config.file=alertmanager.yml ``` -------------------------------- ### Run Alertmanager with Docker Source: https://github.com/prometheus/alertmanager/blob/main/README.md Launches an Alertmanager container for testing purposes. Ensure Alertmanager is reachable at http://localhost:9093/. ```bash docker run --name alertmanager -d -p 127.0.0.1:9093:9093 quay.io/prometheus/alertmanager ``` -------------------------------- ### Successful Basic Configuration Check Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md A successful basic configuration check with `amtool check-config` indicates no warnings or errors. ```bash amtool check-config config.yml Checking 'config.yml' SUCCESS Found: - global config - route - 2 inhibit rules - 2 receivers - 0 templates ``` -------------------------------- ### File Output Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure writing events as JSON lines to a file. The file is reopened on directory rename/remove/create events. ```yaml # Path to the JSONL output file. Will be created if it does not exist. path: ``` -------------------------------- ### YAML Block Matchers with Escaping Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Utilize a YAML block for complex matchers, especially when dealing with intricate strings containing quotes and special characters. This example demonstrates proper escaping for double quotes within the matcher value. ```yaml matchers: - | {quote=~"She said: \"Hi, all!( How're you…)?\""} ``` -------------------------------- ### Readiness Check API Source: https://github.com/prometheus/alertmanager/blob/main/docs/management_api.md Checks if the Alertmanager instance is ready to serve traffic. Returns 200 OK when ready. ```APIDOC ## Readiness Check API ### Description Checks if the Alertmanager instance is ready to serve traffic. Returns 200 OK when ready. ### Method GET, HEAD ### Endpoint /-/ready ### Parameters #### Path Parameters None #### Query Parameters None ### Request Example None ### Response #### Success Response (200) Returns an empty response with a 200 OK status when Alertmanager is ready. #### Response Example ``` -------------------------------- ### Run Single Go Test Package Source: https://github.com/prometheus/alertmanager/blob/main/AGENTS.md Execute tests for a specific Go package or a particular test within a package. Use this for targeted testing during development. ```go go test ./notify/slack/... ``` ```go go test ./dispatch -run TestDispatcherRace -v ``` -------------------------------- ### Alertmanager Fallback Mode Warning: Matcher Disagreement Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md This log message indicates a disagreement between the UTF-8 and classic parsers for a matcher that is valid in both but parses differently. Alertmanager defaults to the classic parser in such cases. The example shows how to adjust the matcher if the intention is to match literal bytes rather than UTF-8 characters. ```log ts=2024-02-11T10:00:00Z caller=parse.go:183 level=warn msg="Matchers input has disagreement" input="qux=\"\\xf0\\x9f\\x99\\x82\"\n" origin=config ``` -------------------------------- ### Query Firing Alerts by Instance Regex Source: https://github.com/prometheus/alertmanager/blob/main/README.md Filters and displays alerts where the instance label matches a regular expression. This allows for flexible alert selection based on instance patterns. ```bash $ amtool -o extended alert query instance=~"\\.+1" ``` -------------------------------- ### View Firing Alerts with Extended Output Source: https://github.com/prometheus/alertmanager/blob/main/README.md Shows all currently firing alerts with detailed information, including labels, annotations, and timestamps. Use this for more in-depth analysis. ```bash $ amtool -o extended alert ``` -------------------------------- ### Build Specific Alertmanager Binary Source: https://github.com/prometheus/alertmanager/blob/main/README.md Builds a specific binary within the Alertmanager repository, such as 'amtool'. ```bash make build BINARIES=amtool ``` -------------------------------- ### Event Recorder Outputs Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure different output types for the event recorder, including file, webhook, and Kafka. ```yaml # JSONL file outputs. file_outputs: [ - ... ] # Webhook outputs. webhook_outputs: [ - ... ] # Kafka outputs. kafka_outputs: [ - ... ] ``` -------------------------------- ### amtool Command for Viewing Routing Tree Source: https://github.com/prometheus/alertmanager/blob/main/README.md Use this command to view the routing tree of a remote Alertmanager instance. ```bash # View routing tree of remote Alertmanager $ amtool config routes --alertmanager.url=http://localhost:9093 ``` -------------------------------- ### Global Configuration Options Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Defines global settings and default values for various notification integrations and client configurations. These parameters apply across all other configuration contexts. ```yaml global: # The default SMTP From header field. [ smtp_from: ] # The default SMTP smarthost used for sending emails, including port number. # Port number usually is 25, or 587 for SMTP over TLS (sometimes referred to as STARTTLS). # Example: smtp.example.org:587 [ smtp_smarthost: ] # The default hostname to identify to the SMTP server. [ smtp_hello: | default = "localhost" ] # SMTP Auth using CRAM-MD5, LOGIN and PLAIN. If empty, Alertmanager doesn't authenticate to the SMTP server. # PLAIN is only supported when using TLS. [ smtp_auth_username: ] # SMTP Auth using LOGIN and PLAIN. [ smtp_auth_password: ] # SMTP Auth using LOGIN and PLAIN. [ smtp_auth_password_file: ] # SMTP Auth using PLAIN. [ smtp_auth_identity: ] # SMTP Auth using CRAM-MD5. [ smtp_auth_secret: ] # SMTP Auth using CRAM-MD5. [ smtp_auth_secret_file: ] # The default SMTP TLS requirement. # Note that Go does not support unencrypted connections to remote SMTP endpoints. [ smtp_require_tls: | default = true ] # The default TLS configuration for SMTP receivers [ smtp_tls_config: ] # Force implicit TLS regardless of SMTP port [ smtp_force_implicit_tls: ] # Default settings for the JIRA integration. [ jira_api_url: ] # The API URL to use for Slack notifications. [ slack_api_url: ] [ slack_api_url_file: ] [ slack_app_token: ] [ slack_app_token_file: ] [ slack_app_url: ] [ victorops_api_key: ] [ victorops_api_key_file: ] [ victorops_api_url: | default = "https://alert.victorops.com/integrations/generic/20131114/alert/" ] [ pagerduty_url: | default = "https://events.pagerduty.com/v2/enqueue" ] [ opsgenie_api_key: ] [ opsgenie_api_key_file: ] [ opsgenie_api_url: | default = "https://api.opsgenie.com/" ] [ rocketchat_api_url: | default = "https://open.rocket.chat/" ] [ rocketchat_token: ] [ rocketchat_token_file: ] [ rocketchat_token_id: ] [ rocketchat_token_id_file: ] [ wechat_api_url: | default = "https://qyapi.weixin.qq.com/cgi-bin/" ] [ wechat_api_secret: ] [ wechat_api_secret_file: ] [ wechat_api_corp_id: ] [ telegram_api_url: | default = "https://api.telegram.org" ] # The default Telegram bot token. It is mutually exclusive with `telegram_bot_token_file`. [ telegram_bot_token: ] # The default configuration to read the Telegram bot token from a file. It is mutually exclusive with `telegram_bot_token`. [ telegram_bot_token_file: ] [ webex_api_url: | default = "https://webexapis.com/v1/messages" ] [ mattermost_webhook_url: ] [ mattermost_webhook_url_file: ] # The default HTTP client configuration [ http_config: ] # ResolveTimeout is the default value used by alertmanager if the alert does # not include EndsAt, after this time passes it can declare the alert as resolved if it has not been updated. # This has no impact on alerts from Prometheus, as they always include EndsAt. [ resolve_timeout: | default = 5m ] ``` -------------------------------- ### Successful UTF-8 Strict Mode Configuration Check Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md A successful check with `--enable-feature="utf8-strict-mode"` confirms the configuration is valid and UTF-8 compliant. ```bash amtool check-config config.yml --enable-feature="utf8-strict-mode" level=warn msg="UTF-8 mode enabled" Checking 'config.yml' SUCCESS Found: - global config - route - 2 inhibit rules - 2 receivers - 0 templates ``` -------------------------------- ### HTTP Client Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure the HTTP client for receivers, including authentication, proxies, and TLS settings. Basic auth and authorization are mutually exclusive. ```yaml # Note that `basic_auth` and `authorization` options are mutually exclusive. # Sets the `Authorization` header with the configured username and password. # password and password_file are mutually exclusive. basic_auth: [ username: ] [ password: ] [ password_file: ] # Optional the `Authorization` header configuration. authorization: # Sets the authentication type. [ type: | default: Bearer ] # Sets the credentials. It is mutually exclusive with # `credentials_file`. [ credentials: ] # Sets the credentials with the credentials read from the configured file. # It is mutually exclusive with `credentials`. [ credentials_file: ] # Optional OAuth 2.0 configuration. # Cannot be used at the same time as basic_auth or authorization. oauth2: [ ] # Whether to enable HTTP2. [ enable_http2: | default: true ] # Optional proxy URL. [ proxy_url: ] # Comma-separated string that can contain IPs, CIDR notation, domain names # that should be excluded from proxying. IP and domain names can # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, http_proxy, HTTPS_PROXY, https_proxy, NO_PROXY, and no_proxy) [ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] # Configure whether HTTP requests follow HTTP 3xx redirects. [ follow_redirects: | default = true ] # Configures the TLS settings. tls_config: [ ] # Custom HTTP headers to be sent along with each request. # Headers that are set by Prometheus itself can't be overwritten. http_headers: [ ] ``` -------------------------------- ### Template File Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Specifies files from which custom notification template definitions are read. Supports wildcard matching for template files. ```yaml templates: [ - ... ] ``` -------------------------------- ### Tracing Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Optional configuration for distributed tracing within Alertmanager. ```yaml traciing: ``` -------------------------------- ### Enable Alertmanager Debug Logging Source: https://github.com/prometheus/alertmanager/blob/main/docs/high_availability.md Run the Alertmanager binary with the debug log level to troubleshoot notification pipeline and gossip issues. ```bash alertmanager --log.level=debug ``` -------------------------------- ### Query Firing Alerts by Name and Instance Regex Source: https://github.com/prometheus/alertmanager/blob/main/README.md Combines alert name filtering with instance regular expression matching to retrieve specific alerts. This provides a powerful way to narrow down alert views. ```bash $ amtool -o extended alert query alertname=~"Test.*" instance=~"\\.+1" ``` -------------------------------- ### Time Interval Configuration Structure Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Defines the basic structure for naming and listing time intervals in Alertmanager configuration. ```yaml name: time_intervals: [ - ... ] ``` -------------------------------- ### Kafka Output Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure producing events to a Kafka topic with options for brokers, topic, format, acknowledgements, compression, buffer size, and TLS. ```yaml # Seed broker list (host:port). At least one entry is required. brokers: [ - ... ] # Topic to produce events to. topic: # Client identifier reported to the brokers. [ client_id: | default = "alertmanager" ] # On-the-wire encoding for each record value: "json" (protojson) or # "protobuf" (binary proto). JSON is the default for symmetry with the # file and webhook outputs; consumers that already use the # eventrecorder.proto schema may prefer protobuf for compactness. [ format: <"json" | "protobuf"> | default = "json" ] # Producer acknowledgement level. "leader" matches the franz-go default # and minimizes Alertmanager's exposure to Kafka latency. "all" enables # the idempotent producer for at-least-once durability at the cost of # higher latency. [ acks: <"none" | "leader" | "all"> | default = "leader" ] # Compression codec for record batches. When omitted, batches are sent uncompressed. [ compression: <"none" | "gzip" | "snappy" | "lz4" | "zstd"> ] # Capacity of the local buffer between the event recorder and franz-go. # Events are dropped when this buffer is full. [ buffer_size: | default = 1024 ] # TLS configuration for the broker connection. When unset, the # connection uses PLAINTEXT. [ tls_config: ] ``` -------------------------------- ### Webhook Output Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure POSTing events as JSON bodies to an HTTP endpoint with retries and exponential backoff. ```yaml # URL to POST events to. url: # HTTP client configuration (TLS, basic auth, OAuth, proxies, ...). [ http_config: ] # HTTP request timeout. [ timeout: | default = 10s ] # Number of concurrent delivery workers. [ workers: | default = 4 ] # Maximum number of delivery attempts per event. [ max_retries: | default = 3 ] # Base backoff between retries; subsequent attempts use exponential # backoff (base * 2^attempt) capped at 30s. [ retry_backoff: | default = 500ms ] ``` -------------------------------- ### TLS Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure TLS settings for secure connections, including CA certificates, client certificates, and TLS version constraints. ```yaml # CA certificate to validate the server certificate with. [ ca_file: ] # Certificate and key files for client cert authentication to the server. [ cert_file: ] [ key_file: ] # ServerName extension to indicate the name of the server. # http://tools.ietf.org/html/rfc4366#section-3.1 [ server_name: ] # Disable validation of the server certificate. [ insecure_skip_verify: | default = false] # Minimum acceptable TLS version. Accepted values: TLS10 (TLS 1.0), TLS11 (TLS # 1.1), TLS12 (TLS 1.2), TLS13 (TLS 1.3). # If unset, Prometheus will use Go default minimum version, which is TLS 1.2. # See MinVersion in https://pkg.go.dev/crypto/tls#Config. [ min_version: ] # Maximum acceptable TLS version. Accepted values: TLS10 (TLS 1.0), TLS11 (TLS # 1.1), TLS12 (TLS 1.2), TLS13 (TLS 1.3). # If unset, Prometheus will use Go default maximum version, which is TLS 1.3. # See MaxVersion in https://pkg.go.dev/crypto/tls#Config. [ max_version: ] ``` -------------------------------- ### Slack Action Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Define interactive elements for Slack messages, such as buttons. Specify text, type, URL or name/value, and optional confirmation. ```yaml text: type: url: name: value: confirm: style: | default = '' ``` -------------------------------- ### OpsGenie Configuration Block Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure OpsGenie notification settings. This block defines parameters like API keys, message templates, and responder details. ```yaml # Whether to notify about resolved alerts. [ send_resolved: | default = true ] # The API key to use when talking to the OpsGenie API. [ api_key: | default = global.opsgenie_api_key ] # The filepath to API key to use when talking to the OpsGenie API. Conflicts with api_key. [ api_key_file: | default = global.opsgenie_api_key_file ] # The base URL for OpsGenie API requests. [ api_url: | default = global.opsgenie_api_url ] # Alert text limited to 130 characters. [ message: | default = '{{ template "opsgenie.default.message" . }}' ] # A description of the alert. [ description: | default = '{{ template "opsgenie.default.description" . }}' ] # A backlink to the sender of the notification. [ source: | default = '{{ template "opsgenie.default.source" . }}' ] # A set of arbitrary key/value pairs that provide further detail # about the alert. # All common labels are included as details by default. [ details: { : , ... } ] # List of responders responsible for notifications. responders: [ - ... ] # Comma separated list of tags attached to the notifications. [ tags: ] # Additional alert note. [ note: ] # Priority level of alert. Possible values are P1, P2, P3, P4, and P5. [ priority: ] # Whether to update message and description of the alert in OpsGenie if it already exists # By default, the alert is never updated in OpsGenie, the new message only appears in activity log. [ update_alerts: | default = false ] # Optional field that can be used to specify which domain alert is related to. [ entity: ] # Comma separated list of actions that will be available for the alert. [ actions: ] # The HTTP client's configuration. [ http_config: | default = global.http_config ] ``` -------------------------------- ### PagerDuty Configuration Options Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Configure PagerDuty notifications by specifying routing keys, API URLs, incident details, and other parameters. Use `routing_key` or `routing_key_file` for Events API v2, and `service_key` or `service_key_file` for the Prometheus integration type. The `details` field allows for custom key/value pairs, including alert summaries. ```yaml send_resolved: | default = true routing_key: routing_key_file: service_key: service_key_file: url: | default = global.pagerduty_url client: | default = '{{ template "pagerduty.default.client" . }}' client_url: | default = '{{ template "pagerduty.default.clientURL" . }}' description: | default = '{{ template "pagerduty.default.description" .}}' severity: | default = 'error' source: | default = client details: { : , ... } | default = { firing: '{{ .Alerts.Firing | toJSON }}' resolved: '{{ .Alerts.Resolved | toJSON }}' num_firing: '{{ .Alerts.Firing | len }}' num_resolved: '{{ .Alerts.Resolved | len }}' } images: [ ... ] links: [ ... ] component: group: class: http_config: | default = global.http_config timeout: | default = 0s ``` -------------------------------- ### Event Recorder Configuration Source: https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md Optional configuration for the event recorder, which captures and ships significant Alertmanager events. Requires the `event-recorder` feature flag to be enabled. ```yaml event_recorder: ```