### Prometheus Server Startup Output

Source: https://flashcat.cloud/docs/content/flashcat-partner/prometheus/quickstart/tutorial/install-prometheus-server

Example output indicating a successful Prometheus Server startup. This output shows the server is initializing the time-series database (TSDB), starting the web server, and loading its configuration.

```text
level=info ts=2018-10-23T14:55:14.499484Z caller=main.go:554 msg="Starting TSDB ..."
level=info ts=2018-10-23T14:55:14.499531Z caller=web.go:397 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-10-23T14:55:14.507999Z caller=main.go:564 msg="TSDB started"
level=info ts=2018-10-23T14:55:14.508068Z caller=main.go:624 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-10-23T14:55:14.509509Z caller=main.go:650 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2018-10-23T14:55:14.509537Z caller=main.go:523 msg="Server is ready to receive web requests.

```

--------------------------------

### Start Prometheus Server (Binary)

Source: https://flashcat.cloud/docs/content/flashcat-partner/prometheus/quickstart/tutorial/install-prometheus-server

Starts the Prometheus Server using the executable file. Prometheus will automatically load the configuration from 'prometheus.yml' in the current directory and begin scraping metrics.

```shell
./prometheus

```

--------------------------------

### Mtail Command Line Example

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale/usage/mtail

This example demonstrates how to start the mtail process, specifying the directory for program files and the log files to monitor. Mtail will automatically listen on port 3903 to expose metrics.

```bash
mtail --progs /etc/mtail --logs /var/log/syslog --logs /var/log/ntp/peerstats
```

--------------------------------

### Install Prometheus with Script

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/usage/data_source/open_source/prometheus

This script automates the installation of Prometheus by downloading a specified version, extracting it, and copying the files to the /opt/prometheus directory. It ensures Prometheus is set up for use.

```shell
# Install small script
version=2.45.3
filename=prometheus-${version}.linux-amd64
mkdir -p /opt/prometheus
wget https://github.com/prometheus/prometheus/releases/download/v${version}/${filename}.tar.gz
tar xf ${filename}.tar.gz
cp -far ${filename}/*  /opt/prometheus/
```

--------------------------------

### Start Categraf Service

Source: https://flashcat.cloud/docs/content/flashcat-monitor/categraf/2-installation

Starts the Categraf service after it has been installed. This can be done using the Categraf command or systemctl.

```bash
sudo ./categraf --start
sudo systemctl start categraf

```

--------------------------------

### Deploy Example Application for Monitoring

Source: https://flashcat.cloud/docs/content/flashcat-partner/prometheus/operator/use-operator-manage-prometheus

This YAML defines a sample Service and Deployment for an example application. The application is exposed via a Service on port 8080 and runs using the 'fabxc/instrumented_app' image. This setup is used to demonstrate monitoring configuration with ServiceMonitor.

```yaml
kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: web
    port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  selector:
    matchLabels:
      app: example-app
  replicas: 3
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: fabxc/instrumented_app
        ports:
        - name: web
          containerPort: 8080
```

--------------------------------

### Start and Enable Prometheus Service

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/usage/data_source/open_source/prometheus

These commands enable the Prometheus service to start automatically on boot, restart the service to apply configurations, and check its current status. This ensures Prometheus is running and managed by systemd.

```shell
# Start service
systemctl enable prometheus
systemctl restart prometheus
systemctl status prometheus
```

--------------------------------

### Example PromQL for MySQL Quick View Configuration

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/usage/metrics/metric-quick

This example demonstrates a PromQL query used to configure a Quick View for MySQL instances. It filters metrics by name, extracts the 'address' label as the instance list, and shows how to use 'ident' for dynamic filtering.

```PromQL
mysql_up{address="10.206.0.16:3306", ident="tt-fc-n9e.nj"}
```

--------------------------------

### Install Nightingale (Shell)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/install/intro

Shell script to download, extract, and install the Nightingale application. It includes steps to create a directory, download the release package using wget, extract the tarball, import the database schema, and start the Nightingale service using nohup.

```shell
# 创建个 n9e 的目录，后面把 n9e 相关的文件解压到这里
mkdir -p /opt/n9e && cd /opt/n9e

# 下载 n9e 发布包，amd64 是 x84 的包，下载站点也提供 arm64 的包，如果需要其他平台的包则要自行编译了
tarball=n9e-v6.0.1-linux-amd64.tar.gz
urlpath=https://download.flashcat.cloud/${tarball}
wget -q $urlpath || exit 1

# 解压缩发布包
tar zxvf ${tarball}

# 解压缩之后，可以看到 n9e.sql 是建表语句，导入数据库
mysql -uroot -p1234 < n9e.sql

# 启动 n9e，先使用 nohup 简单测试，如果需要 systemd 托管，请自行准备 service 文件
nohup ./n9e &> n9e.log &

# 检查 n9e.log 是否有异常日志，检查端口是否在监听，正常应该监听在 17000
ss -tlnp|grep 17000

```

--------------------------------

### Install Nightingale Server and WebAPI (Shell)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale/install/binary

This script installs Nightingale version 5.8.0 by downloading the tarball, extracting it, initializing the MySQL database with provided SQL scripts, and then starting the n9e-server and n9e-webapi components using nohup for background execution. It assumes MySQL and Redis are already installed.

```shell
mkdir -p /opt/n9e && cd /opt/n9e

# 去 https://github.com/didi/nightingale/releases 找最新版本的包，文档里的包地址可能已经不是最新的了
tarball=n9e-5.8.0.tar.gz
urlpath=https://github.com/didi/nightingale/releases/download/v5.8.0/${tarball}
wget $urlpath || exit 1

tar zxvf ${tarball}

mysql -uroot -p1234 < docker/initsql/a-n9e.sql

nohup ./n9e server &> server.log &
nohup ./n9e webapi &> webapi.log &

# check logs
# check port

```

--------------------------------

### Verify Data in Nightingale

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/install/intro

This example shows a query in Nightingale's instant query page to verify if data from Categraf is being successfully received and processed.

```sql
cpu_usage_active
```

--------------------------------

### Alibaba Cloud SLS Log Visualization Query Example

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/usage/logs/log-view

Shows an example of a query designed to visualize log metrics as a time-series chart in Alibaba Cloud SLS. It includes filtering conditions and an analysis statement using SQL syntax to aggregate data by URI and time, setting PV as the metric.

```text
not status:200 and http_user_agent:Go*  |  SELECT   count(*) AS PV, date_trunc('hour', __time__) AS timeHour,uri  group by timeHour,uri
```

--------------------------------

### VictoriaMetrics vminsert Instance Setup (Bash)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale/install/victoriametrics

Command to start a vminsert instance, which handles data ingestion. It requires the addresses of the vmstorage nodes to forward write requests.

```bash
nohup ./vminsert-prod -httpListenAddr :8480 -storageNode=127.0.0.1:8400,127.0.0.1:18400,127.0.0.1:28400 &>vminsert.log &

```

--------------------------------

### VictoriaMetrics vmselect Instance Setup (Bash)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale/install/victoriametrics

Command to start a vmselect instance, responsible for handling data queries. It needs the addresses of the vmstorage nodes to retrieve and merge query results.

```bash
nohup ./vmselect-prod -httpListenAddr :8481 -storageNode=127.0.0.1:8401,127.0.0.1:18401,127.0.0.1:28401 &>vmselect.log &

```

--------------------------------

### Flashcat Cloud Configuration Example

Source: https://flashcat.cloud/docs/content/flashcat-monitor/categraf/6-traces

This snippet shows a basic configuration for Flashcat Cloud. It includes enabling extensions like health_check and pprof, defining a trace pipeline with specific receivers, processors, and exporters, and setting up telemetry logging.

```yaml
extensions: [health_check, pprof]
pipelines:
  traces:
    receivers: [jaeger]
    processors: [batch/example, attributes/example]
    exporters: [jaeger, zipkin]
telemetry:
  logs:
    level: info
    initial_fields:
      service: my-instance
```

--------------------------------

### Basic Configuration Mapping (Authing Example)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/usage/system_configuration/sso/oauth2

Provides a direct mapping between common OIDC/OAuth2 configuration parameters and their corresponding settings in Authing.

```APIDOC
## Basic Configuration Mapping (Authing Example)

This section provides a mapping between the general OAuth2/OIDC configuration parameters and their specific counterparts when using Authing as the identity provider.

### Mapping Table:

| Nightingale Configuration | Authing OIDC Configuration |
| ------------------------ | -------------------------- |
| `RedirectURL`            | `http://n9e-server:port/callback/oauth` (Default) |
| `SsoAddr`                | Authorization Endpoint     |
| `SsoLogoutAddr`          | Logout Endpoint            |
| `TokenAddr`              | Token Endpoint             |
| `UserInfoAddr`           | UserInfo Endpoint          |
| `ClientId`               | App ID                     |
| `ClientSecret`           | App Secret                 |
```

--------------------------------

### VictoriaMetrics Single vmstorage Instance Setup (Bash)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale/install/victoriametrics

Commands to manually start individual vmstorage instances. Each command specifies unique ports and data directories for distinct storage nodes within the cluster.

```bash
nohup ./vmstorage-prod -loggerTimezone Asia/Shanghai -storageDataPath ./vmstorage-data -httpListenAddr :8482 -vminsertAddr :8400 -vmselectAddr :8401 &> vmstor.log &
nohup ./vmstorage-prod -loggerTimezone Asia/Shanghai -storageDataPath ./1vmstorage-data -httpListenAddr :18482 -vminsertAddr :18400 -vmselectAddr :18401 &> 1vmstor.log &
nohup ./vmstorage-prod -loggerTimezone Asia/Shanghai -storageDataPath ./2vmstorage-data -httpListenAddr :28482 -vminsertAddr :28400 -vmselectAddr :28401 &> 2vmstor.log &

```

--------------------------------

### OAuth2 Configuration Example

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/usage/system_configuration/sso/oauth2

This section provides a comprehensive example of the configuration parameters required for OAuth2 integration.

```APIDOC
## OAuth2 Configuration Example

This configuration block details the settings for enabling and customizing OAuth2 login.

### Configuration Parameters

- **Enable** (boolean) - `true`/`false` - Whether to enable OAuth2 login.
- **DisplayName** (string) - The display name for the OAuth2 login option on the frontend.
- **RedirectURL** (string) - The callback URL for the OAuth2 authentication, which should be the callback address of the Nightingale monitoring service.
- **SsoAddr** (string) - The address of the SSO service for authorization.
- **SsoLogoutAddr** (string) - The address for handling user logout requests from the SSO service.
- **TokenAddr** (string) - The address for the SSO service to obtain tokens.
- **UserInfoAddr** (string) - The address for the SSO service to fetch user information.
- **TranTokenMethod** (string) - The method for transmitting the token: `header`, `querystring`, or `formdata`.
- **ClientId** (string) - The App ID for the SSO service.
- **ClientSecret** (string) - The App Secret for the SSO service.
- **CoverAttributes** (boolean) - `true`/`false` - Whether to overwrite user attributes.
- **DefaultRoles** (array of strings) - Default roles to assign to users: `Guest`, `Standard`, `Admin`.
- **UserinfoIsArray** (boolean) - `true`/`false` - Indicates if the user information returned is an array.
- **UserinfoPrefix** (string) - A prefix for user information fields if the returned JSON has a nested structure.
- **Scopes** (array of strings) - The OAuth2 scopes requested for authentication.

### User Attribute Mapping

This section defines how user attributes from the SSO service map to user attributes within the platform.

#### Attributes

- **Username** (string) - Maps to the username field in the platform.
- **Nickname** (string) - Maps to the nickname field in the platform.
- **Phone** (string) - Maps to the phone number field in the platform.
- **Email** (string) - Maps to the email field in the platform.

### Example Configuration

```ini
# Whether to enable OAuth2 login functionality
Enable = true
# The display name for OAuth2 login on the frontend
DisplayName = 'OAuth2 Login'
# The redirect URL after successful OAuth2 authentication. This URL should be the callback address of the Nightingale monitoring service.
RedirectURL = 'http://n9e-server:port/callback/oauth'
# Address of the SSO service
SsoAddr = 'https://<authing-tenant-name>.authing.cn/oauth2/authorize'
# SSO logout address, used to handle user logout requests
SsoLogoutAddr = 'https://<authing-tenant-name>.authing.cn/logout'
# SSO service token acquisition address
TokenAddr = 'https://<authing-tenant-name>.authing.cn/oauth2/token'
# SSO service user information acquisition address
UserInfoAddr = 'https://<authing-tenant-name>.authing.cn/userinfo'
# Method for transmitting the token, options: header | querystring | formdata
TranTokenMethod = 'header'
# SSO service App ID
ClientId = '66***********'
# SSO service App Secret
ClientSecret = 'ccc***********'
# Whether to overwrite user attributes
CoverAttributes = true
# Default roles for users in the Nightingale system: Guest/Standard/Admin
DefaultRoles = ['Standard']
# Whether user information is an array
UserinfoIsArray = false
# Prefix for user information
UserinfoPrefix = 'data'
# Requested OAuth2 scopes
Scopes = ['profile', 'email', 'phone']

# Mapping of user attributes from the SSO service to platform user attributes
[Attributes]
# Mapping for the username field
Username = 'sub'
# Mapping for the nickname field
Nickname = 'nickname'
# Mapping for the phone field
Phone = 'phone_number'
# Mapping for the email field
Email = 'email'
```
```

--------------------------------

### Start Categraf with Custom Service Name

Source: https://flashcat.cloud/docs/content/flashcat-monitor/categraf/2-installation

Starts a Categraf service that was installed with a custom service name. This command ensures the correct service instance is started.

```bash
# For general monitoring
cd /opt/categraf 
./categraf --start # Equivalent to ./categraf --service-name categraf --start

# For Aliyun metrics
cd /opt/categraf.aliyun 
./categraf --service-name categraf.aliyun --start

```

--------------------------------

### Configure VictoriaMetrics Data Source (TOML)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/install/intro

Example TOML configuration snippet for setting up a VictoriaMetrics data source within Flashduty. It specifies the data source name, URL (including the select path for queries), timeout, and the remote write address for data ingestion.

```toml
# Example configuration for a VictoriaMetrics data source
# name = "victoriametrics"
# url = "http://127.0.0.1:8481/select/0/prometheus"
# timeout = 10000
# write_addr = "http://127.0.0.1:8480/insert/0/prometheus/api/v1/write"

```

--------------------------------

### Log Analysis Query Examples (Alibaba Cloud SLS)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/usage/log_analysis/business/tx_cls_hk

Demonstrates how to query and analyze logs using Alibaba Cloud SLS. It covers basic filtering, aggregation, and grouping to extract meaningful insights from log data. These examples are useful for understanding log data patterns and troubleshooting.

```text
status, count(*) as pv GROUP BY status
```

```text
method:GET AND status>400
```

```text
level:(ERROR OR WARNING)
```

```text
NOT level:INFO
```

```text
not remote_user:*
```

```text
remote_user:*
```

```text
not host:123.*
```

--------------------------------

### Configuration Steps

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/usage/system_configuration/notification_set/business/tc-sms

Step-by-step guide to configure Tencent Cloud SMS integration.

```APIDOC
## Configuration Steps

Follow these steps to configure Tencent Cloud SMS integration:

### 1. Create SecretId & SecretKey
Navigate to the API Key Management section in the Tencent Cloud console and create a new key. **Important:** The SecretKey is only shown during creation; please save it securely.

### 2. Create SDK AppID
In the Tencent Cloud console, go to Application Management -> Application List and create a new application. Your SDKAppID will be displayed after creation.

### 3. Create Message Template & Get Template ID
In the Tencent Cloud console, go to Application Management -> Message Template Management and create a new message template. The placeholder `{1}` in the template can be used to include dynamic notification content from FlashCat.

### 4. Configure Tencent Cloud SMS in FlashCat
In FlashCat, navigate to the notification settings and fill in the Tencent Cloud SMS configuration details using the fields described above.

### 5. Configure Notification Templates
Refer to the documentation at https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/faq/go-template/ for guidance on writing notification templates.

### 6. Enable Tencent Cloud SMS Notifications
1. In the notification settings, select `tx-voice` as the notification medium.
2. Choose an alert recipient group. Ensure that members within the group have their correct mobile phone numbers configured to receive SMS notifications.

**User Mobile Number Configuration:**
Configure user mobile numbers in the **Personnel Organization** -> **User Management** section.
```

--------------------------------

### Configure Prometheus Data Source (TOML)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/install/intro

Example TOML configuration snippet for setting up a Prometheus data source within Flashduty. It includes parameters for the data source name, URL, timeout, authentication, SSL verification, custom headers, and remote write address.

```toml
# Example configuration for a Prometheus data source
# name = "prometheus"
# url = "http://your-prometheus-server:9090"
# timeout = 10000
# auth = ""
# skip_ssl_verify = false
# headers = {}
# write_addr = "http://your-prometheus-server:9090/api/v1/write"

```

--------------------------------

### Deploy Jaeger with Docker Compose

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/usage/datasource/jaeger_en

This code snippet demonstrates deploying Jaeger and a demo application (HotROD) using Docker Compose. It requires `git` and `docker-compose` to be installed. The command clones the Jaeger repository, navigates to the example directory, and starts the services. This method is for demonstration and not recommended for production.

```bash
#Please ensure that `git` and `docker-compose` services are installed in advance, and test the network communication for pulling images.

git clone git@github.com:jaegertracing/jaeger.git jaeger && cd jaeger/examples/hotrod && docker compose up

```

--------------------------------

### Configure Event and Callback Settings (HTTP Mode)

Source: https://flashcat.cloud/docs/content/flashcat/firemap/im_lark

Instructions for setting up HTTP mode for event subscriptions and callbacks, including obtaining an IP/domain and generating encryption keys.

```APIDOC
## POST /api/applications/{app_id}/event-subscriptions
## POST /api/applications/{app_id}/callbacks

### Description
Configure event and callback settings using HTTP mode. This involves obtaining a public IP or domain, generating an Encrypt Key and Verification Token, and configuring the callback URLs.

### Method
POST

### Endpoint
- **Event Subscription (HTTP Mode):** `http(s)://<IP_OR_DOMAIN>/welcome`
- **Callback Configuration (HTTP Mode):** `http(s)://<IP_OR_DOMAIN>/act`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
**For Event Subscription:**
- `callback_url` (string) - Required - Your public IP or domain followed by `/welcome`.

**For Callback Configuration:**
- `callback_url` (string) - Required - Your public IP or domain followed by `/act`.

**For Encryption Keys:**
- Interaction via the Feishu management backend UI to generate `Encrypt Key` and `Verification Token`.

**Required Information for Flashcat:**
- App ID
- App Secret
- Encrypt Key
- Verification Token

### Request Example
**Event Subscription Request Body:**
```json
{
  "callback_url": "https://yourdomain.com/welcome"
}
```

**Callback Configuration Request Body:**
```json
{
  "callback_url": "https://yourdomain.com/act"
}
```

### Response
#### Success Response (200)
Confirmation of event and callback URL configuration within the Feishu management backend.

#### Response Example
None
```

--------------------------------

### Install MySQL and Redis (Shell)

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/install/intro

Shell script to install and configure MySQL and Redis on a Linux system. Sets a default MySQL root password and enables/restarts both services. Assumes a YUM-based distribution.

```shell
# install mysql
yum -y install mariadb*
systemctl enable mariadb
systemctl restart mariadb
mysql -e "SET PASSWORD FOR 'root'@'localhost' = PASSWORD('1234');"

# install redis
yum install -y redis
systemctl enable redis
systemctl restart redis

```

--------------------------------

### Prometheus Configuration Example (prometheus.yml)

Source: https://flashcat.cloud/docs/content/flashcat-partner/prometheus/quickstart/tutorial/install-prometheus-server

A sample Prometheus configuration file (prometheus.yml) defining global settings, alerting rules, and scrape configurations. This file is essential for Prometheus to operate correctly.

```yaml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

```

--------------------------------

### Upgrade to Nightingale Professional Trial - Start Process

Source: https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/upgrade

This command stops the current Nightingale process and starts the professional version (n9e-plus) in the background, logging output to n9e.log.

```bash
pkill n9e && nohup ./n9e-plus &> n9e.log &
```