### Start Single Application

Source: https://github.com/didi/knowstreaming/blob/master/km-console/README.md

To start a single application, navigate to its specific directory within `packages` and run `npm run start`. For example, to start the layout-clusters-fe application, use `cd packages/layout-clusters-fe` followed by `npm run start`.

```bash
cd packages/layout-clusters-fe
npm run start
```

--------------------------------

### Install MySQL Server (YUM)

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Installs MySQL server and client using YUM, starts the service, and retrieves the temporary password for modification.

```bash
# 配置yum源
wget https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm
rpm -ivh mysql57-community-release-el7-9.noarch.rpm

# 执行安装
yum -y install mysql-server mysql-client

# 服务启动
systemctl start mysqld

# 获取初始密码并修改
old_pass=`grep 'temporary password' /var/log/mysqld.log | awk '{print $NF}' | tail -n 1`

mysql -NBe "alter user USER() identified by 'Didi_km_678';" --connect-expired-password -uroot -p$old_pass
```

--------------------------------

### Install MySQL Server (RPM)

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Installs MySQL server using RPM packages, extracts them to a temporary directory, starts the service, and retrieves the temporary password for modification.

```bash
# 下载安装包
wget https://s3-gzpu.didistatic.com/knowsearch/mysql5.7.tar.gz

# 解压到指定目录
tar -zxf mysql5.7.tar.gz -C /tmp/

# 执行安装
yum -y localinstall /tmp/libaio-*.rpm /tmp/mysql-*.rpm

# 服务启动
systemctl start mysqld


# 获取初始密码并修改
old_pass=`grep 'temporary password' /var/log/mysqld.log | awk '{print $NF}' | tail -n 1`

mysql -NBe "alter user USER() identified by 'Didi_km_678';" --connect-expired-password -uroot -p$old_pass
```

--------------------------------

### Start Application

Source: https://github.com/didi/knowstreaming/blob/master/km-console/packages/config-manager-fe/README.md

Execute this command to start the application. As a sub-application, it must be viewed within the base application (layout-clusters-fe) at http://localhost:8000. Ensure the base application is running.

```bash
npm run start
```

--------------------------------

### Docker Compose Installation and Verification

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Deploy KnowStreaming using Docker Compose. Ensure Docker and Docker Compose are installed. The provided commands start the services and verify their status.

```shell
# `v3.0.0-beta.2`版本开始(docker镜像为0.2.0版本开始)，默认账号密码为`admin` / `admin`；
# https://hub.docker.com/u/knowstreaming 在此处寻找最新镜像版本
# mysql与es可以使用自己搭建的服务,调整对应配置即可

# 复制docker-compose.yml到指定位置后执行下方命令即可启动
docker-compose up -d

# 验证安装
docker-compose ps
# 验证启动 - 状态为 UP 则表示成功 
        Name                       Command                       State                  Ports
----------------------------------------------------------------------------------------------------
elasticsearch-single    /usr/local/bin/docker-entr ...   Up                      9200/tcp, 9300/tcp
knowstreaming-init      /bin/bash /es_template_cre ...   Up
knowstreaming-manager   /bin/sh /ks-start.sh             Up                      80/tcp
knowstreaming-mysql     /entrypoint.sh mysqld            Up (health: starting)   3306/tcp, 33060/tcp
knowstreaming-ui        /docker-entrypoint.sh ngin ...   Up                      0.0.0.0:80->80/tcp

# 稍等一分钟左右 knowstreaming-init 会退出，表示es初始化完成，可以访问页面
        Name                       Command                  State              Ports
-------------------------------------------------------------------------------------------
knowstreaming-init      /bin/bash /es_template_cre ...   Exit 0
knowstreaming-mysql     /entrypoint.sh mysqld            Up (healthy)   3306/tcp, 33060/tcp
```

--------------------------------

### Install Dependencies

Source: https://github.com/didi/knowstreaming/blob/master/km-console/packages/config-manager-fe/README.md

Run this command to install project dependencies. If dependencies are already installed via 'npm run i' in the km-console directory, this step can be skipped. This method only installs dependencies for the current application.

```bash
npm install
```

--------------------------------

### Download and Run Offline Installation Script

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

This method is for offline installation. Download the archive, transfer it to the server, and extract before running the script.

```bash
# 将安装包下载到本地且传输到目标服务器
wget https://s3-gzpu.didistatic.com/pub/knowstreaming/KnowStreaming-3.0.0-beta.1-offline.tar.gz

# 解压安装包
tar -zxf KnowStreaming-3.0.0-beta.1-offline.tar.gz

# 执行安装脚本
sh deploy_KnowStreaming-offline.sh

# 访问地址
127.0.0.1:8080
```

--------------------------------

### Set up Elasticsearch Instance

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Downloads Elasticsearch, creates a data directory and user, configures user limits and system settings, extracts the archive, sets ownership, modifies configuration files (elasticsearch.yml and jvm.options), and starts the service.

```bash
# 下载安装包
wget https://s3-gzpu.didistatic.com/pub/elasticsearch.tar.gz

# 创建ES数据存储目录
mkdir -p /data/es_data

# 创建ES所属用户
useradd arius

# 配置用户的打开文件数
echo "arius soft nofile 655350" >> /etc/security/limits.conf
echo "arius hard nofile 655350" >> /etc/security/limits.conf
echo "vm.max_map_count = 655360" >> /etc/sysctl.conf
sysctl -p

# 解压安装包
tar -zxf elasticsearch.tar.gz -C /data/

# 更改目录所属组
chown -R arius:arius /data/

# 修改配置文件(参考以下配置)
vim /data/elasticsearch/config/elasticsearch.yml
    cluster.name: km_es
    node.name: es-node1
    node.master: true
    node.data: true
    path.data: /data/es_data
    http.port: 8060
    discovery.seed_hosts: ["127.0.0.1:9300"]

# 修改内存配置
vim /data/elasticsearch/config/jvm.options
    -Xms2g
    -Xmx2g

# 启动服务
su - arius
export JAVA_HOME=/usr/local/java11
sh /data/elasticsearch/control.sh start

# 确认状态
sh /data/elasticsearch/control.sh status
```

--------------------------------

### Set up KnowStreaming Instance

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Downloads and extracts KnowStreaming, creates necessary databases and imports initial data for Know Streaming, Logi-Job, and Logi-Security, initializes Elasticsearch templates, modifies configuration files, and starts the service.

```bash
# 下载安装包
wget https://s3-gzpu.didistatic.com/pub/knowstreaming/KnowStreaming-3.0.0-beta.1.tar.gz

# 解压安装包到指定目录
tar -zxf KnowStreaming-3.0.0-beta.1.tar.gz -C /data/

# 修改启动脚本并加入systemd管理
cd  /data/KnowStreaming/

# 创建相应的库和导入初始化数据
mysql -uroot -pDidi_km_678 -e "create database know_streaming;"
mysql -uroot -pDidi_km_678  know_streaming < ./init/sql/ddl-ks-km.sql
mysql -uroot -pDidi_km_678  know_streaming < ./init/sql/ddl-logi-job.sql
mysql -uroot -pDidi_km_678  know_streaming < ./init/sql/ddl-logi-security.sql
mysql -uroot -pDidi_km_678  know_streaming < ./init/sql/dml-ks-km.sql
mysql -uroot -pDidi_km_678  know_streaming < ./init/sql/dml-logi.sql

# 创建elasticsearch初始化数据
sh ./bin/init_es_template.sh

# 修改配置文件
vim ./conf/application.yml

# 监听端口
server:
    port: 8080 # web 服务端口
    tomcat:
        accept-count: 1000
        max-connections: 10000

# ES地址
es.client.address: 127.0.0.1:8060

# 数据库配置（一共三处地方，修改正确的mysql地址和数据库名称以及用户名密码）
jdbc-url: jdbc:mariadb://127.0.0.1:3306/know_streaming?.....
username: root
password: Didi_km_678

# 启动服务
cd /data/KnowStreaming/bin/
sh startup.sh
```

--------------------------------

### Commit Log Example with Body

Source: https://github.com/didi/knowstreaming/blob/master/docs/contribute_guide/贡献指南.md

Demonstrates a commit log with both Header and Body, detailing optimizations and affected changes.

```git
[Optimize]优化 MySQL & ES 测试容器的初始化

主要的变更
1、knowstreaming/knowstreaming-manager 容器；
2、knowstreaming/knowstreaming-mysql 容器调整为使用 mysql:5.7 容器；
3、初始化 mysql:5.7 容器后，增加初始化 MySQL 表及数据的动作；

被影响的变更：
1、移动 km-dist/init/sql 下的MySQL初始化脚本至 km-persistence/src/main/resource/sql 下，以便项目测试时加载到所需的初始化 SQL；
2、删除无用的 km-dist/init/template 目录；
3、因为 km-dist/init/sql 和 km-dist/init/template 目录的调整，因此也调整 ReleaseKnowStreaming.xml 内的文件内容；

```

--------------------------------

### Build Project with Maven

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/本地源码启动手册.md

Execute this command to package both the backend and frontend code into a final installation package. This command is essential for preparing the project for deployment or local startup.

```bash
mvn install
```

--------------------------------

### Helm Install Command for KnowStreaming

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Installs KnowStreaming using Helm. Ensure Kubernetes and Helm are set up. Adjust NAMESPACE and NAME as needed. Initialization may take a few minutes.

```bash
# 相关镜像在Docker Hub都可以下载
# 快速安装(NAMESPACE需要更改为已存在的，安装启动需要几分钟初始化请稍等~)
helm install -n [NAMESPACE] [NAME] http://download.knowstreaming.com/charts/knowstreaming-manager-0.1.5.tgz

# 获取KnowStreaming前端ui的service. 默认nodeport方式.
# (http://nodeIP:nodeport，默认用户名密码：admin/admin2022_)
# `v3.0.0-beta.2`版本开始（helm chart包版本0.1.4开始），默认账号密码为`admin` / `admin`；

# 添加仓库
helm repo add knowstreaming http://download.knowstreaming.com/charts

# 拉取最新版本
helm pull knowstreaming/knowstreaming-manager
```

--------------------------------

### Download and Run Online Installation Script

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Use this script for an automated online installation. It will reinstall MySQL and store the password in mysql.password.

```bash
# 在服务器中下载安装脚本, 该脚本中会在当前目录下，重新安装MySQL。重装后的mysql密码存放在当前目录的mysql.password文件中。
wget https://s3-gzpu.didistatic.com/pub/knowstreaming/deploy_KnowStreaming-3.0.0-beta.1.sh

# 执行脚本
sh deploy_KnowStreaming.sh

# 访问地址
127.0.0.1:8080
```

--------------------------------

### LDAP Login Service Implementation Example

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/登录系统对接.md

An example implementation of the LoginExtend interface for LDAP authentication. It handles user verification against an LDAP server and manages login/logout states.

```Java
@Service("ksLdapLoginService")
public class LdapLoginServiceImpl implements LoginExtend  {
  

    @Override
    public UserBriefVO verifyLogin(AccountLoginDTO loginDTO,
                                   HttpServletRequest request,
                                   HttpServletResponse response) throws LogiSecurityException {
        String decodePasswd = AESUtils.decrypt(loginDTO.getPw());

        // 去LDAP验证账密
        LdapPrincipal ldapAttrsInfo = ldapAuthentication.authenticate(loginDTO.getUserName(), decodePasswd);
        if (ldapAttrsInfo == null) {
            // 用户不存在，正常来说上如果有问题，上一步会直接抛出异常
            throw new LogiSecurityException(ResultCode.USER_NOT_EXISTS);
        }

        // 进行业务相关操作
        
        // 记录登录状态，Ldap因为无法记录登录状态，因此有KnowStreaming进行记录
        initLoginContext(request, response, loginDTO.getUserName(), user.getId());
        return CopyBeanUtil.copy(user, UserBriefVO.class);
    }

    @Override
    public Result<Boolean> logout(HttpServletRequest request, HttpServletResponse response) {
        
        //清理cookie和session
      
        return Result.buildSucc(Boolean.TRUE);
    }

    @Override
    public boolean interceptorCheck(HttpServletRequest request, HttpServletResponse response, String requestMappingValue, List<String> whiteMappingValues) throws IOException {
       
        // 检查是否已经登录
        String userName = HttpRequestUtil.getOperator(request);
        if (StringUtils.isEmpty(userName)) {
            // 未登录，则进行登出
            logout(request, response);
            return Boolean.FALSE;
        }

        return Boolean.TRUE;
    }
}
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/didi/knowstreaming/blob/master/km-console/README.md

Installs all necessary project dependencies. The project uses `package-lock.json` to prevent automatic dependency upgrades and defaults to the taobao npm mirror.

```bash
npm run i
```

--------------------------------

### Install Lerna Globally

Source: https://github.com/didi/knowstreaming/blob/master/km-console/README.md

Installs Lerna globally, which can be used to manage the project's monorepo. This step is optional if you are not familiar with Lerna.

```bash
npm install -g lerna
```

--------------------------------

### Configure Jasypt Salt via Command Line

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/MYSQL密码加密手册.md

Provide the Jasypt encryption salt as a command-line argument when starting your application.

```bash
java -jar xxx.jar --jasypt.encryptor.password=salt
```

--------------------------------

### Commit Log Header Example

Source: https://github.com/didi/knowstreaming/blob/master/docs/contribute_guide/贡献指南.md

Illustrates the format for the Header section of a commit log, including Type and Message.

```git
[Bugfix]修复新接入的集群，Controller-Host不显示的问题
```

--------------------------------

### Kafka Broker Zookeeper Configuration Example

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/解决连接JMX失败.md

This JSON snippet illustrates a Kafka broker's configuration as stored in ZooKeeper, showing listener security protocols, endpoints, and JMX port. It highlights potential issues where KnowStreaming might connect to the wrong IP (e.g., EXTERNAL instead of INTERNAL).

```json
{
    "listener_security_protocol_map": {
        "EXTERNAL": "SASL_PLAINTEXT",
        "INTERNAL": "SASL_PLAINTEXT"
    },
    "endpoints": [
        "EXTERNAL://192.168.0.1:7092",
        "INTERNAL://192.168.0.2:7093"
    ],
    "jmx_port": 8099,
    "host": "192.168.0.1",
    "timestamp": "1627289710439",
    "port": -1,
    "version": 4
}
```

--------------------------------

### Build Application

Source: https://github.com/didi/knowstreaming/blob/master/km-console/packages/config-manager-fe/README.md

Use this command to build the application for production. The output will be located in the 'km-rest/src/main/resources/templates/config' directory at the project root.

```bash
npm run build
```

--------------------------------

### Initialize Environment Git Commands

Source: https://github.com/didi/knowstreaming/blob/master/docs/contribute_guide/贡献指南.md

Commands to set up the local development environment by forking the main repository, cloning, and adding remote upstream.

```git
git clone git@github.com:xxxxxxx/KnowStreaming.git
```

```git
git remote add upstream https://github.com/didi/KnowStreaming
```

```git
git fetch upstream
```

```git
git fetch origin
```

```git
git checkout -b upstream/master
```

--------------------------------

### Enable JMX in Kafka (kafka-server-start.sh)

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/解决连接JMX失败.md

Modify the kafka-server-start.sh script to define the JMX port. This is a prerequisite for enabling JMX.

```bash
# 在这个下面增加JMX端口的配置
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT=9999  # 增加这个配置, 这里的数值并不一定是要9999 
fi
```

--------------------------------

### Maven Command for Full Project Package

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/源码编译打包手册.md

Execute this command in the KS-KM project directory to perform a full build and generate a distributable package.

```bash
mvn -Prelease-package -Dmaven.test.skip=true clean install -U
```

--------------------------------

### SQL Example to Update JMX Properties

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/解决连接JMX失败.md

This SQL statement demonstrates how to update the 'jmx_properties' field in the 'ks_km_physical_cluster' table to control JMX connection behavior, such as specifying the endpoint to use.

```sql
UPDATE ks_km_physical_cluster SET jmx_properties='{ "maxConn": 10, "username": "xxxxx", "password": "xxxx", "openSSL": false , "useWhichEndpoint": "xxx"}' where id={xxx};
```

--------------------------------

### Kafka Kerberos Authentication Configuration

Source: https://github.com/didi/knowstreaming/blob/master/docs/user_guide/faq.md

Configure KnowStreaming to connect to a Kafka cluster with Kerberos authentication enabled. Ensure the KRB client is installed, krb5.conf is updated, and the correct keytab is provided.

```json
{
  "security.protocol": "SASL_PLAINTEXT",
  "sasl.mechanism": "GSSAPI",
  "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab=\"/etc/keytab/kafka.keytab\" storeKey=true useTicketCache=false principal=\"kafka/kafka@TEST.COM\";",
  "sasl.kerberos.service.name": "kafka"
}
```

--------------------------------

### Kafka JMX Connection Errors (Incorrect IP)

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/解决连接JMX失败.md

These log examples show 'Connection refused' errors when JMX cannot be reached at the specified host. This can occur if the 'java.rmi.server.hostname' is incorrectly configured or points to an inaccessible IP.

```log
# 错误一： 错误提示的是真实的IP，这样的话基本就是JMX配置的有问题了。 
2021-01-27 10:06:20.730 ERROR 50901 --- [ics-Thread-1-62] c.x.k.m.c.utils.jmx.JmxConnectorWrap     : JMX connect exception, host:192.168.0.1 port:9999. java.rmi.ConnectException: Connection refused to host: 192.168.0.1; nested exception is:    

# 错误二：错误提示的是127.0.0.1这个IP，这个是机器的hostname配置的可能有问题。 
2021-01-27 10:06:20.730 ERROR 50901 --- [ics-Thread-1-62] c.x.k.m.c.utils.jmx.JmxConnectorWrap     : JMX connect exception, host:127.0.0.1 port:9999. java.rmi.ConnectException: Connection refused to host: 127.0.0.1;; nested exception is: 
```

--------------------------------

### Initialize ES Index Template for v3.0.1

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/版本升级手册.md

Execute the `bin/init_es_template.sh` script to create the `ks_kafka_zookeeper_metric` index template when upgrading to v3.0.1. This ensures proper indexing for ZooKeeper metrics in Elasticsearch.

```bash
# 新增 ks_kafka_zookeeper_metric 索引模版。
# 可通过再次执行 bin/init_es_template.sh 脚本，创建该索引模版。

```

--------------------------------

### Append Java Options for Kerberos Debugging

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/接入ZK带认证Kafka集群.md

Add these Java options to the JAVA_OPT variable in KnowStreaming's startup script to enable Kerberos debugging and specify configuration files. This helps in troubleshooting authentication issues.

```bash
-Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/zookeeper.jaas
```

--------------------------------

### Implement Balanced Broadcast Task Execution

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/Task模块简介.md

Extend AbstractClusterPhyDispatchTask and implement processSubTask for balanced execution. Annotate with @Task and set consensual to ConsensualEnum.BROADCAST. This example distributes Kafka cluster tasks evenly across hosts, with each host processing its assigned subset every minute.

```java
    // 1、继承AbstractClusterPhyDispatchTask，实现processSubTask方法； 
    // 2、在类上添加@Task注解，并且配置好信息，指定为广播模式； 
    // 效果：在本样例中，每隔1分钟ks会将所有的kafka集群列表在ks集群主机内均衡拆分，每台主机会将分发到自身的Kafka集群依次执行processSubTask方法,实现KS集群的任务协同处理。 
    @Task(name = "kmJobTask",
          description = "km job 模块调度执行任务",
          cron = "0 0/1 * * * ? *",
          autoRegister = true,
          consensual = ConsensualEnum.BROADCAST,
          timeout = 6 * 60) 
    public class KMJobTask extends AbstractClusterPhyDispatchTask {   
      
        @Autowired    
        private JobService jobService;     
      
        @Override    
        protected TaskResult processSubTask(ClusterPhy clusterPhy, long triggerTimeUnitMs) throws Exception { 
              jobService.scheduleJobByClusterId(clusterPhy.getId());        
              return TaskResult.SUCCESS;    
        }
    }
```

--------------------------------

### Troubleshooting NoClassDefFoundError on Startup

Source: https://github.com/didi/knowstreaming/blob/master/docs/user_guide/faq.md

This log snippet indicates a startup failure due to `NoClassDefFoundError: Could not initialize class com.didiglobal.logi.job.core.WorkerSingleton$Singleton`. This often occurs when Logi-Job fails to initialize due to OS information retrieval issues.

```log
# 启动失败，报nested exception is java.lang.NoClassDefFoundError: Could not initialize class com.didiglobal.logi.job.core.WorkerSingleton$Singleton


2023-08-11 22:54:29.842 [main] ERROR class=org.springframework.boot.SpringApplication||Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'quartzScheduler' defined in class path resource [com/didiglobal/logi/job/LogIJobAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.didiglobal.logi.job.core.Scheduler]: Factory method 'quartzScheduler' threw exception; nested exception is java.lang.NoClassDefFoundError: Could not initialize class com.didiglobal.logi.job.core.WorkerSingleton$Singleton
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657)
```

--------------------------------

### Configure JDK Environment

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/单机部署手册.md

Downloads, extracts, and configures JDK 11 by setting environment variables for JAVA_HOME, CLASSPATH, and PATH, then sources the bashrc file.

```bash
# 下载安装包
wget https://s3-gzpu.didistatic.com/pub/jdk11.tar.gz

# 解压到指定目录
tar -zxf jdk11.tar.gz -C /usr/local/

# 更改目录名
mv /usr/local/jdk-11.0.2 /usr/local/java11

# 添加到环境变量
echo "export JAVA_HOME=/usr/local/java11" >> ~/.bashrc
echo "export CLASSPATH=/usr/java/java11/lib" >> ~/.bashrc
echo "export PATH=$JAVA_HOME/bin:$PATH:$HOME/bin" >> ~/.bashrc

source ~/.bashrc
```

--------------------------------

### Configure Task Startup Failure Percentage

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/版本升级手册.md

Defines the acceptable probability of task startup failures within the Connect cluster. Exceeding this threshold may indicate issues with the Connect cluster's stability.

```sql
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_CONNECT_CLUSTER_TASK_STARTUP_FAILURE_PERCENTAGE', '{"value" : 0.05}', 'Connect集群任务启动失败概率', 'admin');
```

--------------------------------

### Configure Jasypt Salt via Environment Variable

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/MYSQL密码加密手册.md

Set the Jasypt encryption salt using an environment variable and pass it to the application startup command.

```bash
export JASYPT_PASSWORD=salt
java -jar xxx.jar --jasypt.encryptor.password=${JASYPT_PASSWORD}
```

--------------------------------

### Initialize Docker Swarm

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/MYSQL密码加密手册.md

Initialize a Docker Swarm cluster. This command is the first step in setting up Docker Swarm for secret management.

```bash
docker swarm init
```

--------------------------------

### Migrate Topic Description Information (SQL)

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/版本升级手册.md

This SQL statement migrates topic description information from a 2.x database to the 3.0.0 version. It should be executed after deploying 3.0.0. Consider running this SQL periodically to migrate incremental data.

```sql
UPDATE ks_km_topic
    INNER JOIN
    (SELECT
    topic.cluster_id AS cluster_id,
    topic.topic_name AS topic_name,
    topic.description AS description
    FROM topic WHERE description != ''
    ) AS t

ON ks_km_topic.cluster_phy_id = t.cluster_id
    AND ks_km_topic.topic_name = t.topic_name
    AND ks_km_topic.id > 0
    SET ks_km_topic.description = t.description;
```

--------------------------------

### Create a Password File for Docker Secret

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/MYSQL密码加密手册.md

Create a plain text file containing the password to be used as a Docker secret.

```bash
echo "admin2022_" > password
```

--------------------------------

### Configure JMX Options in Kafka (kafka-run-class.sh)

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/解决连接JMX失败.md

Update the kafka-run-class.sh script to configure JMX remote access, including authentication, SSL, and the server hostname. Ensure 'java.rmi.server.hostname' is set to the correct IP address of the machine.

```bash
# JMX settings 
if [ -z "$KAFKA_JMX_OPTS" ]; then
    KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=当前机器的IP" 
fi  

# JMX port to use 
if [  $JMX_PORT ]; then
    KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT" 
fi
```

--------------------------------

### Configure Logi-Job and Thread/Client Pools

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/版本升级手册.md

This configuration snippet shows new settings for enabling logi-job tasks and adjusting thread and client pool sizes. Ensure these settings align with your deployment strategy and performance requirements.

```yaml
spring:
  logi-job:  # know-streaming 依赖的 logi-job 模块的数据库的配置，默认与 know-streaming 的数据库配置保持一致即可
    enable: true                # true表示开启job任务, false表关闭。KS在部署上可以考虑部署两套服务，一套处理前端请求，一套执行job任务，此时可以通过该字段进行控制

# 线程池大小相关配置
thread-pool:
  es:
    search:                         # es查询线程池
      thread-num: 20                # 线程池大小
      queue-size: 10000             # 队列大小

# 客户端池大小相关配置
client-pool:
  kafka-admin:
    client-cnt: 1                 # 每个Kafka集群创建的KafkaAdminClient数

```

--------------------------------

### 关联新增权限到角色

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/版本升级手册.md

将新添加的安全权限（permission_id: 2052）关联到指定的角色（role_id: 1677）。

```sql
INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2052', '0', 'know-streaming');
```

--------------------------------

### 新增角色权限配置

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/版本升级手册.md

用于向logi_security_role_permission表中插入新的角色权限关联记录。请确保role_id和permission_id与现有或新增的权限匹配。

```sql
INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2050', '0', 'know-streaming');
```

--------------------------------

### Locate Main Application Class

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/本地源码启动手册.md

The main entry point for the Know Streaming application is located in this Java file. This information is useful for configuring your IDE to run the application directly from source.

```java
km-rest/src/main/java/com/xiaojukeji/know/streaming/km/rest/KnowStreaming.java
```

--------------------------------

### Login Controller Implementation

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/登录系统对接.md

Handles incoming login requests via a POST endpoint. It delegates the authentication process to the loginService and returns the user information or an error.

```java
    @RestController
    public class LoginController {
      

        //登陆接口
        @PostMapping({"/login"})
        public Result<UserBriefVO> login(HttpServletRequest request, HttpServletResponse response, @RequestBody AccountLoginDTO loginDTO) {
            try {
                 //登陆认证
                UserBriefVO userBriefVO = this.loginService.verifyLogin(loginDTO, request, response);
                return Result.success(userBriefVO);
              
            } catch (LogiSecurityException var5) {
                return Result.fail(var5);
            }
        }

    }
```

--------------------------------

### Create Zookeeper Information Table

Source: https://github.com/didi/knowstreaming/blob/master/docs/install_guide/版本升级手册.md

SQL statement to create the `ks_km_zookeeper` table for storing Zookeeper information. Includes columns for cluster ID, host, port, role, version, status, and timestamps.

```sql
DROP TABLE IF EXISTS `ks_km_zookeeper`;
CREATE TABLE `ks_km_zookeeper` (
    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
    `cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT '物理集群ID',
    `host` varchar(128) NOT NULL DEFAULT '' COMMENT 'zookeeper主机名',
    `port` int(16) NOT NULL DEFAULT '-1' COMMENT 'zookeeper端口',
    `role` varchar(16) NOT NULL DEFAULT '' COMMENT '角色, leader follower observer',
    `version` varchar(128) NOT NULL DEFAULT '' COMMENT 'zookeeper版本',
    `status` int(16) NOT NULL DEFAULT '0' COMMENT '状态: 1存活，0未存活，11存活但是4字命令使用不了',
    `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
    `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
    PRIMARY KEY (`id`),
    UNIQUE KEY `uniq_cluster_phy_id_host_port` (`cluster_phy_id`,`host`, `port`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Zookeeper信息表';
```

--------------------------------

### Modify KnowStreaming Startup Script for JAAS

Source: https://github.com/didi/knowstreaming/blob/master/docs/dev_guide/接入ZK带认证Kafka集群.md

This command shows how to modify the KnowStreaming startup script to include the JAAS configuration file path. The '-Djava.security.auth.login.config' Java option points to the location of the 'zk_client_jaas.conf' file.

```bash
# `KnowStreaming/bin/startup.sh` 中的 47 行的 JAVA_OPT 中追加如下设置

-Djava.security.auth.login.config=/xxx/zk_client_jaas.conf
```