### Elasticsearch Quick Start Guide
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
This section provides a quick start guide for integrating BBoss with Elasticsearch. It covers the initial setup and basic configurations required to get started.
```markdown
- [Elasticsearch Quick start](quickstart.md)
```
--------------------------------
### Pagination Configuration Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/datatran-http.md
Example of a query DSL script with pagination enabled.
```APIDOC
## Pagination Configuration
### Description
This section provides an example of a query DSL script configured for pagination.
### Example Query DSL Script
```xml
```
```
--------------------------------
### Query DSL Script Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/datatran-http.md
An example of a query DSL script configuration for paginated queries. It includes parameters for incremental time conditions, pagination start and size, and other service parameters.
```xml
```
--------------------------------
### HTTP Input/Output Plugin Guide
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
Guide on using the HTTP input and output plugins for data transfer with BBoss. Covers configuration for HTTP-based data synchronization.
```markdown
- [http服务输入输出插件使用指南](datatran-http.md)
```
--------------------------------
### BBoss Source Code Build Guide
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
A guide on how to build the BBoss source code. This is useful for developers who need to compile BBoss from its source.
```markdown
- [Bboss源码构建指南](bboss-build.md)
```
--------------------------------
### Initialize and Start MongoDB Data Source
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/MongoDBDatasource.md
Initialize and start the MongoDB data source using MongoDBHelper.init(). The returned boolean indicates if the data source was successfully started (true) or if it was already running or failed to start (false).
```java
boolean started = MongoDBHelper.init(mongoDBConfig)；
```
--------------------------------
### Data Synchronization Plugin Usage Guide
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
A guide to using various plugins for data synchronization tasks within BBoss. Covers different data sources and destinations.
```markdown
- [插件使用指南](datatran-plugins.md)
```
--------------------------------
### BBoss Development Guide
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
A comprehensive guide for developing with BBoss Elasticsearch. It covers core concepts, APIs, and best practices for developers.
```markdown
- [Elasticsearch开发指南](development.md)
```
--------------------------------
### Example of Multi-line SQL with Comments
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/Elasticsearch-SQL-ORM.md
This example demonstrates a multi-line SQL query within a JSON configuration, highlighting the use of #""" for multi-line SQL and the placement of comments.
```json
{
## 指示sql语句中的回车换行符会被替换掉开始符,注意dsl注释不能放到sql语句中，否则会有问题，因为sql中的回车换行符会被去掉，导致回车换行符后面的语句变道与注释一行
## 导致dsl模板解析的时候部分sql段会被去掉
"query": #"""
SELECT * FROM dbclobdemo
where channelId=#[channelId]
""",
## 指示sql语句中的回车换行符会被替换掉结束符
"fetch_size": #[fetchSize]
}
```
--------------------------------
### Set Incremental Start Value
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/usedatatran-in-spring-boot.md
Configure the starting value for incremental data import. This example sets the start date to '2000-01-01' for a timestamp-based incremental import.
```java
SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd");
try {
Date date = format.parse("2000-01-01");
importBuilder.setLastValue(date);//增量起始值配置
} catch (Exception e) {
e.printStackTrace();
}
```
--------------------------------
### Spring Boot中使用bbossESStarter检索数据
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/development.md
使用bbossESStarter组件加载demo.xml配置文件，创建ClientInterface实例。通过Map传递变量参数值，执行searchList方法进行检索，并指定返回的数据对象类型。
```java
//Create a load DSL file demo.xml client instance to retrieve documents use spring boot helper component bbossESStarter, single instance multithread security
ClientInterface clientUtil = bbossESStarter.getConfigRestClient("demo.xml");
//Set query conditions, pass variable parameter values via map,key for variable names in DSL
//通过Map或者Java PO对象传递实际的参数值
Map params = new HashMap();
//Set the values of applicationName1 and applicationName2 variables
params.put("applicationName1","app1");//设置变量applicationName1值
params.put("applicationName2","app2");//设置变量applicationName2值
DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
//Set the time range, and accept the Date or long value as the time parameter
try {
//设置变量startTime值
params.put("startTime",dateFormat.parse("2017-09-02 00:00:00"));
} catch (ParseException e) {
e.printStackTrace();
}
//设置变量endTime值
params.put("endTime",new Date());
//设置变量size值
params.put("size",1000);
//Execute the query
ESDatas esDatas = //ESDatas contains a collection of currently retrieved records, up to 1000 records, specified by the size attribute in the DSL
clientUtil.searchList("demo/_search",//demo as the indice, _search as the search action
"searchDatas",//DSL statement name defined in demo.xml
params,//Query parameters,传递上面定义的变量参数map
Demo.class);//Data object type Demo returned
//Gets a list of result objects and returns max up to 1000 records (specified in DSL)
List demos = esDatas.getDatas();
// String json = clientUtil.executeRequest("demo/_search",//demo as the index table, _search as the search action
// "searchDatas",//DSL statement name defined in esmapper/demo.xml
// params);//Query parameters
// String json = com.frameworkset.util.SimpleStringUtil.object2json(demos);
//Gets the total number of records
long totalSize = esDatas.getTotalSize();
```
--------------------------------
### Integrating Elasticsearch with BBoss
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
Learn how to integrate and configure Elasticsearch within your project using BBoss. This guide covers common project setups.
```markdown
- [集成和配置Elasticsearch bboss](common-project-with-bboss.md)
```
--------------------------------
### MySQL Binlog Input Plugin Guide
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
Instructions for using the MySQL Binlog input plugin with BBoss for capturing database changes and synchronizing them.
```markdown
- [Mysql binlog输入插件使用指南](mysql-binlog.md)
```
--------------------------------
### MongoDB Data Source Definition and Usage
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
Guide on defining and using MongoDB as a data source within BBoss. Covers connection setup and data access.
```markdown
- [MongoDB数据源定义和使用](MongoDBDatasource.md)
```
--------------------------------
### BBoss JobFlow Usage Introduction
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
An introduction to using BBoss JobFlow for workflow management. Covers basic concepts and setup.
```markdown
- [bboss jobflow使用介绍](jobworkflow.md)
```
--------------------------------
### Configure Incremental Import
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/usedatatran-in-spring-boot.md
Set up incremental import parameters, including the incremental field, starting value, and storage path. This example configures a timestamp-based incremental import.
```java
//增量配置开始
////importBuilder.setLastValueColumn("Info:id");//指定数字增量查询字段变量名称
importBuilder.setFromFirst(true);//任务重启时，重新开始采集数据，true 重新开始，false不重新开始，适合于每次全量导入数据的情况，如果是全量导入，可以先删除原来的索引数据
importBuilder.setStatusDbname("hbase233esdemo");
importBuilder.setLastValueStorePath("hbase233esdemo_import");//记录上次采集的增量字段值的文件路径，作为下次增量（或者重启后）采集数据的起点，不同的任务这个路径要不一样
//指定增量字段类型为日期类型，如果没有指定增量字段名称,则按照hbase记录时间戳进行timerange增量检索
importBuilder.setLastValueType(ImportIncreamentConfig.TIMESTAMP_TYPE);
// ImportIncreamentConfig.NUMBER_TYPE 数字类型
// ImportIncreamentConfig.TIMESTAMP_TYPE 日期类型
//设置增量查询的起始值时间
SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd");
try {
Date date = format.parse("2000-01-01");
importBuilder.setLastValue(date);
}
catch (Exception e){
e.printStackTrace();
}
//增量配置结束
```
--------------------------------
### BBoss AI Usage Tutorial
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
A tutorial on how to use BBoss AI functionalities. Covers AI-related features and their applications.
```markdown
- [bboss ai使用教程](bboss-ai.md)
```
--------------------------------
### Get BBoss Elasticsearch Client Instances
Source: https://context7.com/bbossgroups/bboss-elasticsearch/llms.txt
Obtain different types of Elasticsearch client interfaces using ElasticSearchHelper. Supports simple REST clients, configuration-based clients loading DSL from XML, and multi-cluster setups. Verify connection by fetching cluster state.
```java
import org.frameworkset.elasticsearch.ElasticSearchHelper;
import org.frameworkset.elasticsearch.client.ClientInterface;
// Get a simple REST client for direct API calls
ClientInterface restClient = ElasticSearchHelper.getRestClientUtil();
// Get a configuration client that loads DSL from XML files
ClientInterface configClient = ElasticSearchHelper.getConfigRestClientUtil("esmapper/demo.xml");
// Get client for a specific named cluster (multi-cluster support)
ClientInterface logsClient = ElasticSearchHelper.getRestClientUtil("logs");
ClientInterface logsConfigClient = ElasticSearchHelper.getConfigRestClientUtil("logs", "esmapper/demo.xml");
// Verify connection by getting cluster state
String clusterState = restClient.executeHttp("_cluster/state?pretty", ClientInterface.HTTP_GET);
System.out.println(clusterState);
```
--------------------------------
### Add and Get Document by Date and ID
Source: https://github.com/bbossgroups/bboss-elasticsearch/wiki/高性能elasticsearch-ORM开发库使用介绍
This example demonstrates adding a document to an index with a date-based name and then retrieving it by document ID. It uses a DSL script defined in `esmapper/estrace/ESTracesMapper.xml` for document creation. The `addDateDocument` method automatically appends the current date to the index name.
```java
public void testAddDateDocument() throws ParseException{
testGetMapping();
SimpleDateFormat format = new SimpleDateFormat("yyyy.MM.dd");
String date = format.format(new Date());
ClientInterface clientUtil = ElasticSearchHelper.getConfigRestClientUtil("esmapper/estrace/ESTracesMapper.xml");
Demo demo = new Demo();
demo.setDemoId(5l);
demo.setAgentStarttime(new Date());
demo.setApplicationName("blackcatdemo");
demo.setContentbody("this is content body");
//根据dsl脚本创建索引文档，将文档保存到当天的索引表中demo-2018.02.03
String response = clientUtil.addDateDocument("demo",//索引表,自动添加日期信息到索引表名称中
"demo",//索引类型
"createDemoDocument",//创建文档对应的脚本名称，在esmapper/estrace/ESTracesMapper.xml中配置
demo);
System.out.println("addDateDocument-------------------------");
System.out.println(response);
//根据文档id获取索引文档,返回json格式
response = clientUtil.getDocument("demo-"+date,//索引表，手动指定日期信息
"demo",//索引类型
"5");
System.out.println("getDocument-------------------------");
System.out.println(response);
//根据文档id获取索引文档,返回Demo对象
demo = clientUtil.getDocument("demo-"+date,//索引表
"demo",//索引类型
"5",//索引文档ID
Demo.class);
}
```
--------------------------------
### BBoss Introduction
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
An introduction to the BBoss framework, providing an overview of its features and capabilities.
```markdown
- [BBOSS介绍](README.md)
```
--------------------------------
### Create Client and Update Document with Script
Source: https://github.com/bbossgroups/bboss-elasticsearch/wiki/高性能elasticsearch-ORM开发库使用介绍
Demonstrates creating an Elasticsearch client using configuration and updating a document by its path using a script. It also shows how to retrieve the updated document.
```java
ClientInterface clientUtil = ElasticSearchHelper.getConfigRestClientUtil(mappath);
Map params = new HashMap();
DynamicPriceTemplate dynamicPriceTemplate = new DynamicPriceTemplate();
dynamicPriceTemplate.setGoodsId(1);
dynamicPriceTemplate.setGoodName("asd\"国家");
List ruleList = new ArrayList();
Rule rule = new Rule();
rule.setRuleCount(100);
rule.setRuleExist(true);
rule.setRuleId("asdfasdfasdf");
ruleList.add(rule);
rule = new Rule();
rule.setRuleCount(101);
rule.setRuleExist(false);
rule.setRuleId("bbbb");
ruleList.add(rule);
rule = new Rule();
rule.setRuleCount(103);
rule.setRuleExist(true);
rule.setRuleId("ccccc");
ruleList.add(rule);
dynamicPriceTemplate.setRules(ruleList);
//为id为2的文档增加last和nick两个属性
params.put("last","gaudreau");
params.put("nick","hockey");
params.put("id",3);
params.put("dynamicPriceTemplate",dynamicPriceTemplate);
//通过script脚本为id为2的文档增加last和nick两个属性，为了演示效果强制refresh，实际环境慎用
clientUtil.updateByPath("demo/demo/_update_by_query?refresh","scriptDslByQuery",params);
//获取更新后的文档，会看到新加的2个字段属性
String doc = clientUtil.getDocument("demo","demo","3");
System.out.println(doc);
```
--------------------------------
### Install IK Analyzer Plugin for Elasticsearch
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/SpanQuery.md
This command installs the IK Analyzer plugin for Elasticsearch. Ensure the version is compatible with your Elasticsearch version. Restart Elasticsearch after installation.
```bash
/ES_HOMEW/bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.7.0/elasticsearch-analysis-ik-7.7.0.zip
```
--------------------------------
### Data Synchronization and Stream-Batch Processing Demo
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
Demonstrates data synchronization and unified stream-batch processing using BBoss. Provides practical examples and use cases.
```markdown
- [数据采集及流批一体化计算案例](bboss-datasyn-demo.md)
```
--------------------------------
### Set Incremental Sync Starting Value for Numeric Fields
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/db-es-tool.md
Configure the starting value for incremental synchronization when using a numeric field. If not specified, the default start value is 0.
```java
try {
importBuilder.setLastValue(100);//增量起始值配置
}
catch (Exception e){
e.printStackTrace();
}
```
--------------------------------
### Set Incremental Sync Starting Value for Date Fields
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/db-es-tool.md
Configure the starting value for incremental synchronization when using a date field. If not specified, the default start date is 1970-01-01.
```java
SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd");
try {
Date date = format.parse("2000-01-01");
importBuilder.setLastValue(date);//增量起始值配置
}
catch (Exception e){
e.printStackTrace();
}
```
--------------------------------
### XxlJob Lifecycle Task Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/xxljobdatasyn.md
Demonstrates an XxlJob task with custom initialization and destruction logic using init and destroy methods.
```java
@XxlJob(value = "demoJobHandler2", init = "init", destroy = "destroy")
public void demoJobHandler2() throws Exception {
try {
lock.lock();
String param = XxlJobHelper.getJobParam();
externalScheduler.execute( param);
XxlJobHelper.handleSuccess("作业执行ok！");
}
finally {
lock.unlock();
}
}
```
--------------------------------
### File-to-ES Job Started Confirmation
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-control.md
Confirms that the file-elasticsearch data transfer job has been successfully started.
```JSON
file2ES job started.
```
```JSON
file2ES job has started.
```
--------------------------------
### Sample application.properties Configuration
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/mongodbgitchat.md
This is a comprehensive example of an application.properties file for configuring bboss-elasticsearch data synchronization jobs, including main class, Elasticsearch connection details, HTTP client settings, and IP geolocation database paths.
```properties
#同步作业主程序配置
mainclass=org.frameworkset.elasticsearch.imp.Mongodb2ES
# Elasticsearch配置
##x-pack或者searchguard账号和口令
elasticUser=elastic
elasticPassword=changeme
elasticsearch.rest.hostNames=192.168.137.1:9200
#elasticsearch.rest.hostNames=10.180.211.27:9280,10.180.211.27:9281,10.180.211.27:9282
#在控制台输出脚本调试开关showTemplate,false关闭，true打开，同时log4j至少是info级别
elasticsearch.showTemplate=true
elasticsearch.discoverHost=false
##http连接池配置
http.timeoutConnection = 5000
http.timeoutSocket = 50000
http.connectionRequestTimeout=10000
http.retryTime = 1
http.maxLineLength = -1
http.maxHeaderCount = 200
http.maxTotal = 200
http.defaultMaxPerRoute = 100
http.soReuseAddress = false
http.soKeepAlive = false
http.timeToLive = 3600000
http.keepAlive = 3600000
http.keystore =
http.keyPassword =
# ssl 主机名称校验，是否采用default配置，
# 如果指定为default，就采用DefaultHostnameVerifier,否则采用 SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER
http.hostnameVerifier =
# dsl配置文件热加载扫描时间间隔，毫秒为单位，默认5秒扫描一次，<= 0时关闭扫描机制
dslfile.refreshInterval = 3000
# IP地理位置信息库配置
ip.cachesize = 2000
# 库下载地址https://dev.maxmind.com/geoip/geoip2/geolite2/
ip.database = d:/geolite2/GeoLite2-City.mmdb
ip.asnDatabase = d:/geolite2/GeoLite2-ASN.mmdb
```
--------------------------------
### HBase-to-ES Job Started Confirmation
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-control.md
Confirms that the hbase-elasticsearch data transfer job has been successfully started.
```JSON
HBase2ES job started.
```
```JSON
HBase2ES job has started.
```
--------------------------------
### Create and Initialize DSL Configuration Table
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/db-dsl.md
This Java code demonstrates how to create a 'dslconfig' table in a SQLite database and then populate it with DSL configurations loaded from an XML file. It includes error handling for table creation and uses SQLExecutor for database operations. Ensure the 'testdslconfig' data source is properly configured.
```java
String createStatusTableSQL = new StringBuilder()
.append("create table dslconfig (ID string,name string,namespace string,dslTemplate TEXT,vtpl number(1),multiparser number(1) ")
.append(",referenceNamespace string,referenceTemplateName string,PRIMARY KEY (ID))").toString();
try {
String exist = "select 1 from dslconfig";
//SQLExecutor.updateWithDBName("gencode","drop table BBOSS_GENCODE");
SQLExecutor.queryObjectWithDBName(int.class,"testdslconfig", exist);
logger.info("重建建dslconfig表："+createStatusTableSQL+"。");
SQLExecutor.updateWithDBName("testdslconfig","drop table dslconfig");
SQLExecutor.updateWithDBName("testdslconfig",createStatusTableSQL);
logger.info("重建建dslconfig表成功。");
} catch (Exception e) {
logger.info("dslconfig table 不存在，创建dslconfig表："+createStatusTableSQL+"。");
try {
SQLExecutor.updateWithDBName("testdslconfig",createStatusTableSQL);
logger.info("创建dslconfig表成功："+createStatusTableSQL+"。");
} catch (SQLException e1) {
logger.info("创建dslconfig表失败："+createStatusTableSQL+".",e1);
e1.printStackTrace();
}
}
//初始化dsl配置：将配置文件中的sql转存到数据库中
String dslpath = "esmapper/demo.xml";
final String namespace = "testnamespace";
AOPTemplateContainerImpl aopTemplateContainer = ElasticSearchHelper.getAOPTemplateContainerImpl(dslpath);
int perKeyDSLStructionCacheSize = aopTemplateContainer.getPerKeyDSLStructionCacheSize();
boolean alwaysCacheDslStruction = aopTemplateContainer.isAlwaysCacheDslStruction();
List templateMetaList = aopTemplateContainer.getTemplateMetas(namespace);
//保存dsl到表dslconfig
SQLExecutor.insertBeans("testdslconfig",
"insert into dslconfig(ID,name,namespace,dslTemplate,vtpl,multiparser,referenceNamespace,referenceTemplateName) " +
"values(#[id]," + //主键
"#[name]," + //dsl名称
"#[namespace]," + //dsl所属命名空间
"#[dslTemplate]," + //dsl语句
"#[vtpl]," + //一般设置为true， dsl语句中是否包含velocity语法内容，包含为true，否则为false（避免进行velocity语法解析，提升性能），默认为true
"#[multiparser]," + // 一般设置为true，dsl如果包含velocity动态语法，是否需要对每次动态生成的dsl进行模板变量#[xxxx]语法解析，true 需要，false不需要，默认true
"#[referenceNamespace]," + // 如果对应的配置是一个引用，则需要通过referenceNamespace指定引用的dsl所属的命名空间
"#[referenceTemplateName])",
templateMetaList);
```
--------------------------------
### DB-to-ES Job Started Confirmation
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-control.md
Confirms that the db-elasticsearch data transfer job has been successfully started.
```JSON
db2ESImport job started.
```
```JSON
db2ESImport job has started.
```
--------------------------------
### Elasticsearch SQL Introduction
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
An introduction to using SQL queries with Elasticsearch through BBoss. Covers the basics of Elasticsearch SQL support.
```markdown
- [Elasticsearch SQL介绍](Elasticsearch-6-SQL.md)
```
--------------------------------
### Build and Start Job Flow
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/jobflow-remotefile-parrel.md
Builds the job flow using the configured `JobFlowBuilder` and then starts its execution.
```java
JobFlow jobFlow = jobFlowBuilder.build();
jobFlow.start();
```
--------------------------------
### Standard Project Environment Initialization
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/development.md
Obtain singleton, thread-safe `ClientInterface` instances using static factory methods from `ElasticSearchHelper`.
```APIDOC
## Standard Project Environment Initialization
### Description
Obtain singleton, thread-safe `ClientInterface` instances using static factory methods from `ElasticSearchHelper`.
### Methods
- `ElasticSearchHelper.getConfigRestClientUtil(String configFile)`
- `ElasticSearchHelper.getConfigRestClientUtil(String elasticSearch, String configFile)`
- `ElasticSearchHelper.getRestClientUtil()`
- `ElasticSearchHelper.getRestClientUtil(String elasticSearch)`
### Usage Examples
#### Loading DSL from Configuration File
```java
// Load configuration file, create ES client toolkit, operate on the default ES data source
ClientInterface clientUtil = ElasticSearchHelper.getConfigRestClientUtil("estrace/ESTracesqlMapper.xml");
// Load configuration file, create ES client toolkit, operate on the specified ES data source 'order'
ClientInterface clientUtil = ElasticSearchHelper.getConfigRestClientUtil("order", "estrace/ESTracesqlMapper.xml");
```
#### Direct DSL Operation
```java
// Get a RestClientUtil instance, single instance multi-thread security
ClientInterface clientUtil = ElasticSearchHelper.getRestClientUtil();
// Get a RestClientUtil instance for a specific cluster named 'othercluster'
ClientInterface clientUtil = ElasticSearchHelper.getRestClientUtil("othercluster");
// Example of direct DSL query
public void testDirectDslQuery(){
String queryAll = "{\"query\": {\"match_all\": {}}}";
// Operate on the default ES data source
ClientInterface clientUtil = ElasticSearchHelper.getRestClientUtil();
// Operate on the 'order' ES data source
ClientInterface clientUtil = ElasticSearchHelper.getRestClientUtil("order");
ESDatas esDatas = clientUtil.searchList("demo/_search", // 'demo' is the index, '_search' is the search action
queryAll, // queryAll variable corresponds to the DSL statement
Demo.class);
// Get the list of result objects
List demos = esDatas.getDatas();
// Get the total number of records
long totalSize = esDatas.getTotalSize();
System.out.println(totalSize);
}
```
### Notes
- Instances obtained via `getConfigRestClientUtil` and `getRestClientUtil` are thread-safe and singleton.
- `getConfigRestClientUtil` instances are subclasses of `getRestClientUtil` instances and possess all their functionalities.
- When using configuration files with parameters, pass parameters using Map or Bean objects.
```
--------------------------------
### Start Log Collection Job
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/filelog-oom.md
Builds and executes the data stream for log collection. Logs a message indicating the job has started.
```java
/**
* 启动采集日志文件作业
*/
DataStream dataStream = importBuilder.builder();
dataStream.execute();//启动同步作业
logger.info("job started.");
```
--------------------------------
### Spring Boot Environment Initialization
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/development.md
Obtain singleton, thread-safe `ClientInterface` instances using factory methods provided by `BBossESStarter`.
```APIDOC
## Spring Boot Environment Initialization
### Description
In a Spring Boot environment, obtain singleton, thread-safe `ClientInterface` instances using factory methods provided by `BBossESStarter`.
### Setup
Ensure `BBossESStarter` is injected into your component:
```java
@Autowired
private BBossESStarter bbossESStarter;
```
### Methods
- `bbossESStarter.getConfigRestClient(String mappath)`
- `bbossESStarter.getRestClient()`
### Usage Examples
#### Loading DSL from Configuration File
```java
// Get a ConfigRestClientUtil instance to load configuration files, single instance multithreaded security
ClientInterface clientUtil = bbossESStarter.getConfigRestClient(mappath);
```
#### Direct DSL Operation
```java
// Build a RestClientUtil instance, single instance multi-thread security
ClientInterface clientUtil = bbossESStarter.getRestClient();
```
### Notes
- Instances obtained via `BBossESStarter` are thread-safe and singleton.
```
--------------------------------
### Start File-to-ES Data Transfer Job
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-control.md
Initiates the file-elasticsearch data transfer job. This endpoint starts the file synchronization process.
```HTTP
http://localhost:808/schedulecontrol/startfile2es
```
--------------------------------
### Start HBase-to-ES Data Transfer Job
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-control.md
Initiates the hbase-elasticsearch data transfer job. This endpoint starts the data synchronization process.
```HTTP
http://localhost:808/schedulecontrol/scheduleHBase2ESJob
```
--------------------------------
### Elasticsearch Pagination: From-Size
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
Demonstrates how to implement pagination in Elasticsearch using the 'from' and 'size' parameters with BBoss.
```markdown
- [from-size](from-size.md)
```
--------------------------------
### Create and Use Elasticsearch Client for Document Retrieval
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/development.md
Initializes an Elasticsearch client instance using a configuration file and demonstrates how to set query parameters, execute a search, and retrieve results. Ensure the demo.xml configuration file is properly set up.
```java
ClientInterface clientUtil = ElasticSearchHelper.getConfigRestClientUtil("demo.xml");
Map params = new HashMap();
params.put("applicationName1","app1");//设置变量applicationName1值
params.put("applicationName2","app2");//设置变量applicationName2值
DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
try {
//设置变量startTime值
params.put("startTime",dateFormat.parse("2017-09-02 00:00:00"));
} catch (ParseException e) {
e.printStackTrace();
}
//设置变量endTime值
params.put("endTime",new Date());
//设置变量size值
params.put("size",1000);
ESDatas esDatas = //ESDatas contains a collection of currently retrieved records, up to 1000 records, specified by the size attribute in the DSL
clientUtil.searchList("demo/_search",//demo as the indice, _search as the search action
"searchDatas",//DSL statement name defined in esmapper/demo.xml
params,//Query parameters,传递上面定义的变量参数map
Demo.class);
List demos = esDatas.getDatas();
// String json = clientUtil.executeRequest("demo/_search",//demo as the index table, _search as the search action
// "searchDatas",//DSL statement name defined in esmapper/demo.xml
// params);//Query parameters
// String json = com.frameworkset.util.SimpleStringUtil.object2json(demos);
long totalSize = esDatas.getTotalSize();
```
--------------------------------
### Example Output of Field Collapsing Inner Hits
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/FiledCollapsing.md
This is an example of the output returned by the Java code executing the Elasticsearch field collapsing query with inner hits. It shows the structure of the `RecipesPo` objects, including nested inner hit data (though null in this specific output example).
```text
RecipesPo{name='鲫鱼汤（变态辣）', rating=5, type='湘菜', innerHitsRecipesPo=null}
RecipesPo{name='鲫鱼汤（微辣）', rating=4, type='湘菜', innerHitsRecipesPo=null}
RecipesPo{name='广式鲫鱼汤', rating=5, type='粤菜', innerHitsRecipesPo=null}
RecipesPo{name='奶油鲍鱼汤', rating=2, type='西菜', innerHitsRecipesPo=null}
RecipesPo{name='鱼香肉丝', rating=null, type='川菜', innerHitsRecipesPo=null}
```
--------------------------------
### ServerRealm RESTful Service Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/httpproxy.md
Example Java code for a RESTful service that returns the ServerRealm. This service is used in ServerRealm mode for Kerberos authentication.
```java
@RequestMap("/elasticsearch/serverrealm")
public @ResponseBody String serverrealm(String appName){
// return "BBOSSGROUPS.COM";
return "elasticsearch/hadoop.bbossgroups.com@BBOSSGROUPS.COM";
}
```
--------------------------------
### Elasticsearch to File and Upload to Minio Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-demo.md
Demonstrates collecting data from Elasticsearch, processing it, outputting it to a file, and then uploading the file to Minio object storage. This is useful for data archiving and backup.
```Java
https://gitee.com/bboss/elasticsearch-file2ftp/blob/main/src/main/java/org/frameworkset/elasticsearch/imp/ES2FileMinioDemo.java
```
--------------------------------
### Boosting Query Explanation Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/DocRelevancy.md
Example of the explanation returned by Elasticsearch for a boosting query. This shows the score calculation, including the positive score and the effect of the negative_boost.
```json
{
"value" : 0.09808561352022904,
"description" : "weight(FunctionScoreQuery(title:es title:相关性, scored by boost(queryboost(score(content:编程))^0.2))), result of:",
"details" : [
{
"value" : 0.09808561352022904,
"description" : "product of:",
"details" : [
{
"value" : 0.49042806,
"description" : "sum of:",
"details" : []
},
{
"value" : 0.2,
"description" : "Matched boosting query score(content:编程)",
"details" : [ ]
}
]
}
]
}
```
--------------------------------
### File Log Collection Guide
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/_sidebar.md
Instructions on how to use the File Log plugin for collecting log data with BBoss. Covers configuration and usage for log aggregation.
```markdown
- [日志文件采集使用指南](filelog-guide.md)
```
--------------------------------
### Build and Start Job Flow with Nodes
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/jobworkflow.md
This code snippet demonstrates the construction and initiation of a BBoss job flow, including the addition of various job flow node builders and condition nodes with triggers.
```java
protected ImportBuilder buildImportBuilder(JobFlowNodeExecuteContext jobFlowNodeExecuteContext) {
ImportBuilder importBuilder = OpptyMetricsToFeishuTableOnceJob.buildImportBuilder(jobFlowNodeExecuteContext);
jobFlowNodeExecuteContext.addJobFlowContextData("executeDatatranJobFlowNode",true);
return importBuilder;
}
};
sequenceJobFlowNodeBuilder.addJobFlowNodeBuilder(datatranJobFlowNodeBuilder);
CallableJobFlowNodeBuilder saveUpdateTimeJobFlowNodeBuilder = new CallableJobFlowNodeBuilder("记录最后工作总结更新时间") {
@Override
public Void call(JobFlowNodeExecuteContext jobFlowNodeExecuteContext) throws Exception {
JobFlowExecuteContext jobFlowExecuteContext = jobFlowNodeExecuteContext.getJobFlowExecuteContext();
SynObjectHolder updateTimeHolder = (SynObjectHolder)jobFlowExecuteContext.getContextData("updateTimeHolder");
if(updateTimeHolder != null){
Long updateTime = updateTimeHolder.getObject();
opptyMetricsService.saveUpdateTime("metrics-to-feishu-table",updateTime);
}
return null;
}
};
sequenceJobFlowNodeBuilder.addJobFlowNodeBuilder(saveUpdateTimeJobFlowNodeBuilder);
//作为默认节点添加到流程中删除、采集条件分支节点，如果没有数据需要删除时，则执行采集作业节点
jobFlowBuilder.addConditionJobFlowNodeBuilder(sequenceJobFlowNodeBuilder,true);
NodeTrigger researchdataNodeTrigger = new NodeTrigger();
researchdataNodeTrigger.setTriggerScriptAPI(new TriggerScriptAPI() {
@Override
public boolean needTrigger(NodeTriggerContext nodeTriggerContext) throws Exception {
Boolean executeDatatranJobFlowNode = (Boolean) nodeTriggerContext.getFlowContextData("executeDatatranJobFlowNode");//如果已经重新采集过数据，则不在执行查询和删除数据操作
if(executeDatatranJobFlowNode != null){
return false;
}
return true;
}
});
//5. 添加条件节点：如果还有数据则循环执行数据查询和删除节点
//如果已经重新采集过数据，则不再执行查询和删除数据操作
jobFlowBuilder.addAnotherConditionJobFlowNodeBuilder(searchdataFlowNodeBuilder,researchdataNodeTrigger);
JobFlow jobFlow = jobFlowBuilder.build();
jobFlow.start();
```
--------------------------------
### Collect Binlog Data to DB Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-demo.md
Shows how to collect data from MySQL binlog and output it to a database. This is useful for replicating database changes to another system.
```Java
https://gitee.com/bboss/bboss-datatran-demo/blob/main/src/main/java/org/frameworkset/datatran/imp/binlog/Binlog2DBOutput.java
```
--------------------------------
### Start Data Synchronization Job
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-control.md
Call the execute method on a DataStream object to start a scheduled synchronization job. The job will run periodically based on its configured timer.
```java
/**
* 创建一个数据同步作业对象
*/
DataStream dataStream = importBuilder.builder();
dataStream.execute();//启动作业
```
--------------------------------
### Built-in Timer Scheduling Strategy Example
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/bboss-datasyn-demo.md
Provides an example of using the built-in timer scheduling strategy for job flows. This is useful for automating data processing tasks.
```Java
https://gitee.com/bboss/bboss-datatran-demo/blob/main/src/main/java/org/frameworkset/datatran/imp/jobflow/JobFlowTest.java
```
--------------------------------
### Build and Execute Deepseek JobFlow
Source: https://github.com/bbossgroups/bboss-elasticsearch/blob/master/docs/jobflow-deepseek.md
Initializes the Deepseek service and constructs a JobFlow with two sequential tasks: one to write poetry and another to evaluate it. This example demonstrates setting up the JobFlow, defining node functions, and managing conversation context.
```java
public static void main(String[] args) {
//初始化Deepseek服务
initDeepseekService();
//构建流程
JobFlowBuilder jobFlowBuilder = new JobFlowBuilder();
jobFlowBuilder.setJobFlowName("Deepseek写诗-评价诗词流程")
.setJobFlowId("测试id");
JobFlowScheduleConfig jobFlowScheduleConfig = new JobFlowScheduleConfig();
jobFlowScheduleConfig.setExecuteOneTime(true);
jobFlowBuilder.setJobFlowScheduleConfig(jobFlowScheduleConfig);
/**
* 1.构建第一个任务节点：单任务节点 写诗
*/
DeepseekJobFlowNodeBuilder jobFlowNodeBuilder = new DeepseekJobFlowNodeBuilder("1", "Deepseek-chat-写诗", new DeepseekJobFlowNodeFunction() {
@Override
public Object call(JobFlowNodeExecuteContext jobFlowNodeExecuteContext) throws Exception {
List deepseekMessageList = new ArrayList<>();
DeepseekMessage deepseekMessage = new DeepseekMessage();
deepseekMessage.setRole("system");
deepseekMessage.setContent("你是一位唐代诗人.");
deepseekMessageList.add(deepseekMessage);
//将通话记录添加到工作流上下文中，保存Deepseek通话记录
jobFlowNodeExecuteContext.addJobFlowContextData("messages", deepseekMessageList);
deepseekMessage = new DeepseekMessage();
deepseekMessage.setRole("user");
deepseekMessage.setContent("模仿李白的风格写一首七律.飞机!");
//将问题添加到工作流上下文中，保存Deepseek通话记录
deepseekMessageList.add(deepseekMessage);
DeepseekMessages deepseekMessages = new DeepseekMessages();
deepseekMessages.setMessages(deepseekMessageList);
deepseekMessages.setModel(model);
deepseekMessages.setStream(stream);
deepseekMessages.setMax_tokens(this.max_tokens);
//调用Deepseek 对话api提问
Map response = HttpRequestProxy.sendJsonBody(this.getDeepseekService(), deepseekMessages, "/chat/completions",Map.class);
List choices = (List) response.get("choices");
Map message = (Map) ((Map)choices.get(0)).get("message");
deepseekMessage = new DeepseekMessage();
deepseekMessage.setRole("assistant");
deepseekMessage.setContent((String)message.get("content"));
//将问题答案添加到工作流上下文中，保存Deepseek通话记录
deepseekMessageList.add(deepseekMessage);
logger.info(deepseekMessage.getContent());
return response;
}
}).setDeepseekService("deepseek").setModel("deepseek-chat").setMax_tokens(4096);
/**
* 2 将第一个节点添加到工作流构建器
*/
jobFlowBuilder.addJobFlowNodeBuilder(jobFlowNodeBuilder);
/**
* 3.构建第二个任务节点：单任务节点 分析诗
*/
jobFlowNodeBuilder = new DeepseekJobFlowNodeBuilder("2", "Deepseek-chat-分析诗", new DeepseekJobFlowNodeFunction() {
@Override
public Object call(JobFlowNodeExecuteContext jobFlowNodeExecuteContext) throws Exception {
//从工作流上下文中，获取Deepseek历史通话记录
List deepseekMessageList = (List) jobFlowNodeExecuteContext.getJobFlowContextData("messages");
//将第二个问题添加到工作流上下文中，保存Deepseek通话记录
DeepseekMessage deepseekMessage = new DeepseekMessage();
deepseekMessage.setRole("user");
deepseekMessage.setContent("帮忙评估上述诗词的意境");
deepseekMessageList.add(deepseekMessage);
DeepseekMessages deepseekMessages = new DeepseekMessages();
deepseekMessages.setMessages(deepseekMessageList);
deepseekMessages.setModel(model);
deepseekMessages.setStream(stream);
deepseekMessages.setMax_tokens(this.max_tokens);
//调用Deepseek 对话api提问
Map response = HttpRequestProxy.sendJsonBody(this.getDeepseekService(), deepseekMessages, "/chat/completions", Map.class);
List choices = (List) response.get("choices");
Map message = (Map) ((Map) choices.get(0)).get("message");
deepseekMessage = new DeepseekMessage();
deepseekMessage.setRole("assistant");
deepseekMessage.setContent((String) message.get("content"));
//将第二个问题答案添加到工作流上下文中，保存Deepseek通话记录
deepseekMessageList.add(deepseekMessage);
logger.info(deepseekMessage.getContent());
return response;
}
}).setDeepseekService("deepseek").setModel("deepseek-chat").setMax_tokens(4096);
/**
* 4 将第二个节点添加到工作流构建器
*/
jobFlowBuilder.addJobFlowNodeBuilder(jobFlowNodeBuilder);
//构建和运行与Deepseek通话流程
JobFlow jobFlow = jobFlowBuilder.build();
```