通过logstash-input-jdbc将mysql数据导入es5.x

logstash-input-jdbc是logstash的一个jdbc插件

1.在es5.x中通过如下命令安装该插件

1
2
3
4
5
cd /Users/xy/Workspace/search/logstash/logstash-5.3.0
bin/logstash-plugin install logstash-input-jdbc
Validating logstash-input-jdbc
Installing logstash-input-jdbc
Installation successful

2.配置一个mysql计划任务的logstash config文件

1
2
cd /Users/xy/Workspace/search/logstash/logstash-5.3.0/config
touch mysqlsync.conf

2.1文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
input {
stdin {
}
jdbc {
# mysql jdbc connection string to our backup databse
jdbc_connection_string => "jdbc:mysql://localhost:3306/btdb"
# the user we wish to excute our statement as
jdbc_user => "root"
jdbc_password => "123456"
# the path to our downloaded jdbc driver
jdbc_driver_library => "/Users/xy/Workspace/search/logstash/logstash-5.3.0/bin/mysql-connector-java-5.1.41.jar"
# the name of the driver class for mysql
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_default_timezone => "Asia/Shanghai"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement_filepath => "/Users/xy/Workspace/search/logstash/logstash-5.3.0/config/mysqlsync.sql"
schedule => "* * * * *"
type => "jdbc"
use_column_value => true
tracking_column => time_created
record_last_run => true
last_run_metadata_path => "/Users/xy/Workspace/search/logstash/logstash-5.3.0/config/last_run_metadata"
}
}

filter {
json {
source => "message"
remove_field => ["message"]
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "tmtpost"
document_type => "auction"
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}

2.2output中的document_id是指定的mysqlsync.sql中的一个字段名,一般使用主键就行

2.3input中的jdbc_driver_library从下面地址中下载

1
https://repo1.maven.org/maven2/mysql/mysql-connector-java/

2.4.statement_filepath指向的是需要同步的sql语句,logstash会监测sql语句中指定的table的update和insert操作,mysqlsync.sql示例如下

1
select id, name as title,body as summary, time_created from a

3.配置完成后在logstash根目录启动服务

1
bin/logstash -f /Users/xy/Workspace/search/logstash/logstash-5.3.0/config/mysqlsync.conf

看到如下输出则代表启动成功

试着更新mysql表中数据看看是否有同步到es中