利用Logstash同步MySql数据至ElasticSearch

安装Logstash

参考链接Installing Logstash
Download and install the public signing key:

rpm --importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo

[logstash-7.x]name=Elastic repository for 7.x packages

baseurl=https://artifacts.elastic.co/packages/7.x/yum

gpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1autorefresh=1type=rpm-md

And your repository is ready for use. You can install it with:

sudo yum install logstash

yum方式安装好后，程序目录位于/usr/share/logstash/, 配置文件位于/etc/logstash/，

配置MySql数据同步

在/etc/logstash中创建mysql.conf配置文件,参考链接jdbc input plugin

input {
  jdbc {
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://xxx.mysql.rds.aliyuncs.com/xxx"
    jdbc_user => "root"
    jdbc_password => "******"
    schedule => "* * * * *"
    statement => "SELECT * from biz_goods  where create_time > :sql_last_value AND create_time < NOW() AND is_delete=0 ORDER BY create_time desc"
  }
}

output {
    elasticsearch {
         hosts => ["127.0.0.1:9200"]
         index => "goods"
         document_id => "%{id}"
    }
}

在官方文档中input->jdbc中需要指定jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"，此处如果将mysq-connector-java.jar放在常规目录通过路径指定，将会出现如下错误

LogStash::ConfigurationError
com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?

解决方式为在mvnrepository下载jar文件后，将该文件拷贝至/usr/share/logstash/logstash-core/lib/jars 中，然后无需在input->jdbc中配置jdbc_driver_library

运行logstash开启同步

./logstash -f /etc/logstash/mysql.conf