Jdbc input plugin
Plugin version: v4.2.1
Released on: 2017-06-23
插件版本:v4.2.1
日期:2017-06-23
Getting Help
For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.
有关插件的问题,打开Discuss 论坛中的主题。对于bugs和特性的询问,在Github查看问题。对于一系列Elastic支持的插件,请参阅Elastic Support Matrix。
Description
This plugin was created as a way to ingest data in any database with a JDBC interface into Logstash. You can periodically schedule ingestion using a cron syntax (see schedule
setting) or run the query one time to load data into Logstash. Each row in the resultset becomes a single event. Columns in the resultset are converted into fields in the event.
这插件被创建作为在任何一种具有JDBC接口的数据库将数据摄入到Logstash。你可以使用cron语法定期摄取或执行一次查询来把数据加载如Logstash。结果的每一行都会成为单个事件。结果的每一列都会转换成事件中的字段。
Drivers
This plugin does not come packaged with JDBC driver libraries. The desired jdbc driver library must be explicitly passed in to the plugin using the jdbc_driver_library
configuration option.
这插件不包含JDBC驱动程序库。 必须使用jdbc_driver_library配置选项将所需的jdbc驱动程序库显式传递到插件。
Scheduling
Input from this plugin can be scheduled to run periodically according to a specific schedule. This scheduling syntax is powered by rufus-scheduler. The syntax is cron-like with some extensions specific to Rufus (e.g. timezone support ).
插件的输入可以按照特定的计划表定期执行。该计划表语法通过rufus-scheduler来调度实现。该语法类似于cron,包含特定于Rufus的某些扩展(例如时区支持)。
1):* 5 * 1-3 *
will execute every minute of 5am every day of January through March.
将于1月至3月的每一天上午5点每分钟执行一次。
2):0 * * * *
will execute on the 0th minute of every hour every day.
将在每一小时的第0分钟执行。
3):0 6 * * * America/Chicago
will execute at 6:00am (UTC/GMT -5) every day.
将每天上午6:00(UTC / GMT -5)执行。
Further documentation describing this syntax can be found here.
更多语法描述见文档.
State
The plugin will persist the sql_last_value
parameter in the form of a metadata file stored in the configured last_run_metadata_path
. Upon query execution, this file will be updated with the current value of sql_last_value
. Next time the pipeline starts up, this value will be updated by reading from the file. If clean_run
is set to true, this value will be ignored and sql_last_value
will be set to Jan 1, 1970, or 0 if use_column_value
is true, as if no query has ever been executed.
该插件将以存储在配置的last_run_metadata_path中的元数据文件的形式持久保存sql_last_value参数。 执行查询时,这个文件会被sql_last_value当前值进行更新。下一次管道开启时,该值会从该文件里面读取。如果clean_run被设为true,该值会被忽略并且sql_last_value值会被设为1970年1月1日,如果use_column_value为true,则为0,因为没有执行任何查询。
Dealing With Large Result-sets
Many JDBC drivers use the fetch_size
parameter to limit how many results are pre-fetched at a time from the cursor into the client’s cache before retrieving more results from the result-set. This is configured in this plugin using the jdbc_fetch_size
configuration option. No fetch size is set by default in this plugin, so the specific driver’s default size will be used.
很多JDBC驱动使用fetch_size参数来限制在从结果集中检索更多结果之前,从光标到客户端的缓存中一次预取的结果数量。这个插件使用jdbc_fetch_size配置选项进行配置。 在此插件中,默认情况下未设置fetch size大小,因此将使用特定的驱动程序的默认大小。
Usage:edit
Here is an example of setting up the plugin to fetch data from a MySQL database. First, we place the appropriate JDBC driver library in our current path (this can be placed anywhere on your filesystem). In this example, we connect to the mydb database using the user: mysql and wish to input all rows in the songs table that match a specific artist. The following examples demonstrates a possible Logstash configuration for this. The schedule
option in this example will instruct the plugin to execute this input statement on the minute, every minute.
以下是设置插件以从MySQL数据库中获取数据的示例。 首先,我们将相应的JDBC驱动程序库放在我们当前的路径中(这可以放置在文件系统的任何位置)。 在此示例中,我们使用用户mysql连接到mydb数据库,并希望输入歌曲表中与特定艺术家匹配的所有行。 以下示例演示了可能的Logstash配置。 该示例中的schedule选项将指示插件在每分钟的时间内执行此输入语句。
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb" jdbc_user => "mysql"
parameters =>
{
"favorite_artist" => "Beethoven"
}
schedule => "* * * * *"
statement => "SELECT * from songs where artist = :favorite_artist"
}
}
Configuring SQL statement
A sql statement is required for this input. This can be passed-in via a statement option in the form of a string, or read from a file (statement_filepath
). File option is typically used when the SQL statement is large or cumbersome to supply in the config. The file option only supports one SQL statement. The plugin will only accept one of the options. It cannot read a statement from a file as well as from the statement
configuration parameter.
此输入需要sql语句。 这可以通过字符串形式的语句选项或从文件(statement_filepath)读取来传入。 通常在配置文件中提供SQL语句大或繁琐时使用“文件”选项。 该文件选项仅支持一个SQL语句。 该插件将只接受其中一个选项。 它不能从文件以及语句配置参数中读取语句。
Configuring multiple SQL statements
Configuring multiple SQL statements is useful when there is a need to query and ingest data from different database tables or views. It is possible to define separate Logstash configuration files for each statement or to define multiple statements in a single configuration file. When using multiple statements in a single Logstash configuration file, each statement has to be defined as a separate jdbc input (including jdbc driver, connection string and other required parameters).
当需要查询和从不同的数据库表或试图中获取数据时,配置多个SQL语句是很有用的,可以为每个语句定义单独的Logstash配置文件,或者在单个配置文件中定义多个语句。在单个Logstash配置文件中使用多个语句时,必须将每个语句定义为单独的jdbc输入(包括jdbc驱动程序,连接字符串和其他必须的参数)
Please note that if any of the statements use the sql_last_value
parameter (e.g. for ingesting only data changed since last run), each input should define its own last_run_metadata_path
parameter. Failure to do so will result in undesired behaviour, as all inputs will store their state to the same (default) metadata file, effectively overwriting each other’s sql_last_value
请注意,如果任何语句使用了sql_last_value参数(例如获取自上次运行以来更改的数据),则每个输入应该定义其自己的last_run_metadata_path参数。否则会导致不必要的行为,因为所有的输入都将其状态存储到相同的(默认)元数据文件,从而有效地覆盖彼此的sql_last_value
Predefined Parameters
Some parameters are built-in and can be used from within your queries. Here is the list:
一些参数是内置的,可以在您的查询中使用。 这是列表:sql_last_value
The value used to calculate which rows to query. Before any query is run, this is set to Thursday, 1 January 1970, or 0 if use_column_value
is true and tracking_column
is set. It is updated accordingly after subsequent queries are run.
该值用来计算行的值,在执行任何查询前,这个会被设置为1970年1月1日星期四,如果use_column_value为true并且设置了tracking_column,则为0。 在运行后续查询后,它会相应更新。Example:
input {
jdbc {
statement => "SELECT id, mycolumn1, mycolumn2 FROM my_table WHERE id > :sql_last_value"
use_column_value => true
tracking_column => "id"
# ... other configuration bits
}
}
Jdbc Input Configuration Options
This plugin supports the following configuration options plus the Common Options described later.
此插件支持以下配置选项以及稍后描述的常用选项。(具体选项就不列了,根据需求自己查询)
网友评论