Linux环境下安装FLink1.10.0并启动SQL-clie

Linux环境下安装FLink1.10.0并启动SQL-client读取Hive数据

首先去官网下载Flink1.10.0的tgz的包，教程如上篇文章上半部分流程一样，然后配置一下FLINK_HOME/conf/sql-client-defaults.yaml：

catalogs:
   - name: myhive   #自己定个名字就行
     type: hive
     hive-conf-dir: /etc/hive/conf  # hive-site.xml的路径
     hive-version: 1.2.1    # hive版本
 execution:
  # select the implementation responsible for planning table programs
  # possible values are 'blink' (used by default) or 'old'
  planner: blink
  # 'batch' or 'streaming' execution
  type: batch  #这里streaming和batch都行
  # allow 'event-time' or only 'processing-time' in sources
  time-characteristic: event-time
  # interval in ms for emitting periodic watermarks
  periodic-watermarks-interval: 200
  # 'changelog' or 'table' presentation of results
  result-mode: table
  # maximum number of maintained rows in 'table' presentation of results
  max-table-result-rows: 1000000
  # parallelism of the program
  parallelism: 1
  # maximum parallelism
  max-parallelism: 128
  # minimum idle state retention in ms
  min-idle-state-retention: 0
  # maximum idle state retention in ms
  max-idle-state-retention: 0
  # current catalog ('default_catalog' by default)
  current-catalog: myhive
  # current database of the current catalog (default database of the catalog by default)
  current-database: secoo_tmp
  # controls how table programs are restarted in case of a failures
  restart-strategy:
    # strategy type
    # possible values are "fixed-delay", "failure-rate", "none", or "fallback" (default)
    type: fallback

配置/etc/profile文件：

export HADOOP_HOME=/usr/hdp/2.4.0.0-169/hadoop
export YARN_CONF_DIR=/etc/hadoop/conf
export HADOOP_CLASSPATH=`hadoop classpath` #非常重要，不添加 运行flink命令时会报错

在FLink安装目录启动yarn-session.sh:

./bin/yarn-session.sh -n 5 -tm 4096 -s 4 -nm 应用名称 -q 队列名称 -d(这个参数可以保证在我们退出客户端时，任务不被立即杀死，还在yarn上持续运行着)

yarn-session的参数介绍
  -n ： 指定TaskManager的数量；
  -d: 以分离模式运行；
  -id：指定yarn的任务ID；
  -j:Flink jar文件的路径;
  -jm：JobManager容器的内存（默认值：MB）;
  -nl：为YARN应用程序指定YARN节点标签;
  -nm:在YARN上为应用程序设置自定义名称;
  -q:显示可用的YARN资源（内存，内核）;
  -qu:指定YARN队列;
  -s:指定TaskManager中slot的数量;
  -st:以流模式启动Flink;
  -tm:每个TaskManager容器的内存（默认值：MB）;
  -z:命名空间，用于为高可用性模式创建Zookeeper子路径;

在yarn页面查看Flink-session任务：

image

提交程序报错：

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to instantiate java compiler
    ···
Caused by: java.lang.IllegalStateException: Unable to instantiate java compiler
    ···
Caused by: java.lang.ClassCastException: org.codehaus.janino.CompilerFactory cannot be cast to org.codehaus.commons.compiler.ICompilerFactory

解决办法：

<dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>1.2.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.codehaus.janino</groupId>
                    <artifactId>janino</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.codehaus.janino</groupId>
                    <artifactId>commons-compiler</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

紧接着又报了 libfb303-0.9.2.jar缺失：

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to create Hive Metastore client
Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: Failed to create Hive Metastore client
Caused by: java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface

将libfb303-0.9.2.jar复制到flink安装路径lib目录下，接着又来一个错误：

ava.lang.LinkageError: ClassCastException: attempting to castjar:file:/data/flink-1.10.0/flink-alter-price-1.0-SNAPSHOT.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-core-1.9.jar!/javax/ws/rs/ext/RuntimeDelegate.class

原因是 jersey-core-1.9.jar 冲突了，解决：

 <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-hadoop-compatibility_2.11</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.sun.jersey</groupId>
                    <artifactId>jersey-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

以上问题都是把Flink程序的依赖打进jar导致的，直接把依赖jar导进 FLINK_HOME/lib下以上问题基本可以避免。

随即而来是另一个错误：

------------------------------------------------------------
 The program finished with the following exception:

Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 67 more

这个问题足足困扰我三天，痛苦至极，这对于初学者真的不友好，官方文档确实有写，但是很隐蔽，不在Hive集成章节，而是在部署运维里的Hadoop集成里有讲：这里如果官网给的对应Hadoop版本的jar和你的Hadoop正对应就不用下载源码编译了，否则将flink-shaded-hadoop-2-uber源码下载下来，指定对应hadoop版本进行编译，然后打包上传到FLINK_HOME/lib下即可，如果还是不行，看看FLINK_HOME/lib是否缺少flink-hadoop-compatibility_2.11-1.10.0.jar缺少拉进来试试。

注:如果是FLink On Yarn 模式每次修改是最好是杀死session进程，再次重启，方便定位问题。

官方的地址及该问题的说明：

https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/ops/deployment/hadoop.html

Adding Hadoop to /lib
The Flink project releases Hadoop distributions for specific versions, that relocate or exclude several dependencies to reduce the risk of dependency clashes. These can be found in the Additional Components section of the download page. For these versions it is sufficient to download the corresponding Pre-bundled Hadoop component and putting it into the /lib directory of the Flink distribution.

If the used Hadoop version is not listed on the download page (possibly due to being a Vendor-specific version), then it is necessary to build flink-shaded against this version. You can find the source code for this project in the Additional Components section of the download page.

Note If you want to build flink-shaded against a vendor specific Hadoop version, you first have to configure the vendor-specific maven repository in your local maven setup as described here.

Run the following command to build and install flink-shaded against your desired Hadoop version (e.g., for version 2.6.5-custom):

mvn clean install -Dhadoop.version=2.6.5-custom
After this step is complete, put the flink-shaded-hadoop-2-uber jar into the /lib directory of the Flink distribution.

附上我的FLINK_HOME/lib下jar的截图：

image

接下来一个小问题的整理(JAR包齐全的话可忽略)：

java.lang.ClassNotFoundException: org.apache.thrift.TBase
缺少hive-exec的包

启动sql-client：

pcsjob@center1:/data/flink-1.10.0$ bin/sql-client.sh embedded
select * from tmp_flink_test_2 ;

image

结果正确，大功告成。过程中遇到的问题越多越好，它会培养你解决问题的思路，及时没人指导你也不要放弃，多去官网看看，大部分问题都能解决。