Flink 的yarn 两种提交方式
标题第一种
1.yarn-session.sh + flink run(提交任务)
命令：
优缺点：
特点：需要事先申请资源，启动JobManager和TaskManger
优点：不需要每次递交作业申请资源，而是使用已经申请好的资源，从而提高执行效率
缺点：作业执行完成以后，资源不会被释放，因此一直会占用系统资源
应用场景：适合作业递交比较频繁的场景，小作业比较多的场景
/opt/module/flink-1.12.0/bin/yarn-session.sh -n 2 -tm 800 -s 1 -d
参数说明：
申请2个CPU、1600M内存
-n 表示申请2个容器，这里指的就是多少个taskmanager
-tm 表示每个TaskManager的内存大小
-s 表示每个TaskManager的slots数量
-d 表示以后台程序方式运行
2. 执行成功后会在yarn 的界面出现：
在这里插入图片描述
3. 提交一个任务：
/opt/module/flink-1.12.0/bin/flink run /opt/module/flink-1.12.0/examples/batch/WordCount.jar
4.查看执行结果
[root@master01 batch]# /opt/module/flink-1.12.0/bin/flink run /opt/module/flink-1.12.0/examples/batch/WordCount.jar
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
2021-03-07 13:30:59,306 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-root.
2021-03-07 13:30:59,306 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-root.
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
2021-03-07 13:31:01,314 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/opt/module/flink-1.12.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-03-07 13:31:02,644 INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl [] - Timeline service address: http://master02.pxx.com:8188/ws/v1/timeline/
2021-03-07 13:31:02,984 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at master01.pxx.com/192.168.1.151:8050
2021-03-07 13:31:03,320 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-03-07 13:31:03,568 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface master02.pxx.com:39632 of application 'application_1613230903820_0014'.
Job has been submitted with JobID 3b4c527309843dc73b467eb44e733aed
Program execution finished
Job with JobID 3b4c527309843dc73b467eb44e733aed has finished.
Job Runtime: 22853 ms
Accumulator Results:
- 2106e844138b8bd3e937dde396d9d172 (java.util.ArrayList) [170 elements]
(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(fair,1)
(fardels,1)
(flesh,1)
(fly,1)
(for,2)
(fortune,1)
(from,1)
(give,1)
[root@master01 batch]#
表示执行成功
第二种：
1.采用 Per-Job分离模式
特点
特点：每次递交作业都需要申请一次资源
优点：作业运行完成，资源会立刻被释放，不会一直占用系统资源
缺点：每次递交作业都需要申请资源，会影响执行效率，因为申请资源需要消耗时间
应用场景：适合作业比较少的场景、大作业的场景
操作实例：
命令：
/opt/module/flink-1.12.0/bin/flink run -m yarn-cluster -yjm 1024 -ytm 1024 /opt/module/flink-1.12.0/examples/batch/WordCount.jar
参数解释：
-m jobmanager的地址
-yjm 1024 指定jobmanager的内存信息
-ytm 1024 指定taskmanager的内存信息
如图：
在这里插入图片描述
命令行执行结果：
[root@master01 batch]# /opt/module/flink-1.12.0/bin/flink run -m yarn-cluster -yjm 1024 -ytm 1024 /opt/module/flink-1.12.0/examples/batch/WordCount.jar
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
2021-03-07 13:44:56,723 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-root.
2021-03-07 13:44:56,723 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-root.
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
2021-03-07 13:44:57,920 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/opt/module/flink-1.12.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-03-07 13:44:58,957 INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl [] - Timeline service address: http://master02.pxx.com:8188/ws/v1/timeline/
2021-03-07 13:44:59,206 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at master01.pxx.com/192.168.1.151:8050
2021-03-07 13:44:59,530 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-03-07 13:44:59,872 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured JobManager memory is 1024 MB. YARN will allocate 4096 MB to make up an integer multiple of its minimum allocation memory (4096 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 3072 MB may not be used by Flink.
2021-03-07 13:44:59,873 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured TaskManager memory is 1024 MB. YARN will allocate 4096 MB to make up an integer multiple of its minimum allocation memory (4096 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 3072 MB may not be used by Flink.
2021-03-07 13:44:59,873 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=4096, taskManagerMemoryMB=1024, slotsPerTaskManager=2}
2021-03-07 13:45:01,029 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2021-03-07 13:45:07,915 INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl [] - Timeline service address: http://master02.pxx.com:8188/ws/v1/timeline/
2021-03-07 13:45:07,918 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1613230903820_0015
2021-03-07 13:45:08,230 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1613230903820_0015
2021-03-07 13:45:08,231 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
2021-03-07 13:45:08,236 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED
2021-03-07 13:45:26,144 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - YARN application has been deployed successfully.
2021-03-07 13:45:26,148 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface master02.pxx.com:44213 of application 'application_1613230903820_0015'.
Job has been submitted with JobID 2e6cfab4814d80e167ea0b539ea71a7e
Program execution finished
Job with JobID 2e6cfab4814d80e167ea0b539ea71a7e has finished.
Job Runtime: 24345 ms
Accumulator Results:
- 40fb3a5a662876772a8b702eda2f6f80 (java.util.ArrayList) [170 elements]
(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(and,12)
(arms,1)
(arrows,1)
(awry,1)
(ay,1)
(bare,1)
(be,4)
(bear,3)
(bodkin,1)
(bourn,1)
(but,1)
(by,2)
(calamity,1)
(cast,1)
(coil,1)
[root@master01 batch]#
执行成功.
在flink 通过提交yarn 模式的时候 会出现错误
[root@master01 bin]# yarn-session.sh -n 2 -tm 800 -s 1 -d
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
2021-03-07 13:02:22,315 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, master01.pxx.com
2021-03-07 13:02:22,325 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2021-03-07 13:02:22,325 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 2048m
2021-03-07 13:02:22,326 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 2048m
2021-03-07 13:02:22,326 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 2
2021-03-07 13:02:22,326 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2021-03-07 13:02:22,326 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2021-03-07 13:02:22,327 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: web.submit.enable, true
2021-03-07 13:02:22,328 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.archive.fs.dir, hdfs://master01.pxx.com:8020/flink/completed-jobs/
2021-03-07 13:02:22,328 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: historyserver.web.address, master01.pxx.com
2021-03-07 13:02:22,328 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: historyserver.web.port, 8082
2021-03-07 13:02:22,328 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: historyserver.archive.fs.dir, hdfs://master01.pxx.com:8020/flink/completed-jobs/
2021-03-07 13:02:23,178 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-07 13:02:23,432 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to root (auth:SIMPLE)
2021-03-07 13:02:23,456 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-5714068345138708081.conf.
2021-03-07 13:02:23,495 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/opt/module/flink-1.12.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-03-07 13:02:23,575 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Error while running the Flink session.
java.lang.NoClassDefFoundError: com/sun/jersey/core/util/FeaturesAndProperties
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_251]
at java.lang.ClassLoader.defineClass(ClassLoader.java:756) ~[?:1.8.0_251]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_251]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_251]
at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_251]
at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_251]
at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_251]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_251]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_251]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) ~[?:1.8.0_251]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_251]
at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at org.apache.flink.yarn.YarnClusterClientFactory.getClusterDescriptor(YarnClusterClientFactory.java:83) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:63) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:45) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:479) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:730) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_251]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.12.0.jar:1.12.0]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:730) [flink-dist_2.12-1.12.0.jar:1.12.0]
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.core.util.FeaturesAndProperties
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_251]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_251]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) ~[?:1.8.0_251]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_251]
... 26 more
------------------------------------------------------------
The program finished with the following exception:
java.lang.NoClassDefFoundError: com/sun/jersey/core/util/FeaturesAndProperties
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.flink.yarn.YarnClusterClientFactory.getClusterDescriptor(YarnClusterClientFactory.java:83)
at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:63)
at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:45)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:479)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:730)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:730)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.core.util.FeaturesAndProperties
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 26 more
意思是说缺少jar 包： 将这几个包下载下来放如到flink目录中的lib中就可以了
下载地址：
https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api/2.1
jersey-common-2.27.jar
javax.ws.rs-api-2.1.jar
jersey-core-1.19.jar
flink-hadoop-compatibility_2.12-1.12.0.jar
网友评论