Apache Beam编译&运行Demo

ApacheBeam源码变异

# 调整下build.gradle
// mavenCentral()
maven { url 'http://maven.aliyun.com/nexus/content/groups/public/'}

# 编译spark-runner
 ./gradlew beam-runners-spark:build

gradle命令学习

./gradlew tasks --all|grep spark

# 编译example示例
cd beam/examples/java &&  ../../gradlew build

常见gradle命令学习：

./gradlew -v 版本号，首次运行，没有gradle的要下载的哦。
./gradlew clean 删除HelloWord/app目录下的build文件夹
./gradlew build 检查依赖并编译打包
./gradlew assembleDebug 编译并打Debug包
./gradlew assembleRelease 编译并打Release的包
./gradlew installRelease Release模式打包并安装
./gradlew uninstallRelease 卸载Release模式包

基于Spark环境运行Beam程序

• 在Spark的环境中添加beam相关依赖Jar包：

bash-3.2$ ls jars/|grep beam
beam-model-pipeline-2.9.0-SNAPSHOT.jar
beam-runners-core-construction-java-2.9.0-SNAPSHOT.jar
beam-runners-core-java-2.9.0-SNAPSHOT.jar
beam-runners-spark-2.9.0-SNAPSHOT.jar
beam-sdks-java-core-2.9.0-SNAPSHOT.jar
beam-sdks-java-io-hadoop-file-system-2.9.0-SNAPSHOT-unshaded.jar
beam-vendor-grpc-v1_13_1-2.9.0-SNAPSHOT.jar

# 同时需要注意，beam依赖guava-20.0.jar的版本，而spark 2.3.0的默认版本为16.0，所以需要升级guava版本到20.0。

• 基于Spark环境运行

# 基于local模式运行(测试验证使用)
bin/spark-submit --class org.apache.beam.examples.WordCount --master local[1] ./beam-examples-java-2.9.0-SNAPSHOT.jar  --runner=SparkRunner --inputFile=hdfs:///tmp/README.txt --output=hdfs:///tmp/beam.txt
# 基于yarn模式运行
bin/spark-submit --class org.apache.beam.examples.WordCount --master yarn ./beam-examples-java-2.9.0-SNAPSHOT.jar  --runner=SparkRunner --inputFile=hdfs:///tmp/README.txt --output=hdfs:///tmp/beam.txt

TODO: 基于Flink运行Beam程序

参考：

• https://blog.csdn.net/u012063409/article/details/54572900
• https://blog.csdn.net/dream_an/article/details/56277784
• https://blog.csdn.net/chenhaifeng2016/article/details/72519343
• https://beam.apache.org/get-started/wordcount-example/