目前flink集群上云成为趋势,ck的保存原来on yarn的时候都是保存在hdfs上,现在改为保存在s3上。
flink-conf.yaml 全部的配置如下:
state.backend: rocksdb
state.checkpoints.dir: s3://flink/flink-checkpoints # "s3://<your-bucket>/<endpoint>"
state.savepoints.dir: s3://flink/flink-savepoints
state.backend.incremental: true
s3.access-key: ***************
s3.secret-key: ***************
s3.ssl.enabled: false
s3.path.style.access: true
s3.endpoint: s3-stage.aibee.com
添加flink-s3-fs-presto-1.11.2.jar到flink的安装目录的plugin目录
wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-presto/1.11.2/flink-s3-fs-presto-1.11.2.jar
mkdir ./plugins/s3-fs-presto
cp ./opt/flink-s3-fs-presto-1.11.2.jar ./plugins/s3-fs-presto/
成功
image.png
网友评论