Flink总结-Parallel配置

Parallel Execution

https://ci.apache.org/projects/flink/flink-docs-master/dev/parallel.html

配置Parallel

Operator Level

final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<String> text = [...] DataStream<Tuple2<String, Integer>> wordCounts = text .flatMap(new LineSplitter()) .keyBy(0) .timeWindow(Time.seconds(5)) .sum(1).setParallelism(5); wordCounts.print(); env.execute("Word Count Example");

Execution Environment Level

final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setParallelism(3);

Client Level

在提交程序的时候设置

./bin/flink run -p 10 ../examples/WordCount-java.jar

System Level

setting the parallelism.default property in ./conf/flink-conf.yaml

坑

Using the parallelism provided by the remote cluster (16). To use another parallelism, set it at the ./bin/flink client.

这个通常表示，集群能够提供的并行度没有达到用户设置的并行度

在运行命令的时候 -yn 4 -ys 4 决定了程序的并行度。

最大并行度= container个数 * 每个container的槽位

在程序中设置的并行度 parallelism 不能大于 最大并行度