11-flink如何正确分流

作者: 蜗牛写java | 来源:发表于2019-11-24 22:58 被阅读0次

11-flink如何正确分流
教育分流
11-flink读写MySQL
分流
分流
分流
分流
分流
分流
分流

11-flink如何正确分流

分流方式

filter分流
split分流 (只能一次分流，分流后的流不能继续分流)
side output分流 (推荐使用)

场景

flink-分流场景.png

输入数据：

{"key":"001","type":"1","data":"data1"}
{"key":"001","type":"11","data":"data11"}
{"key":"001","type":"12","data":"data12"}
{"key":"002","type":"2","data":"data2"}
{"key":"002","type":"21","data":"data21"}
{"key":"002","type":"22","data":"data22"}

Operator中处理：转为jsonObject处理,并在jsonObject对象中添加流标志；

pom中需要引入fastjson (fastjson使用教程);

<dependency>
  <groupId>com.alibaba</groupId>
  <artifactId>fastjson</artifactId>
  <version>1.2.60</version>
</dependency>

输出结果为：

11> {"data":"data1","stream":"stream1","type":"1","key":"001"}
12> {"data":"data11","stream":"stream1","type":"11","key":"001"}
12> {"data":"data11","stream":"stream11","type":"11","key":"001"}
1> {"data":"data12","stream":"stream1","type":"12","key":"001"}
1> {"data":"data12","stream":"stream12","type":"12","key":"001"}
2> {"data":"data2","stream":"stream2","type":"2","key":"002"}
3> {"data":"data21","stream":"stream2","type":"21","key":"002"}
3> {"data":"data21","stream":"stream21","type":"21","key":"002"}
4> {"data":"data22","stream":"stream2","type":"22","key":"002"}
4> {"data":"data22","stream":"stream22","type":"22","key":"002"}

发送数据客户端

nc程序发送(nc程序实现)

Fliter分流

代码实现：

public class FliterStreamTest {

    public static final String TYPE = "type";

    public static void main(String[] args) throws Exception {

        //获取执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        final ParameterTool params = ParameterTool.fromArgs(args);
        String hostName = params.get("hostname", "localhost");
        int port = params.getInt("port", 9000);

        DataStream<String> sourceStream = env.socketTextStream(hostName, port, "\n");

        SingleOutputStreamOperator<JSONObject> jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));

        DataStream<JSONObject> stream1 = jsonObjectStream
                .filter(e -> e.get(TYPE).equals("1") || e.get(TYPE).equals("11") || e.get(TYPE).equals("12"));
        DataStream<JSONObject> stream11 = stream1.filter(e -> e.get(TYPE).equals("11"));
        DataStream<JSONObject> stream12 = stream1.filter(e -> e.get(TYPE).equals("12"));

        DataStream<JSONObject> stream2 = jsonObjectStream
                .filter(e -> e.get(TYPE).equals("2") || e.get(TYPE).equals("21") || e.get(TYPE).equals("22"));
        DataStream<JSONObject> stream21 = stream2.filter(e -> e.get(TYPE).equals("21"));
        DataStream<JSONObject> stream22 = stream2.filter(e -> e.get(TYPE).equals("22"));

        //数据去向
        stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
        stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
        stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
        stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
        stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
        stream22.map(e -> {e.put("stream", "stream22");return e;}).print();

        env.execute("SocketStreamTest");
    }
}

输出结果(与正确结果一直)：

11> {"data":"data1","stream":"stream1","type":"1","key":"001"}
12> {"data":"data11","stream":"stream1","type":"11","key":"001"}
12> {"data":"data11","stream":"stream11","type":"11","key":"001"}
1> {"data":"data12","stream":"stream1","type":"12","key":"001"}
1> {"data":"data12","stream":"stream12","type":"12","key":"001"}
2> {"data":"data2","stream":"stream2","type":"2","key":"002"}
3> {"data":"data21","stream":"stream2","type":"21","key":"002"}
3> {"data":"data21","stream":"stream21","type":"21","key":"002"}
4> {"data":"data22","stream":"stream2","type":"22","key":"002"}
4> {"data":"data22","stream":"stream22","type":"22","key":"002"}

Split分流

代码实现：

public class SplitStreamTest {

    public static final String TYPE = "type";

    public static void main(String[] args) throws Exception {

        //获取执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        final ParameterTool params = ParameterTool.fromArgs(args);
        String hostName = params.get("hostname", "localhost");
        int port = params.getInt("port", 9000);

        DataStream<String> sourceStream = env.socketTextStream(hostName, port, "\n");

        SingleOutputStreamOperator<JSONObject> jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));

        //flink split流已被标记为过期
        SplitStream<JSONObject> splitStream = jsonObjectStream.split(new OutputSelector<JSONObject>() {
            @Override
            public Iterable<String> select(JSONObject jsonObject) {
                List<String> tags = new ArrayList<>();
                String type = jsonObject.get("type").toString();
                if (type.equals("1") || type.equals("11") || type.equals("12")) {
                    tags.add("stream1");
                } else if (type.equals("2") || type.equals("21") || type.equals("22")) {
                    tags.add("stream2");
                }
                return tags;
            }
        });

        DataStream<JSONObject> stream1 = splitStream.select("stream1");

        DataStream<JSONObject> stream2 = splitStream.select("stream2");

        SplitStream<JSONObject> splitStream1 = stream1.split(new OutputSelector<JSONObject>() {
            @Override
            public Iterable<String> select(JSONObject jsonObject) {
                List<String> tags = new ArrayList<>();
                String type = jsonObject.get("type").toString();
                if (type.equals("11")) {
                    tags.add("stream11");
                } else if (type.equals("12")) {
                    tags.add("stream12");
                }
                return tags;
            }
        });

        SplitStream<JSONObject> splitStream2 = stream1.split(new OutputSelector<JSONObject>() {
            @Override
            public Iterable<String> select(JSONObject jsonObject) {
                List<String> tags = new ArrayList<>();
                String type = jsonObject.get("type").toString();
                if (type.equals("21")) {
                    tags.add("stream21");
                } else if (type.equals("22")) {
                    tags.add("stream22");
                }
                return tags;
            }
        });

        DataStream<JSONObject> stream11 = splitStream1.select("stream11");
        DataStream<JSONObject> stream12 = splitStream1.select("stream12");
        DataStream<JSONObject> stream21 = splitStream2.select("stream21");
        DataStream<JSONObject> stream22 = splitStream2.select("stream22");

        //数据去向
        stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
        stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
        stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
        stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
        stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
        stream22.map(e -> {e.put("stream", "stream22");return e;}).print();

        env.execute("SocketStreamTest");
    }

}

输出结果：

Flink 1.9以前可以输出，但是第二次split将会失效；Flink 1.9 如果二次split，直接报错并提示使用side-outputs

Exception in thread "main" java.lang.IllegalStateException: Consecutive multiple splits are not supported. Splits are deprecated. Please use side-outputs.

Side-outputs分流(推荐使用)

步骤：

定义 OutputTag
process 算子中收集数据
1. ProcessFunction
2. KeyedProcessFunction
3. CoProcessFunction
4. ProcessWindowFunction
5. ProcessAllWindowFunction

代码实现：

public class SideOutputTest {

    public static final String TYPE = "type";

    public static void main(String[] args) throws Exception{
        //获取执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        final ParameterTool params = ParameterTool.fromArgs(args);
        String hostName = params.get("hostname", "localhost");
        int port = params.getInt("port", 9000);

        DataStream<String> sourceStream = env.socketTextStream(hostName, port, "\n");

        SingleOutputStreamOperator<JSONObject> jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));

        //定义OutputTag
        OutputTag<JSONObject> outputTag1 = new OutputTag<JSONObject>("stream1") {};
        OutputTag<JSONObject> outputTag2 = new OutputTag<JSONObject>("stream2") {};

        SingleOutputStreamOperator<JSONObject> outputStream = jsonObjectStream.process(new ProcessFunction<JSONObject, JSONObject>() {

            @Override
            public void processElement(JSONObject jsonObject, Context context, Collector<JSONObject> collector)
                    throws Exception {
                String type = jsonObject.getString(TYPE);
                if (type.equals("1") || type.equals("11") || type.equals("12")) {
                    context.output(outputTag1, jsonObject);
                } else if (type.equals("2") || type.equals("21") || type.equals("22")) {
                    context.output(outputTag2, jsonObject);
                }
            }
        });

        DataStream<JSONObject> stream1 = outputStream.getSideOutput(outputTag1);
        DataStream<JSONObject> stream2 = outputStream.getSideOutput(outputTag2);

        //定义OutputTag
        OutputTag<JSONObject> outputTag11 = new OutputTag<JSONObject>("stream11") {};
        OutputTag<JSONObject> outputTag12 = new OutputTag<JSONObject>("stream12") {};

        SingleOutputStreamOperator<JSONObject> outputStream1 = stream1.process(new ProcessFunction<JSONObject, JSONObject>() {

            @Override
            public void processElement(JSONObject jsonObject, Context context, Collector<JSONObject> collector)
                    throws Exception {
                String type = jsonObject.getString(TYPE);
                if (type.equals("11")) {
                    context.output(outputTag11, jsonObject);
                } else if (type.equals("12")) {
                    context.output(outputTag12, jsonObject);
                }
            }
        });

        DataStream<JSONObject> stream11 = outputStream1.getSideOutput(outputTag11);
        DataStream<JSONObject> stream12 = outputStream1.getSideOutput(outputTag12);

        //定义OutputTag
        OutputTag<JSONObject> outputTag21 = new OutputTag<JSONObject>("stream21") {};
        OutputTag<JSONObject> outputTag22 = new OutputTag<JSONObject>("stream22") {};

        SingleOutputStreamOperator<JSONObject> outputStream2 = stream2.process(new ProcessFunction<JSONObject, JSONObject>() {

            @Override
            public void processElement(JSONObject jsonObject, Context context, Collector<JSONObject> collector)
                    throws Exception {
                String type = jsonObject.getString(TYPE);
                if (type.equals("21")) {
                    context.output(outputTag21, jsonObject);
                } else if (type.equals("22")) {
                    context.output(outputTag22, jsonObject);
                }
            }
        });

        DataStream<JSONObject> stream21 = outputStream2.getSideOutput(outputTag21);
        DataStream<JSONObject> stream22 = outputStream2.getSideOutput(outputTag22);

        //数据去向
        stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
        stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
        stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
        stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
        stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
        stream22.map(e -> {e.put("stream", "stream22");return e;}).print();

        env.execute("SocketStreamTest");
    }
}

输出结果(与结果一致)：

3> {"data":"data1","stream":"stream1","type":"1","key":"001"}
4> {"data":"data11","stream":"stream11","type":"11","key":"001"}
4> {"data":"data11","stream":"stream1","type":"11","key":"001"}
5> {"data":"data12","stream":"stream12","type":"12","key":"001"}
5> {"data":"data12","stream":"stream1","type":"12","key":"001"}
6> {"data":"data2","stream":"stream2","type":"2","key":"002"}
7> {"data":"data21","stream":"stream21","type":"21","key":"002"}
7> {"data":"data21","stream":"stream2","type":"21","key":"002"}
8> {"data":"data22","stream":"stream22","type":"22","key":"002"}
8> {"data":"data22","stream":"stream2","type":"22","key":"002"}

11-flink如何正确分流
11-flink如何正确分流分流方式 filter分流 split分流 (只能一次分流，分流后的流不能继续分流)...
教育分流
在讲述我们的教育分流前先来了解下走在我们国家前几十年的新加坡分流，看看新加坡是如何执行这种分流制度的，以及分流的制...
11-flink读写MySQL
一、读MySQL 1、通过JDBC方式定义MySQLDataSource类 1.1首先加入JDBC依赖1.2定义J...
分流
数据今时已成灾，剥皮抽筋无骨骸。惟待江流入海日，轻舟一叶任风缘！
分流
鱼儿浅游水儿轻流你我流失爱的理由船在港口就要远走眼泪纷飞不能停留你放了手爱已远走徘徊...
分流
走出小山回归大城市的时候，既有新鲜感也有恐惧。山里的三年就像是辟谷的修行，严苛和单调就像是一层保护罩，让人在罩子里...
分流
这几天经常在想一个问题，起因是和朋友之间聊读书的事。有一个很真实的状态，现在读书的人越来越少，为什么会这样？这种...
分流
老生常谈的阶级分层、流动和固化。如果这些都是以不同的人所处于的不同阶级为划分的标准的话，那么对于个人来说个体的主...
分流
我该分去哪里角落里的盆栽笑的灿烂距离光不过一米之隔偏偏愤怒的踩烂那支纸船点燃起香烟闭着眼摸索正确或错误的...
分流
女儿的一个同学中途转向学技术去了，她很是费解，不懂，一系列的为什么…… 每每她听到有关她同学的消息后都是很难理解的...

11-flink如何正确分流

11-flink如何正确分流

分流方式

场景

发送数据客户端

Fliter分流

Split分流

Side-outputs分流(推荐使用)

相关文章

11-flink如何正确分流

教育分流

11-flink读写MySQL

分流

分流

分流

分流

分流

分流

分流

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读