美文网首页
使用protobuf实现跨语言序列化 Java和Python实例

使用protobuf实现跨语言序列化 Java和Python实例

作者: TonyJiangWJ | 来源:发表于2019-11-01 15:09 被阅读0次

    使用protobuf实现跨语言序列化 Java和Python实例

    首先下载安装protoc

    • github.releases
    • 对于OS X可以通过brew直接安装 brew install protobuf
    • 安装完可以通过protoc --version查看版本信息

    创建proto文件-带Any类型的版本

    带Any类型的只能导出pb2类型的Python文件，没法导出Python3 的版本，暂时不知道如何解决

    • Any类型的字段可以在java中实现泛型的功能

    • MessageDto.proto

      syntax = "proto3";
import "google/protobuf/any.proto";
message MessageDto {
  string action=1;
  int32 statte=2;
  google.protobuf.Any data=3;
}

    • RpcCmd.proto

      syntax = "proto3";
import "MessageDto.proto";
message RpcCmd {
  MessageDto message=1;
  string randomKey=2;
  string remoteAddressKey=3;
}

    • Point2PointMessage.proto

      syntax = "proto3";
message Point2PointMessage {
  string targetAddressKey;
  string message;
}

    • BytesData.proto

      syntax = "proto3";
message BytesData {
    bytes content=1;
}

    导出相应的对象定义文件

    Python版

    • protoc --python_out=./gen_pb2 RpcCmd.proto MessageDto.proto Point2PointMessage.proto BytesData.proto

    • 生成的文件名为XXX_pb2.py

      Java版

    • protoc --java_out=./gen_java RpcCmd.proto MessageDto.proto Point2PointMessage.proto BytesData.proto

    • 生成的文件名为XXXOuterClass.java

    在Python中使用

    • 首先要导入生成的文件，放到自己喜欢的包下，然后修改导入包的地址，比如RpcCmd_pb2.py中修改

      import com.tony.proto.py2.MessageDto_pb2 as MessageDto__pb2

    • 然后开始使用

      from com.tony.proto.py2 import RpcCmd_pb2, Point2PointMessage_pb2, BytesData_pb2, MessageDto_pb2


def serialize_to_file(file_path):
    p2p_msg = Point2PointMessage_pb2.Point2PointMessage()
    p2p_msg.message = "Hello, p2p from python"
    p2p_msg.targetAddressKey = "/127.0.0.1:38211"

    # bytes_data = BytesData_pb2.BytesData()
    # bytes_data.content = b"Hello, bytes data from python"

    rpc_cmd = RpcCmd_pb2.RpcCmd()
    rpc_cmd.randomKey = "random-key-key-random"
    rpc_cmd.remoteAddressKey = "/127.0.0.1:1234"
    rpc_cmd.message.action = "p2p"
    rpc_cmd.message.state = 100
    rpc_cmd.message.data.Pack(p2p_msg)
    # rpc_cmd.message.data.Pack(bytes_data)

    bytes_write = rpc_cmd.SerializeToString()

    fw = open(file_path, mode="wb")
    fw.write(bytes_write)
    fw.flush()
    fw.close()
    print("write bytes to file:", bytes_write)


def deserialize_from_file(file_path):
    fo = open(file_path, mode="rb")
    bytes_read = fo.read()
    fo.close()
    print("read bytes from file:", bytes_read)
    rpc_cmd = RpcCmd_pb2.RpcCmd()
    rpc_cmd.ParseFromString(bytes_read)
    print(rpc_cmd)

    p2p_msg = Point2PointMessage_pb2.Point2PointMessage()
    # bytes_data = BytesData_pb2.BytesData()
    # rpc_cmd.message.data.Unpack(bytes_data)
    rpc_cmd.message.data.Unpack(p2p_msg)
    
    print("msg_content:", p2p_msg.message)
    print("msg_target:", p2p_msg.targetAddressKey)
    # print("bytes_data:", str(bytes_data.content, 'utf-8'))


if __name__ == "__main__":
    serialize_file_path = "/trans-data-pb2.dat"
    serialize_to_file(serialize_file_path)
    deserialize_from_file(serialize_file_path)

    • 执行结果如下，可以将bytes_data相关的注释取消同时注释掉p2p_msg相关的测试BytesData类型的序列化和反序列化

      write bytes to file: b'\n]\n\x03p2p\x10d\x1aT\n&type.googleapis.com/Point2PointMessage\x12*\n\x10/127.0.0.1:38211\x12\x16Hello, p2p from python\x12\x15random-key-key-random\x1a\x0f/127.0.0.1:1234'
read bytes from file: b'\n]\n\x03p2p\x10d\x1aT\n&type.googleapis.com/Point2PointMessage\x12*\n\x10/127.0.0.1:38211\x12\x16Hello, p2p from python\x12\x15random-key-key-random\x1a\x0f/127.0.0.1:1234'
message {
  action: "p2p"
  state: 100
  data {
    type_url: "type.googleapis.com/Point2PointMessage"
    value: "\n\020/127.0.0.1:38211\022\026Hello, p2p from python"
  }
}
randomKey: "random-key-key-random"
remoteAddressKey: "/127.0.0.1:1234"

msg_content: Hello, p2p from python
msg_target: /127.0.0.1:38211

    在Java中使用

    • 同样导入到喜欢的包下，修改对应的包名即可

      @Slf4j
public class ProtobufSerializeDemo {

    @Test
    public void serializeToFile() throws Exception {
        Point2PointMessageOuterClass.Point2PointMessage.Builder p2pMsgBuilder = Point2PointMessageOuterClass.Point2PointMessage.newBuilder();
        p2pMsgBuilder.setTargetAddressKey("/127.0.0.1:1233");
        p2pMsgBuilder.setMessage("hello from java");

//        BytesDataOuterClass.BytesData.Builder bytesBuilder = BytesDataOuterClass.BytesData.newBuilder();
//        bytesBuilder.setContent(ByteString.copyFrom("bytes data from java".getBytes(StandardCharsets.UTF_8)));


        MessageDtoOuterClass.MessageDto.Builder messageBuilder = MessageDtoOuterClass.MessageDto.newBuilder();
        messageBuilder.setAction("p2p");
        messageBuilder.setState(100);
        messageBuilder.setData(Any.pack(p2pMsgBuilder.build()));
//        messageBuilder.setData(Any.pack(bytesBuilder.build()));


        RpcCmdOuterClass.RpcCmd.Builder builder = RpcCmdOuterClass.RpcCmd.newBuilder();
        builder.setRandomKey("RANDOM_KEY_JAVA");
        builder.setRemoteAddressKey("/127.0.0.1:1234");
        builder.setMessage(messageBuilder.build());

        builder.build().writeTo(new FileOutputStream("java_protobuf.dat"));
    }


    @Test
    public void deserializeFromFile() throws Exception {
        RpcCmdOuterClass.RpcCmd rpcCmd = RpcCmdOuterClass.RpcCmd.parseFrom(new FileInputStream("java_protobuf.dat"));
        Point2PointMessageOuterClass.Point2PointMessage p2pMsg = rpcCmd.getMessage().getData().unpack(Point2PointMessageOuterClass.Point2PointMessage.class);
//        BytesDataOuterClass.BytesData bytesData = rpcCmd.getMessage().getData().unpack(BytesDataOuterClass.BytesData.class);
        log.info("deserialize rpcCmd: \n{}", rpcCmd);
        log.info("deserialize p2pMsg: \n{}", p2pMsg);
//        log.info("deserialize bytesData: \n{}", bytesData);
    }
}

    • 执行结果，可以将bytes_data相关的注释取消同时注释掉p2p_msg相关的测试BytesData类型的序列化和反序列化

      10:22:03.118 [main] INFO com.tony.proto.ProtobufSerializeDemo - deserialize rpcCmd: 
message {
  action: "p2p"
  state: 100
  data {
    type_url: "type.googleapis.com/Point2PointMessage"
    value: "\n\017/127.0.0.1:1233\022\017hello from java"
  }
}
randomKey: "RANDOM_KEY_JAVA"
remoteAddressKey: "/127.0.0.1:1234"

10:22:03.168 [main] INFO com.tony.proto.ProtobufSerializeDemo - deserialize p2pMsg: 
targetAddressKey: "/127.0.0.1:1233"
message: "hello from java"

    然后是Java和Python之间互相序列化和反序列化

    • 只需要修改对应的文件地址就可以进行测试
    Python反序列化Java
    java_serialize_file_path = $path_to_java_serialized$
deserialize_from_file(java_serialize_file_path)

    • 执行结果，这里演示的是BytesData类型的

      read bytes from file: b'\n@\n\x03p2p\x10d\x1a7\n\x1dtype.googleapis.com/BytesData\x12\x16\n\x14bytes data from java\x12\x0fRANDOM_KEY_JAVA\x1a\x0f/127.0.0.1:1234'
message {
  action: "p2p"
  state: 100
  data {
    type_url: "type.googleapis.com/BytesData"
    value: "\n\024bytes data from java"
  }
}
randomKey: "RANDOM_KEY_JAVA"
remoteAddressKey: "/127.0.0.1:1234"

bytes_data: bytes data from java
    Java反序列化Python
        @Test
    public void deserializeFromPythonFile() throws Exception {
        RpcCmdOuterClass.RpcCmd rpcCmd = RpcCmdOuterClass.RpcCmd.parseFrom(new FileInputStream($path_to_python_serialize$));
//        Point2PointMessageOuterClass.Point2PointMessage p2pMsg = rpcCmd.getMessage().getData().unpack(Point2PointMessageOuterClass.Point2PointMessage.class);
        BytesDataOuterClass.BytesData bytesData = rpcCmd.getMessage().getData().unpack(BytesDataOuterClass.BytesData.class);
        log.info("deserialize rpcCmd: \n{}", rpcCmd);
//        log.info("deserialize p2pMsg: \n{}", p2pMsg);
        log.info("deserialize bytesData: \n{}", bytesData);
    }

    • 执行结果，同样是BytesData类型的

      10:33:03.360 [main] INFO com.tony.proto.ProtobufSerializeDemo - deserialize rpcCmd: 
message {
  action: "p2p"
  state: 100
  data {
    type_url: "type.googleapis.com/BytesData"
    value: "\n\035Hello, bytes data from python"
  }
}
randomKey: "random-key-key-random"
remoteAddressKey: "/127.0.0.1:1234"

10:33:03.402 [main] INFO com.tony.proto.ProtobufSerializeDemo - deserialize bytesData: 
content: "Hello, bytes data from python"

    在Java平台，还有个更好用的工具可以不用手写proto文件

    • 这个工具是io.protostuff

    • 通过maven导入依赖

      <dependency>
  <groupId>io.protostuff</groupId>
  <artifactId>protostuff-core</artifactId>
  <version>1.6.0</version>
</dependency>
<dependency>
  <groupId>io.protostuff</groupId>
  <artifactId>protostuff-runtime</artifactId>
  <version>1.6.0</version>
</dependency>
<!-- 用于创建对象 -->
<dependency>
  <groupId>org.objenesis</groupId>
  <artifactId>objenesis</artifactId>
  <version>2.2</version>
</dependency>

    创建序列化工具类

    import io.protostuff.LinkedBuffer;
import io.protostuff.ProtobufIOUtil;
import io.protostuff.Schema;
import io.protostuff.runtime.DefaultIdStrategy;
import io.protostuff.runtime.RuntimeSchema;
import lombok.extern.slf4j.Slf4j;
import org.objenesis.Objenesis;
import org.objenesis.ObjenesisStd;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

/**
 * 基于Protostuff优化版的ProtobufIOUtil实现序列化，理论上可以支持跨语言序列化
 *
 * @author jiangwenjie 2019/10/30
 */
@Slf4j
public class ProtobufSerializer {


    private final static Objenesis OBJENESIS = new ObjenesisStd(true);

    private ProtobufSerializer() {
    }

    private static class SingletonHolder {
        final static ProtobufSerializer INSTANCE = new ProtobufSerializer();
    }

    public static ProtobufSerializer getInstance() {
        return ProtobufSerializer.SingletonHolder.INSTANCE;
    }

    public void serialize(Object obj, OutputStream outputStream) {
        Class clz = obj.getClass();
        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
        try {
            Schema schema = getSchema(clz);
            ProtobufIOUtil.writeTo(outputStream, obj, schema, buffer);
        } catch (IOException e) {
            log.error("序列化对象失败", e);
        } finally {
            buffer.clear();
        }
    }

    public byte[] serialize(Object obj) {
        Class clz = obj.getClass();
        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
        try (ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream()) {
            Schema schema = getSchema(clz);
            ProtobufIOUtil.writeTo(arrayOutputStream, obj, schema, buffer);
            return arrayOutputStream.toByteArray();
        } catch (IOException e) {
            log.error("序列化对象失败", e);
        } finally {
            buffer.clear();
        }
        return new byte[0];
    }

    public <T> T deSerialize(InputStream inputStream, Class<T> clazz) {
        T object = OBJENESIS.newInstance(clazz);
        Schema<T> schema = getSchema(clazz);
        try {
            ProtobufIOUtil.mergeFrom(inputStream, object, schema);
            return object;
        } catch (IOException e) {
            log.error("反序列化对象失败", e);
        }
        return null;
    }

    public <T> T deSerialize(byte[] param, Class<T> clazz) {
        T object = OBJENESIS.newInstance(clazz);
        Schema<T> schema = getSchema(clazz);
        try (ByteArrayInputStream inputStream = new ByteArrayInputStream(param)) {
            ProtobufIOUtil.mergeFrom(inputStream, object, schema);
            return object;
        } catch (IOException e) {
            log.error("反序列化对象失败", e);
        }
        return null;
    }


    private <T> Schema<T> getSchema(Class<T> clz) {
        return RuntimeSchema.createFrom(clz, new DefaultIdStrategy());
    }
}

    创建序列化对象

    import lombok.Data;
import lombok.extern.slf4j.Slf4j;

import java.io.Serializable;

/**
 * @author jiangwenjie 2019/10/22
 */
@Data
@Slf4j
public class RpcCmd implements Serializable {
    private MessageDto message;
    private String randomKey;
    /**
     * 目标地址，不需要序列化传输
     */
    private transient String remoteAddressKey;
}



import com.tony.constants.EnumNettyState;
import com.tony.serializer.impl.ProtobufSerializer;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;

import java.io.Serializable;

/**
 * 消息对象
 *
 * @author jiangwenjie 2019/10/22
 */
@Slf4j
@Data
@ToString
@EqualsAndHashCode
public class MessageDto implements Serializable {

    private String action;

    private int state = 100;

    /**
     * 跨语言使用Protostuff中提供的protobuff序列化传递复杂对象
     */
    private byte[] bytesData;

    private Serializable serialData;

    public <T> T dataOfClazz(Class<T> clazz, boolean isStuff) {
        if (isStuff) {
            return serialDataOfClazz(clazz);
        } else {
            return bytesDataOfClass(clazz);
        }
    }

    public <T extends Serializable> void setData(T object, boolean isStuff) {
        if (isStuff) {
            setSerialData(object);
        } else {
            setBytesData(object);
        }
    }

    @SuppressWarnings("unchecked")
    private <T> T serialDataOfClazz(Class<T> clazz) {
        if (serialData == null) {
            return null;
        }
        if (clazz.isInstance(serialData)) {
            return (T)serialData;
        } else {
            throw new IllegalArgumentException("data is not instance of class:" + clazz.getName());
        }
    }

    private <T> T bytesDataOfClass(Class<T> clazz) {
        if (bytesData == null) {
            return null;
        }
        try {
            return ProtobufSerializer.getInstance().deSerialize(bytesData, clazz);
        } catch (Exception e) {
            log.error("反序列化data对象失败，请确认对象是否为：{} 类型", clazz);
        }
        return null;
    }

    private <T extends Serializable> void setBytesData(T data) {
        this.bytesData = ProtobufSerializer.getInstance().serialize(data);
    }
}


import lombok.Data;

import java.io.Serializable;

/**
 * 点对点通信data对象
 *
 * @author jiangwenjie 2019/10/26
 */
@Data
public class Point2PointMessage implements Serializable {
    private String targetAddressKey;
    private String message;
}

    序列化测试

    import lombok.extern.slf4j.Slf4j;
import org.junit.Test;

import java.io.FileInputStream;
import java.io.FileOutputStream;

/**
 * @author jiangwenjie 2019/11/1
 */
@Slf4j
public class JavaProtostuffSerializeDemo {

    @Test
    public void serializeToFile() throws Exception {
        Point2PointMessage p2pMsg = new Point2PointMessage();
        p2pMsg.setTargetAddressKey("/127.0.0.1:1233");
        p2pMsg.setMessage("message from java");
        MessageDto messageDto = new MessageDto();
        messageDto.setAction("p2p");
        messageDto.setState(100);
        messageDto.setData(p2pMsg, false);

        RpcCmd rpcCmd = new RpcCmd();
        rpcCmd.setMessage(messageDto);
        rpcCmd.setRandomKey("RANDOM_KEY_JAVA");
        rpcCmd.setRemoteAddressKey("/127.0.0.1:1234");
        ProtobufSerializer.getInstance().serialize(rpcCmd, new FileOutputStream("java_proto_simple.dat"));
    }

    @Test
    public void deserializeFromFile() throws Exception {
        RpcCmd rpcCmd = ProtobufSerializer.getInstance().deSerialize(new FileInputStream("java_proto_simple.dat"), RpcCmd.class);
        log.info("deserialize cmd:\n{}", rpcCmd);
        log.info("deserialize p2p msg:\n{}", rpcCmd.getMessage().dataOfClazz(Point2PointMessage.class, false));
    }
}

    测试输出

    11:02:45.646 [main] INFO com.tony.simple.JavaProtostuffSerializeDemo - deserialize cmd:
RpcCmd(message=MessageDto(action=p2p, state=100, bytesData=[10, 15, 47, 49, 50, 55, 46, 48, 46, 48, 46, 49, 58, 49, 50, 51, 51, 18, 17, 109, 101, 115, 115, 97, 103, 101, 32, 102, 114, 111, 109, 32, 106, 97, 118, 97], serialData=null, isFromBuff=false), randomKey=RANDOM_KEY_JAVA, remoteAddressKey=null)
11:02:45.651 [main] INFO com.tony.simple.JavaProtostuffSerializeDemo - deserialize p2p msg:
Point2PointMessage(targetAddressKey=/127.0.0.1:1233, message=message from java)

    MessageDto中的Data 可以泛型化使用

    /**
 * 跨语言使用Protostuff中提供的protobuff序列化传递复杂对象
 */
private byte[] bytesData;
private Serializable serialData;

    • 当序列化和反序列化不需要跨平台使用时，可以直接使用Serializable类型，反之需要用byte数组保存数据，进行二次序列化和反序列化。同时可以在序列化工具类ProtobufSerializer中将ProtobufIOUtil修改为ProtostuffIOUtil

    • 通过setData方法进行响应的操作

      public <T extends Serializable> void setData(T object, boolean isStuff) {
  if (isStuff) {
    setSerialData(object);
  } else {
    setBytesData(object);
  }
}

    跨语言Python中反序列化

    创建proto文件

    • MessageDto.proto

      syntax = "proto3";

message MessageDto {
    string action=1;
    int32 state=2;
    bytes data=3;
}

    • RpcCmd.proto

      syntax = "proto3";
import "MessageDto.proto";

message RpcCmd {
    MessageDto message = 1;
    string randomKey = 2;
    string remoteAddressKey = 3;
}

    • Point2PointMessage.proto

      syntax = "proto3";

message Point2PointMessage {
    bytes java_class = 127;
    string targetAddressKey = 1;
    string message = 2;
}

    • BytesData.proto

      syntax = "proto3";
message BytesData {
    bytes content=1;
}
    导出Python3对象定义文件
    • 此时没有用到Any类型 可以直接导出为Python3的py文件
    • protoc --python3_out=./gen RpcCmd.proto MessageDto.proto Point2PointMessage.proto BytesData.proto
    • 和pb2的区别是序列化和反序列化的方法名称进行了修改
      • pb2中用的是ParseFromString和SerializeToString
      • pb3中修改成了encode_to_bytes和parse_from_bytes
    在Python中使用

    • 同样的放到喜欢的包下，修改对应包名 这里不赘述

      #!/usr/bin/python3
# -*-coding: utf-8 -*-
from com.tony.proto.py3 import RpcCmd, Point2PointMessage, MessageDto, BytesData


def serialize_to_file(file_path):
    p2p_msg = Point2PointMessage.Point2PointMessage()
    p2p_msg.message = "Hello, p2p from python"
    p2p_msg.targetAddressKey = "/127.0.0.1:38211"

    # bytes_data = BytesData.BytesData()
    # bytes_data.content = "bytes data from python"

    rpc_cmd = RpcCmd.RpcCmd()
    rpc_cmd.randomKey = "random-key-key-random"
    rpc_cmd.remoteAddressKey = "/127.0.0.1:1234"
    rpc_cmd.message.action = "p2p"
    rpc_cmd.message.state = 100
    rpc_cmd.message.data = p2p_msg.encode_to_bytes()
    # rpc_cmd.message.data = bytes_data.encode_to_bytes()
    bytes_write = rpc_cmd.encode_to_bytes()

    fw = open(file_path, mode="wb")
    fw.write(bytes_write)
    fw.flush()
    fw.close()
    print("write bytes to file:", bytes_write)


def deserialize_from_file(file_path):
    fo = open(file_path, mode="rb")
    bytes_read = fo.read()
    fo.close()
    print("read bytes from file:", bytes_read)
    rpc_cmd = RpcCmd.RpcCmd()
    rpc_cmd.parse_from_bytes(bytes_read)
    print(rpc_cmd)

    msg_bytes = rpc_cmd.message.data
    print("message bytes", msg_bytes)
    p2p_msg = data_of(rpc_cmd.message, Point2PointMessage.Point2PointMessage)
    # bytes_data = data_of(rpc_cmd.message, BytesData.BytesData)
    print("msg_content:", p2p_msg.message)
    print("msg_target:", p2p_msg.targetAddressKey)
    # print("bytes_data:", bytes_data.content)


def data_of(message: RpcCmd.MessageDto, message_identify):
    content = message_identify()
    content.parse_from_bytes(message.data)
    return content


if __name__ == "__main__":
    serialize_file_path = "./trans-data.dat"
    serialize_to_file(serialize_file_path)
    deserialize_from_file(serialize_file_path)

    • 执行结果如下，同样的可以将bytes_data相关的注释取消同时注释掉p2p_msg相关的测试BytesData类型的序列化和反序列化

      write bytes to file: b'\n3\n\x03p2p\x10d\x1a*\n\x10/127.0.0.1:38211\x12\x16Hello, p2p from python\x12\x15random-key-key-random\x1a\x0f/127.0.0.1:1234'
read bytes from file: b'\n3\n\x03p2p\x10d\x1a*\n\x10/127.0.0.1:38211\x12\x16Hello, p2p from python\x12\x15random-key-key-random\x1a\x0f/127.0.0.1:1234'
<Message(RpcCmd)>
    <MessageField(id=1, optional)>:
        <Message(MessageDto)>            <StringField(id=1, optional)>:                p2p            <Int32Field(id=2, optional)>:                100            <BytesField(id=3, optional)>:                b'\n\x10/127.0.0.1:38211\x12\x16Hello, p2p from python'
    <StringField(id=2, optional)>:
        random-key-key-random
    <StringField(id=3, optional)>:
        /127.0.0.1:1234
message bytes b'\n\x10/127.0.0.1:38211\x12\x16Hello, p2p from python'
msg_content: Hello, p2p from python
msg_target: /127.0.0.1:38211
    Python和Java互转
    • 同样是仅仅修改序列化文件地址即可
    Python反序列化Java
    • java_serialize_file_path = $path_to_java_serialized$
deserialize_from_file(java_serialize_file_path)

    • 执行结果

      read bytes from file: b'\n-\n\x03p2p\x10d\x1a$\n\x0f/127.0.0.1:1233\x12\x11message from java\x12\x0fRANDOM_KEY_JAVA'
<Message(RpcCmd)>
    <MessageField(id=1, optional)>:
        <Message(MessageDto)>            <StringField(id=1, optional)>:                p2p            <Int32Field(id=2, optional)>:                100            <BytesField(id=3, optional)>:                b'\n\x0f/127.0.0.1:1233\x12\x11message from java'
    <StringField(id=2, optional)>:
        RANDOM_KEY_JAVA
message bytes b'\n\x0f/127.0.0.1:1233\x12\x11message from java'
msg_content: message from java
msg_target: /127.0.0.1:1233
    Java反序列化Python
    • @Test
public void deserializeFromPythonFile() throws Exception {
  RpcCmd rpcCmd = ProtobufSerializer.getInstance()
    .deSerialize(new FileInputStream($python_serialize_path$), RpcCmd.class);
  log.info("deserialize cmd:\n{}", rpcCmd);
  log.info("deserialize p2p msg:\n{}", rpcCmd.getMessage().dataOfClazz(Point2PointMessage.class, false));
}

    • 执行结果

      13:15:17.821 [main] INFO com.tony.simple.JavaProtostuffSerializeDemo - deserialize cmd:
RpcCmd(message=MessageDto(action=p2p, state=100, bytesData=[10, 16, 47, 49, 50, 55, 46, 48, 46, 48, 46, 49, 58, 51, 56, 50, 49, 49, 18, 22, 72, 101, 108, 108, 111, 44, 32, 112, 50, 112, 32, 102, 114, 111, 109, 32, 112, 121, 116, 104, 111, 110], serialData=null), randomKey=random-key-key-random, remoteAddressKey=null)
13:15:17.828 [main] INFO com.tony.simple.JavaProtostuffSerializeDemo - deserialize p2p msg:
Point2PointMessage(targetAddressKey=/127.0.0.1:38211, message=Hello, p2p from python)

    io.protostuff使用总结

    • 在java平台可以直接定义普通的POJO而不需要手写proto文件并生成对应的对象文件，仅仅通过其所提供的ProtobufIOUtil或者ProtostuffIOUtil来实现序列化和反序列化即可。
    • 当需要进行跨语言序列化和反序列化时，需要其他语言中编写对应的proto文件并生成对象文件，而在Java中的泛型实例变量则需要进行修改，改成二次序列化的byte数组，方便在Python等语言中进行解析。Java中的序列化也应采用ProtobufIOUtil来实现。此时，Python中可以根据业务类型反序列化成指定的对象，Java中也以该对象来序列化，反过来也是一样的操作。以此来达到的目的是定义MessageDto之后如果需要扩展，不需要修改MessageDto，仅仅需要定义更多的data类型然后赋值给MessageDto.$data。
    • 对比纯protobuf实现的来说，在编码上更加简单，不需要写大量的Any.pack()和Any.unpack()

    原文发布在我的github blog

    相关文章

      网友评论

          本文标题：使用protobuf实现跨语言序列化 Java和Python实例

          本文链接：https://www.haomeiwen.com/subject/tqpiyttx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|使用protobuf实现跨语言序列化 Java和Python实例|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！