序列化战争：主流序列化框架BenchmarkGitHub上有这样一个关于序列化的Benchmark，被好多文章引用。但这个项目考虑到完整性，代码有些复杂。为了个人学习，自己实现了个简单的Benchmark测试类，也算是总结一下当今主流序列化框架的用法。1.序列化的战争按照序列化后的数据格式，主流的序列化框架主要可以分为四大类：JSON、二进制、XML、RPC。从更高层次来说，JSON和XML都可以算作是文本类的，而RPC类因为不只是序列化，框架往往还提供了底层RPC以及跨语言代码生成等基础设施，所以单列作一类。具体说来，本次测试涵盖了以下这些：JSON类非常流行的JacksonGoogle的Gson类JSON的MessagePack阿里的FastJSON二进制类老牌劲旅Hessian(以前很喜欢用的)功能全面而强大的FST后起之秀KryoXML类StAX(Streaming API for XML)Thoughwork的XStreamRPC类Protobuf：这里“偷了点懒”，因为Protobuf和Thrift都要安装、编译，所以这里使用了Protostuff，可以在运行时自动获取对象的Schema信息，省去了额外安装和手动编写协议格式文件的过程(Protostuff真是太好了！)。Thrift、Apache Avro：同上，都需要预编译。Why does Jackson-JSON call BSON the “smile format” of JSON?BSON and Smile are two distinct binary formats. They are related in that they are both based on the logical format of JSON (i.e., key-value objects) but they are distinct in that they write incompatible binary formats (you can neither directly read Smile as BSON nor vice-versa). They also have different incompatible features (e.g., BSON defines a date type, while Smile does not as far as I can tell.) BSON is the binary serialization used by MongoDB for network transfer and disk serialization. Smile is the binary JSON format used by the Jackson project.
com.fasterxml.jackson.corejackson-databind2.5.4com.fasterxml.jackson.modulejackson-module-afterburner2.5.4com.fasterxml.jackson.modulejackson-module-scala_2.102.5.3com.google.code.gsongson2.3.1com.alibabafastjson1.2.6io.fastjsonboon0.33com.fasterxml.jackson.dataformatjackson-dataformat-smile2.5.4org.msgpackmsgpack0.6.12org.mongodbbson3.0.2com.fasterxml.jackson.dataformatjackson-dataformat-yaml2.5.4com.cauchohessian4.0.38de.ruedigermoellerfst2.31com.esotericsoftwarekryo3.0.2com.thoughtworks.xstreamxstream1.4.8com.fasterxmlaalto-xml0.9.11io.protostuffprotostuff-core1.3.5io.protostuffprotostuff-runtime1.3.5org.apache.avroavro1.7.72.Benchmark代码2.1 测试对象用Serializer接口实现表示不同的序列化框架，作为测试对象集合。测试主要关注序列化数据大小、序列化时间消耗、反序列化时间消耗三个指标。publicclassSerializerBenchmark{privatestaticfinalintWARMUP_COUNT =100;privatestaticfinalintTEST_COUNT =1000*1000;/** Column index */privatestaticfinalintCOL_SER_SIZE =0;privatestaticfinalintCOL_SER_COST =1;privatestaticfinalintCOL_DER_COST =2;/** Dictionary for random generation */privatestaticfinalchar[] ALPHA ="abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ".toCharArray();publicstaticvoidmain(String[] args)throwsException {
Serializer[] serializers =
{// ============= JSON ==============newSerializer
() {privateObjectMapper mapper =newObjectMapper();@OverridepublicStringname() {return"Jackson";
}@Overridepublicbyte[]serialize(Person obj)throwsException {returnmapper.writeValueAsBytes(obj);
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {returnmapper.readValue(data, type);
}
},newSerializer() {privateGson gson =newGsonBuilder().create();@OverridepublicStringname() {return"Gson";
}@Overridepublicbyte[]serialize(Person obj) {returngson.toJson(obj).getBytes();
}@OverridepublicPersondeserialize(byte[] data, Class type) {returngson.fromJson(newString(data), type);
}
},newSerializer() {@OverridepublicStringname() {return"FastJSON";
}@Overridepublicbyte[]serialize(Person obj) {returnJSON.toJSONBytes(obj);
}@OverridepublicPersondeserialize(byte[] data, Class type) {returnJSON.parseObject(data, type);
}
},// ============= JSON-like ==============newSerializer() {privateObjectMapper mapper =newObjectMapper(newSmileFactory());@OverridepublicStringname() {return"Jackson-smile";
}@Overridepublicbyte[]serialize(Person obj)throwsException {returnmapper.writeValueAsBytes(obj);
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {returnmapper.readValue(data, type);
}
},newSerializer() {privateObjectMapper mapper =newObjectMapper(newSmileFactory());
{
mapper.registerModule(newAfterburnerModule());
}@OverridepublicStringname() {return"Jackson-smile-afterburner";
}@Overridepublicbyte[]serialize(Person obj)throwsException {returnmapper.writeValueAsBytes(obj);
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {returnmapper.readValue(data, type);
}
},newSerializer() {privateObjectMapper mapper =newObjectMapper(newSmileFactory());
{
mapper.registerModule(newDefaultScalaModule());
}@OverridepublicStringname() {return"Jackson-smile-scala";
}@Overridepublicbyte[]serialize(Person obj)throwsException {returnmapper.writeValueAsBytes(obj);
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {returnmapper.readValue(data, type);
}
},newSerializer() {privateObjectMapper mapper =newObjectMapper(newYAMLFactory());@OverridepublicStringname() {return"Jackson-yaml";
}@Overridepublicbyte[]serialize(Person obj)throwsException {returnmapper.writeValueAsBytes(obj);
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {returnmapper.readValue(data, type);
}
},newSerializer() {privateMessagePack msgpack =newMessagePack();
{
msgpack.register(Person.class);
}@OverridepublicStringname() {return"MessagePack";
}@Overridepublicbyte[]serialize(Person obj)throwsException {returnmsgpack.write(obj);
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {returnmsgpack.read(data, Person.class);
}
},// ============= Binary ==============newSerializer() {privateSchema schema = RuntimeSchema.getSchema(Person.class);privateLinkedBuffer buffer = LinkedBuffer.allocate();@OverridepublicStringname() {return"Protostuff";
}@Overridepublicbyte[]serialize(Person obj) {byte[] data = ProtobufIOUtil.toByteArray(obj, schema, buffer);
buffer.clear();returndata;
}@OverridepublicPersondeserialize(byte[] data, Class type) {
Person obj =newPerson();
ProtobufIOUtil.mergeFrom(data, obj, schema);returnobj;
}
},newSerializer() {@OverridepublicStringname() {return"Hessian";
}@Overridepublicbyte[]serialize(Person obj)throwsException {
ByteArrayOutputStream bytes =newByteArrayOutputStream();
Hessian2Output output =newHessian2Output(bytes);
output.writeObject(obj);
output.close();// flush to avoid EOF errorreturnbytes.toByteArray();
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {
Hessian2Input input =newHessian2Input(newByteArrayInputStream(data));return(Person) input.readObject();
}
},newSerializer() {privateFSTObjectInput input =newFSTObjectInput();privateFSTObjectOutput output =newFSTObjectOutput();@OverridepublicStringname() {return"FST";
}@Overridepublicbyte[]serialize(Person obj)throwsException {
output.resetForReUse();
output.writeObject(obj);returnoutput.getCopyOfWrittenBuffer();
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {
input.resetForReuseUseArray(data);return(Person) input.readObject();
}
},newSerializer() {privateKryo kryo =newKryo();
{
kryo.setReferences(false);
kryo.setRegistrationRequired(true);
kryo.register(Person.class);
}privatebyte[] buffer =newbyte[512];privateOutput output =newOutput(buffer, -1);privateInput input =newInput(buffer);@OverridepublicStringname() {return"Kryo";
}@Overridepublicbyte[]serialize(Person obj) {
output.setBuffer(buffer, -1);// resetkryo.writeObject(output, obj);returnoutput.toBytes();
}@OverridepublicPersondeserialize(byte[] data, Class type) {
input.setBuffer(data);returnkryo.readObject(input, type);
}
},newSerializer() {@OverridepublicStringname() {return"JDK Built-in";
}@Overridepublicbyte[]serialize(Person obj)throwsException {
ByteArrayOutputStream out =newByteArrayOutputStream();newObjectOutputStream(out).writeObject(obj);returnout.toByteArray();
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {return(Person)newObjectInputStream(newByteArrayInputStream(data)).readObject();
}
},// ============= XML ==============newSerializer() {privateXStream xstream =newXStream();@OverridepublicStringname() {return"XStream";
}@Overridepublicbyte[]serialize(Person obj)throwsException {
ByteArrayOutputStream out =newByteArrayOutputStream();
xstream.toXML(obj, out);returnout.toByteArray();
}@OverridepublicPersondeserialize(byte[] data, Class type)throwsException {return(Person) xstream.fromXML(newByteArrayInputStream(data));
}
},
};// Sheetint[] testCase = {10,100,1000};
String[] sheetNames =newString[testCase.length];for(inti =0; i < sheetNames.length; i++) {
sheetNames[i] ="Size="+ testCase[i];
}// RowString[] rowNames =newString[serializers.length];for(inti =0; i < rowNames.length; i++) {
rowNames[i] = serializers[i].name();
}// ColumnString[] colNames =newString[3];
colNames[0] ="Size";
colNames[1] ="Ser";
colNames[2] ="Der";
Reporter reporter =newReporter(sheetNames, rowNames, colNames);for(inti =0; i < testCase.length; i++) {intlength = testCase[i];
System.out.printf("===== Round [%d]: %d =====\n", i, length);for(intj =0; j < serializers.length; j++) {
testSerializer(reporter, length, i, j, serializers[j]);
}
}
System.out.println(reporter.generateFinalReport());
}
...
}2.2 测试Runner每轮测试前都先Warmup并GC，避免JIT和GC对测试的影响。同时，Warmup时检测序列化和反序列化的正确性。privatestaticvoidtestSerializer(Reporter reporter,intlength,intsheet,introw,
Serializer serializer)throwsException {
System.out.println("===== "+ serializer.name() +" =====");// 1.Warm-up and validateSystem.out.println("Pre-warmup & Check correctness...");
Person p1 = newPerson(length);for(inti =0; i < WARMUP_COUNT; i++) {byte[] bytes = serializer.serialize(p1);
Person p2 = serializer.deserialize(bytes, Person.class);if(!p1.equals(p2)) {thrownewIllegalStateException(p1 +" not equals to "+ p2);
}
}intserSize = serializer.serialize(p1).length;
System.out.printf("%s serialization size[%d]\n", serializer.name(), serSize);
reporter.report(sheet, row, COL_SER_SIZE, serSize);
doGc();// 2.SerializationlongstartTime = System.currentTimeMillis();for(inti =0; i < TEST_COUNT; i++) {
serializer.serialize(p1);
}longserCostTime = System.currentTimeMillis() - startTime;
System.out.printf("%s serialization benchmark[%d]\n", serializer.name(), serCostTime);
reporter.report(sheet, row, COL_SER_COST, serCostTime);// Warm up againfor(inti =0; i < WARMUP_COUNT; i++) {byte[] bytes = serializer.serialize(p1);
serializer.deserialize(bytes, Person.class);
}
doGc();// 3.De-Serializationbyte[] bytes = serializer.serialize(p1);
startTime = System.currentTimeMillis();for(inti =0; i < TEST_COUNT; i++) {
serializer.deserialize(bytes, Person.class);
}longderCostTime = System.currentTimeMillis() - startTime;
System.out.printf("%s de-serialization benchmark[%d]\n", serializer.name(), derCostTime);
reporter.report(sheet, row, COL_DER_COST, derCostTime);
System.out.println();
}3.测试报告3.1 报告生成这里“偷了点小懒”，用Apache Common Lang提供的StringUtils中的pad()方法排版。staticclass Reporter {privatefinalString[] sheetNames;privatefinalString[] rowNames;privatefinalString[] colNames;privatefinallong[][][] table;
Reporter(String[] sheetNames,
String[] rowNames,
String[] colNames) {this.sheetNames = sheetNames;this.rowNames = rowNames;this.colNames = colNames;this.table =newlong[sheetNames.length]
[rowNames.length]
[colNames.length];
}publicvoidreport(intsheet,introw,intcol,longval) {
table[sheet][row][col] = val;
}publicStringgenerateFinalReport() {
StringBuilder report =newStringBuilder();for(inti =0; i < table.length; i++) {
report.append(center(sheetNames[i],50,'*'))
.append("\n");// 1.Headerfinalintwidth0 =30;finalintwidth1 =10;
report.append(rightPad("", width0));for(String colName : colNames) {
report.append(rightPad(colName, width1));
}
report.append("\n");// 2.Rowfor(intj =0; j < table[i].length; j++) {
report.append(rightPad(rowNames[j], width0));for(intk =0; k < table[i][j].length; k++) {
report.append(rightPad(
String.valueOf(table[i][j][k]), width1));
}
report.append("\n");
}
report.append("\n");
}returnreport.toString();
}
}3.2 测试结果测试结果可以简单总结如下：Kryo占用空间最小，其次是MessagePack和Protostuff(Protobuf)。Protostuff在不同数据长度下表现都非常出色！JSON以及类JSON框架中，Jackson+Smile格式+Afterburner模块的组合表现最好。XStream出奇地慢，印象中XStream挺快的吧，难道有优化参数没配？*********************Size=10**********************Size Ser DerJackson 39 602 758
Gson 38 1204 1181
FastJSON 38 573 608
Jackson-smile 35 415 465
Jackson-smile-afterburner 35 305 377
Jackson-smile-scala 34 522 590
Jackson-yaml 39 4233 5638
MessagePack 15 891 1075
Protostuff 17 148 130
Hessian 84 2459 1233
FST 73 334 481
Kryo 13 98 117
JDK Built-in 138 1462 4526
XStream 169 6088 13007*********************Size=100*********************Size Ser DerJackson 129 403 565
Gson 128 1056 1248
FastJSON 129 522 571
Jackson-smile 126 426 472
Jackson-smile-afterburner 126 454 371
Jackson-smile-scala 126 452 639
Jackson-yaml 129 5250 5330
MessagePack 108 948 976
Protostuff 107 172 192
Hessian 176 2528 1513
FST 163 288 470
Kryo 105 440 134
JDK Built-in 228 1332 4559
XStream 259 5913 12797********************Size=1000*********************Size Ser DerJackson 1029 1412 1411
Gson 1029 4614 3855
FastJSON 1029 2476 2011
Jackson-smile 1026 1052 1343
Jackson-smile-afterburner 1025 1105 1232
Jackson-smile-scala 1025 1058 1452
Jackson-yaml 1029 18983 13065
MessagePack 1008 2101 2010
Protostuff 1008 1172 838
Hessian 1075 4358 6587
FST 1063 1083 1567
Kryo 1005 2675 921
JDK Built-in 1128 2502 8537
XStream 1158 10633 16981