Notes on the Object Serialization Stream Protocol

0. Preface

This article mainly translates Object Serialization Stream Protocol and then summarizes the serialization format.

1. Overview

The stream format meets the following design goals:

• A compact structure that supports efficient reading;
• Allows the stream's structure and format to be used without detailed knowledge of its contents or invoking code from any class.
• Only the stream may access and manipulate the data;

2. Stream Elements

This defines the basic structure required for objects in a stream. Every property of an object must be represented, including its class and the fields of that class. These values are written and later read by class-specific methods. The representation of objects in a stream can be described using a grammar.null objects, new objects, classes, arrays, stringsand back-references to any object already present in the stream have special representations. Every object written to the stream is assigned aHandle, used to refer back to that object. Starting at0x7E0000are allocated sequentially beginning atHandle. When the stream is reset, handle allocation0x7E0000restarts at.

Class objectis represented by the following object:

• ItsObjectStreamClassobject

  ​ Non-dynamic proxyclass'sObjectStreamClassAn object is represented by the following expression:

• SUID of the compatible class

• A set of flags indicating class properties, such as whether the class defines writeObject method, and whether the class is serializable, externalizable, or an enum type

• Number of serializable fields

• By default, field types for class-field arrays and object fields are included as strings and must use the field-descriptor format specified by the Java Virtual Machine Specification (for example,Ljava/lang/Object;）

• is composed ofannotateClassOptional block data written by the method (Data-Block) record or object

• of its supertype ObjectStreamClass(null if the superclass is not serializable)

dynamic proxy class'sObjectStreamClassAn object is represented by the following expression:

• Number of interfaces implemented by the dynamic proxy class
• Names of all interfaces implemented by the dynamic proxy class; these interfaces are obtained by calling Class'sgetInterfacesmethod's return value, sorted and listed;
• Optional block data (Data-Block) record or byAnnotationProxyClassObject written by the method
• corresponding to the superclassjava.lang.reflect.ProxyofObjectStreamClass

A string object is represented by length information followed by MUTF-8-encoded string content. MUTF-8 is the encoding used by the Java Virtual Machine andJava.io.DataInputandDataOutputuses the same encoding as the interface; when representingSupplementary characterandNull characterdiffers from standard UTF-8 in several respects. The form of the length field depends on the length of the string's MUTF-8 encoding. If it is less than 65,536 bytes, the length is written as a two-byte unsigned 16-bit integer. Beginning with Java 2 Platform, Standard Edition v1.3, if the MUTF-8 encoding is 65,536 bytes or longer, its length is represented as an eight-byte signed 64-bit integer. A type code before the string in the serialization stream identifies the format used to write it.

arrayis represented by:

• TheirObjectStreamClassobject
• number of elements.
• value order. The value type is implicit in the array type. For example, values in a byte array have type byte.

Enum constantis represented by the following expression:

• of the constant's underlying enum type ObjectStreamClass object
• Name string of the constant

in the streamNew object（New objects) is represented by:

• Derived-class information for every object class;
• Data for every serializable class of an object is written beginning with its highest superclass. The information for each class in the stream includes: -- Information for the class's serializable fields; -- If the class containswriteObject/readObjectmethod, data may be written throughwriteObjectOptional object or primitive-type block data written by the method (Data-Block) record, followed by use ofendDataBlockcode;

All primitive data written by the class is buffered and wrapped in block-data records, regardless of whether the data is written inside writeObject whether written to the stream inside the method or read from writeObject written directly to the stream outside the method. This data can only be read through the correspondingreadObjectmethod or read directly from the stream.writeObjectObjects written by the method terminate any preceding block-data record and are written, as appropriate, as an ordinary object, null reference, or back-reference. Block-data records permit optional data to be discarded during error recovery. When invoked from within a class, the stream can discard data or objects up toendBlockData。

3. Stream Protocol Versions

It is necessary toJDK1.2changes the serialization stream format in, which is compatible withJDK1.1minor versions are not backward-compatible. To support applications that require backward compatibility, Oracle added a feature that specifies which protocol version to use when writing a serialization stream.ObjectOutputStreaminuseProtocolVersionaccepts an argument selecting the serializable byte-stream protocol version.

The stream protocol versions are as follows:

• ObjectStreamConstants.PROTOCOL_VERSION_1: indicates the initial stream format.
• ObjectStreamConstants.PROTOCOL_VERSION_2: indicates the new external-data format. Primitive data is written in block-data mode and terminated byTC_ENDBLOCKDATAterminated.

Block-data boundaries are standardized. Primitive data written in block-data mode is normalized into blocks no larger than 1,024 bytes. This makes the serialized-data format in the stream more regular, and the change is fully forward- and backward-compatible.

JDK1.2By default, writesPROTOCOL_VERSION_2 JDK1.1By default, writesPROTOCOL_VERSION_1 JDK1.1.7and later versions can read both versions JDK1.1.7Earlier versions can read onlyPROTOCOL_VERSION_1

4. Stream-Format Grammar

The following table contains the stream-format grammar. Nonterminal symbols appear in italics, while terminal symbols use a fixed-width font. A nonterminal definition is followed by a：. Each following line of the definition contains one or more alternatives. The table below describes the symbols:

Note that the symbol (utf) specifies a string written with a two-byte length, while (long utf) specifies one written with an eight-byte length.

a. Grammar Rules

A serialization stream may be represented by any stream that satisfies these rules. The official documentation classifies field values without explaining each one; the detailed meanings here come from the referenced serialization draft.

stream: magic version contents

　　The complete data-stream format is divided into three parts: magic is the magic numberSTREAM_MAGICmarker; version identifies the serialization versionSTREAM_VERSION，contentsrepresents the content of the final generated sequence;

contents: content contents content

　　This part represents the content of the generated binary sequence. It may be a single content item [content] or a collection of multiple items [contents].

content: object blockdata

　　An independent content item [content] in the binary sequence may contain object-definition data [object] or block-format data [blockdata]. The grammar above also permits blockdata to precede object data.

object: newObject newClass newArray newString newEnum newClassDesc prevObject nullReference exception == TC_RESET==

This part represents byte-stream data contained in the object. Its elements have no ordering relationship; they merely indicate which marker-defined data may be present in the object.newObjectrepresents a new object type, newClassrepresents an object of type Class,newArrayrepresents an array object,newStringrepresents a string object,newEnumrepresents an enum constant,newClassDescrepresents the object's class descriptor,preObjectrepresents an object that appeared earlier,nullReferencerepresents a null reference,exceptionrepresents an exception object,TC_RESETis the reset marker [fixed value].

newClass: ==TC_CLASS== classDesc newHandle

　　This part represents a new object of type Class.TC_CLASSindicates a type marker,classDescrepresents class descriptor information,newHandlerepresents a new reference;

classDesc: newClassDesc nullReference (ClassDesc)prevObject // Must be an object of type ClassDesc This part represents an object's class descriptor,newClassDescindicates the appearance of a new class descriptor,nullReferencerepresents a null reference,prevObjectrepresents an object that appeared earlier;

superClassDesc: classDesc

　　This part represents the parent-class descriptor. Its content is aclassDesc, namely the class descriptor shown above;

newClassDesc: ==TC_CLASSDESC== className serialVersionUID newHandle classDescInfo ==TC_PROXYCLASSDESC== newHandle proxyClassDescInfo

This section demonstrates the two forms of class-descriptor information: ordinary class descriptors and dynamic-proxy class descriptors.clsssNamerepresents the class name,serialVersionUIDrepresents the defined in this classserialVersionUIDcorresponding value,newHandlerepresents a new reference,classDescInforepresents information about the class descriptor itself,proxyClassDescInforepresents information about the dynamic-proxy class descriptor itself;

classDescInfo: classDescFlags fields classAnnotation superClassDesc

This part contains detailed class descriptor information,classDescFlagsis the class-descriptor marker,fieldsrepresents descriptor information for all fields in the class,classAnnotationrepresents class-relatedAnnotationdescriptor information,superClassDescrepresents descriptor information for the parent class.

className: (utf)

　　Fully qualified class name: the binary sequence of a UTF-8 string describing the current object's class name;

serialVersionUID: (long)

Corresponds to a field in the class definitionserialVersionUIDinformation;

classDescFlags: (byte) // Defined under terminal symbols and constants

Class-descriptor marker: one byte used to define terminal symbols and constants;

proxyClassDescInfo: (int) proxyInterfaceName[count] classAnnotation superClassDesc

　 Descriptor information for a dynamic proxy class,<count>represents the total number of interfaces implemented by this dynamic proxy class; its type isinttype.proxyInterfaceName[count]represents information for all interfaces implemented by the current dynamic proxy class,classAnnotationrepresents the corresponding for this dynamic proxy classAnnotationdescriptor information,superClassDescrepresents descriptor information for the current dynamic proxy class's parent class,

proxyInterfaceName: (utf)

Name of a proxy interface implemented by the dynamic proxy class, stored as the binary sequence of a UTF-8 string;

fields: (short) fieldDesc[count]

　　<count>represents the total number of fields [member properties] in the class; its data type isshorttype.fieldDesc[count]represents detailed descriptor information for every field in a class; the number of fields matches the preceding count;

fieldDesc: primitiveDesc

​ objectDesc

　　This marker represents field descriptor information, which includestwosection information,primitiveDescrepresents descriptor information for primitive-type data,objectDescrepresents descriptor information for object-type data;

primitiveDesc: prim_typecode fieldName

　　Descriptor information for a primitive-type field,prim_typecoderepresents the field-type identifier, which indicates the current field's type,fieldNamerepresents the field name, stored as the binary sequence of the field-name string;

objectDesc: obj_typecode fieldName className1

Descriptor information for an object-type field,obj_typecoderepresents a field-type identifier describing the class information corresponding to an object field,fieldNamerepresents the field name, stored as the binary sequence of the field-name string,className1represents the type signature of this member field;

fieldName: (utf)

Field-name string, represented as a UTF-8-encoded binary sequence

className1: (String)object // String containing the field type, in field-descriptor format

　　The fully qualified name of the object's class, represented as descriptor information for a String object;

classAnnotation: endBlockData contents endBlockData // Content written by annotateClass

Within the class to which this object belongs, theAnnotationdescriptor information,endBlockDatais a block that stores object data [Data-Block]'sEnd marker, which is the terminator,contentsrepresents a collection of multiple content items [contents] in this class;

prim_typecode: B // byte C // char D // double F // float I // integer J // long S // short Z // boolean

　　Type identifier for a primitive field. It identifies the primitive data type; the meaning of each code appears in the comments of the definition.

obj_typecode: [ // array L // object

　　Type identifier for an object-type field. It identifies the object's type; the meaning of each code appears in the comments of the definition.

newArray: ==TC_ARRAY== classDesc newHandle (int)<size> values[size]

Creates a descriptor for a new array.TC_ARRAYindicates that the following sequence is an array and marks the beginning of the array sequence,classDescis the class descriptor for the current array,newHandlerepresents a reference to the current array object,<size>represents the array length, stored as an int,values[size]represents the value content of every element in the current array;

newObject: ==TC_OBJECT== classDesc newHandle classdata[] // Data for each class

Creates descriptor information for a new object.TC_OBJECTindicates that the following sequence is a new object and marks the start of the object,classDescis the class descriptor of the current object,newHandlerepresents a reference to the current object,classdata[]Class-related data for every Class corresponding to this object;

classdata: nowrclass // SC_SERIALIZABLE & classDescFlag && !(SC_WRITE_METHOD & classDescFlags) wrclass objectAnnotation // SC_SERIALIZABLE &classDescFlag&&SC_WRITE_METHOD&classDescFlags externalContents // SC_EXTERNALIZABLE & classDescFlag && !(SC_BLOCKDATA & classDescFlags objectAnnotation // SC_EXTERNALIZABLE & classDescFlag&& SC_BLOCKDATA & classDescFlags

This part describes all content in the class data. The following sections cover the variousDifferentDescription of class data;

nowrclass: values // Fields in class-descriptor order

　　Values of all serializable fields in a class, ordered as defined by the class descriptor;

wrclass: nowrclass

　　This data has the same content as the precedingnowrclasshas the same contents; it represents the values of a class's serializable fields.

objectAnnotation: endBlockData contents endBlockData // Written bywriteObjectorwriteExternal PROTOCOL_VERSION_2content written by.

　　The content of this data is the same asclassAnnotationhas the same data structure; it represents theAnnotationdescriptor information,endBlockDatais a block that stores object data [Data-Block]'sEnd marker, which is the terminator,contentsrepresents a collection of multiple content items [contents] in this class;

blockdata: blockdatashort blockdatalong

　　In Java serialization, block data is stored intwotwo forms: the default block format with a short length, and a large-capacity block format with an int length;

blockdatashort: ==TC_BLOCKDATA== (unsigned byte)<size> (byte)[size]

Describes the default block structure with a short length;

blockdatalong: == TC_BLOCKDATALONG== (int)<size> (byte)[size]

　　describes the structure of a data block whose length has type int;

endBlockData : ==TC_ENDBLOCKDATA==

　　represents the end-of-block-data marker. It normally indicates that the current data block has ended or that this object type's descriptor has beenEnd;

externalContent: // Can only be parsed by readExternal ( bytes) // Primitive data object

　　This section describes externalized content,(bytes)data can only bereadExternalmethod, and generally contains primitive-type data,objectrepresents an object data type;

externalContents: // InPROTOCOL_VERSION_1is composed ofwriteExternalexternal content written by. externalContent

​ externalContents externalContent

　　This part is a collection of the externalized content described above. In general, this sectioncontains onlyusingwriteExternalmethod inPROTOCOL_VERSION_1version writes its data to the byte stream;

newString: ==TC_STRING== newHandle (utf) ==TC_LONGSTRING== newHandle (long-utf)

　　represents string data, and string data likewise hastwotypes:STRINGandLONGSTRING；

newEnum: ==TC_ENUM== classDesc newHandle enumConstantName

Represents data of an Enum type.TC_ENUMis the enum-type marker, indicating that the following sequence is an enum,classDescis a class descriptor for an enum type,newHandleis the reference for this enum object,enumConstantNamevalue is obtained by calling the enum type'sname()string literal corresponding to the enum value returned by the method;

enumConstantName: (String)object The literal string name of the enum constant, itself a string;

prevObject ==TC_REFERENCE== (int)handle

Represents an object reference to an object already written to the byte stream.TC_REFERENCEis the reference marker, indicating that the following data type is a Java reference type.

nullReference ==TC_NULL==

　　contains only one byte and represents null; this value normally denotes a null object reference;

exception: ==TC_EXCEPTION== reset (Throwable)object reset

　　Describes exception information.TC_EXCEPTIONis the exception-information marker, indicating that the following sequence is an exception object;

magic: STREAM_MAGIC

Magic number;

version STREAM_VERSION

Serialization version information; the default value used in this article is05；

values: // Current object'sClassDecdescribes size and type

　　For the current object'sclassDescThe corresponding class descriptor provides the size of the described type;

newHandle: // The next number in the sequence is assigned to the object being serialized or deserialized

　　The next value in the sequence is assigned to an object reference that can be serialized or deserialized;

reset: // Discard the known-object set so the exceptional object cannot overlap with objects sent previously or with objects that may be sent after the exception

　　A set of known objects will bediscarded, resetting the byte stream;

b. Terminal Symbols and Constants

java.io.ObjectStreamConstantsdefines the terminal and constant values expected in the stream using the following symbols.

    final static short STREAM_MAGIC = (short)0xaced;
    final static short STREAM_VERSION = 5;
    final static byte TC_NULL = (byte)0x70;
    final static byte TC_REFERENCE = (byte)0x71;
    final static byte TC_CLASSDESC = (byte)0x72;
    final static byte TC_OBJECT = (byte)0x73;
    final static byte TC_STRING = (byte)0x74;
    final static byte TC_ARRAY = (byte)0x75;
    final static byte TC_CLASS = (byte)0x76;
    final static byte TC_BLOCKDATA = (byte)0x77;
    final static byte TC_ENDBLOCKDATA = (byte)0x78;
    final static byte TC_RESET = (byte)0x79;
    final static byte TC_BLOCKDATALONG = (byte)0x7A;
    final static byte TC_EXCEPTION = (byte)0x7B;
    final static byte TC_LONGSTRING = (byte) 0x7C;
    final static byte TC_PROXYCLASSDESC = (byte) 0x7D;
    final static byte TC_ENUM = (byte) 0x7E;
    final static  int   baseWireHandle = 0x7E0000;

Flag byteclassDescFlagsmay contain the following values:

    final static byte SC_WRITE_METHOD = 0x01; //if SC_SERIALIZABLE
    final static byte SC_BLOCK_DATA = 0x08;    //if SC_EXTERNALIZABLE
    final static byte SC_SERIALIZABLE = 0x02;
    final static byte SC_EXTERNALIZABLE = 0x04;
    final static byte SC_ENUM = 0x10;

If the serializable class written to the stream haswriteObjectmethod, and if that method has written other data to the stream, the flag is set toSC_WRITE_METHOD. In this case,TC_ENDBLOCKDATAmarker is always expected to terminate the data for this class.

If usingSC_BLOCKDATASetExternalizableclass writes to the stream, the flag is set toSC_BLOCKDATA. By default, inJDK 1.2sets inExternalizableProtocol for writing objects to the stream.JDK1.1writes intoSTREAM_PROTOCOL_1

If the class writing the stream extendsjava.io.SERIALIZABLErather thanjava.io.Externalizable, the flag is set to SC_SERIALIZABLE, the class reading the stream must also extendjava.io.SERIALIZABLE, using the default serialization mechanism.

If the class writing the stream extendsjava.io.EXTERNALIZABLEclass, the class reading the data must also extendEXTERNALIZABLE, and if itswriteExternalandreadExternalmethod reads the data, the marker is set toSC_EXTERNALIZABLE。

If the class written to the stream is an enum type, the following flag is set:SC_ENUM. The corresponding class on the receiving side must also be an enum type.

The preceding material can be summarized in the following diagram:

5. References

Official documentation:https://docs.oracle.com/javase/8/docs/platform/serialization/spec/protocol.html

Java serialization draft:https://blog.csdn.net/silentbalanceyh/article/details/8183849