Machine to machine conversation with Avro, Protobuf or Thrift

Introduction

API data exchange requires an established mode of transport such as a network protocol or data file system. The data being transferred should also fit an agreed upon format. SOAP solves that problem by establishing XML as the data format. Modern architectures have deviated from XML because of its payload overhead attributed to markups, and also having a dependency on developer expertise. Moreover, XML processing can be expensive and require more resources than are available on small devices.

A popular data protocol alternative is JSON which is lightweight, schema enforceable and simple to understand. In order for machines/APIs to 'talk to each' other JSON would need to be translated into the API language i.e. JAVA , Python etc. This requires some data extraction and translation on the fly. A serialization framework, SF, such as Avro, Thrift and protobuf solve the language bridge by making extraction and translation seamless by using pre generated client stubs. A translation layer based on a defined schema is created for both clients in their respective languages such that JAVA can talk to Python or other supported language. SF's use an interface description language IDL, to define the generated translation layer source code.

Here are some examples in JAVA:

Avro

It allows for native type definition, nesting, accumulations (array and maps) and complex types.  It uses json as the IDL thus familiar. Precompiled translation/serialization layer is not required but depending on business use case you may require them.  On the fly schema driven data translation makes it easier to work with a growing schema or adopt new schemas making development process decoupled from data definition. However, minimal development is required for all clients to implements avro apis.
Language support is generous as it supports Java, C++, C#, python, js among others.

Download avro tools jar from apache

Create schema. For example create file CustomMessage.avcs

{"namespace": "avro",

 "type": "record",

 "name": "CustomMessage",

 "fields": [

     {"name": "id", "type": "string"},

     {"name": "payload",  "type": "string"}

 ]

}

As mentioned before you have two options: to generate stub, or create a generic record parser.
Option 1 Generate stub:

java -jar ./avro-tools-1.7.7.jar compile schema  avro/CustomMessage.avsc .

Option 2 stubless avro parser: See my parser example on github.

Run tests using maven dependency:

<dependency>

  <groupId>org.apache.avro</groupId>

  <artifactId>avro</artifactId>

  <version>1.7.7</version>

</dependency>

Protobuf

Is a google creation which is widely used in its ecosystem. It has limited language support but the documentation is very detailed. It requires code stubs which are precompiled making slightly  inflexible for the development process. The IDL is simple and easy to understand but its not json.

It requires system installation:

1. download
2. ./configure
3. make
4. make check
5. sudo make install
6. protoc --version

Create schema. For example, create file CustomMessage.proto :

package syntax2;

option java_outer_classname="CustomMessageProtobuf";

message CustomMessage {

    optional int32 age = 1;

    optional string name = 2;

}

Note the native type support and package (protocol3 which can also be replace with protocol2 ). See below. 'Optional' keyword is removed.

package syntax3;

option java_outer_classname="CustomMessageProtobuf";

message CustomMessage {

    int32 age = 1;

    string name = 2;

}

Generating protobuf java objects
protoc CustomMessage.proto --java_out ../

When testing, note the relationship between syntax and maven dependencies:

Maven dependencies for syntax2:

<dependency>

<groupId>com.google.protobuf</groupId>

<artifactId>protobuf-java</artifactId>

<version>2.6.1</version>

<scope>compile</scope>

</dependency>

Maven dependencies for syntax3:

                 <dependency>

<groupId>com.google.protobuf</groupId>

<artifactId>protobuf-java</artifactId>

<version>3.0.0-beta-2</version>

<scope>compile</scope>

</dependency>

Thrift

Thrift, like avro supports more languages than just java, C++ and python. It has great history with popular usage in big data technologies such as HBase's multi language support. Thrift suffers a back draw; poor documentation and support. It is also developer expertise driven.