Thursday, July 14, 2016

Missing in Gora: Avro Phoenix translation

I was disappointed to see that while Gora supports Avro-SQL (for Mysql etc) bridging, it does not support Apache Phoenix SQL. Note that Gora supports HBase data which is logically different from Phoenix.
I found a way around this void. And that is to create a custom Avro-Phoenix translator.

The challenge is be aware of Avro or Phoenix data types and do a programmatic mapping onto a prepared statement. This can be extended to any jdbc driven connectivity:

My Avro Schema




Create phoenix table:
!sql CREATE TABLE TEST_TABLE (CREATED_TIME TIMESTAMP NOT NULL, ID VARCHAR(255) NOT NULL, MESSAGE VARCHAR(255) , INTERNAL_ID VARCHAR CONSTRAINT PK PRIMARY KEY ( CREATED_TIME, ID ) ) SALT_BUCKET=20;

Write to phoenix to test
!sql UPSERT INTO TEST_TABLE (CREATED_TIME, ID, MESSAGE) VALUES (now(),'id1', 'my message');

Write Avro to Phoenix translation


View the github project.

Read Phoenix Data to Avro translation ( Future work ).


Docs:

https://avro.apache.org/docs/1.7.7/spec.html#schema_record
https://avro.apache.org/docs/1.5.0/api/java/org/apache/avro/Schema.Type.html
https://phoenix.apache.org/language/datatypes.html

No comments:

Post a Comment