Rust avro->json tool performance

Here’s the tool written in Rust language which I’m going to compare its performance with java written tool.

ad-hoc exploration of data sets easy without having to use more heavy-weight tools like SQL/MapReduce/custom programs. rq fills a similar niche as tools like awk or sed

dflemstr/rq: Record Query – A tool for doing record analysis and transformation

For data, I used Cloudera file which contains Hive metrics. As you see below, rq was 2.4 times faster then reference implementation in Java.

1. Java

$ time (for i in i {1..30};do java -jar \
/opt/cloudera/parcels/CDH/lib/avro/avro-tools.jar tojson hive.avro > /dev/null 2>&1;done)

real    0m48.286s
user    2m54.242s
sys     0m10.124s

2. rq

$ time (for i in {1..30};do rq -a < hive.avro > /dev/null 2>&1;done)

real    0m20.760s
user    0m15.796s
sys     0m4.972s

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s