Sunday, March 15, 2009

A new appreciation for convention over configuration

At work, I'm working on revamping our logging infrastructure to use Hypertable (an opensource Bigtable clone) for storage, and Hadoop (an opensource Map/Reduce framework) for log processing.

I'm extremely excited about the potential these tools have for allowing us to do all sorts of ridiculously cool analyses on our logs, and at some point I'll write about some of the things we are doing with them. However, in the meantime I just wanted to share my renewed appreciation for the simplicity of convention over configuration that I've gotten used to with Rails.

You see, Hadoop is written in Java, and uses Apache Ant for builds. We're talking 60-100 line XML files for to configure simple example jobs. Why would anyone voluntarily set their system up this way? To paraphrase one of my coworkers... enterprise-quality appears to be roughly translatable as 'over-engineer everything'!