Tag Archives: MRunit

Testing Hadoop Programs With MRUnit

via Testing Hadoop Programs With MRUnit – Random Thoughts on Coding.

This post will take a slight detour from implementing the patterns found in Data-Intensive Processing with MapReduce to discuss something equally important, testing. I was inspired in part from a presentation by Tom Wheeler that I attended while at the 2012 Strata/Hadoop World conference in New York. When working with large data sets, unit testing might not be the first thing that comes to mind. However, when you consider the fact that no matter how large your cluster is, or how much data you have, the same code is pushed out to all nodes for running the MapReduce job, Hadoop mappers and reducers lend themselves very well to being unit tested. But what is not easy about unit testing Hadoop, is the framework itself. Luckily there is a library that makes testing Hadoop fairly easy – MRUnit. MRUnit is based on JUnit and allows for the unit testing of mappers, reducers and some limited integration testing of the mapper – reducer interaction along with combiners, custom counters and partitioners. We are using the latest release of MRUnit as of this writing, 0.9.0. All of the code under test comes from the previous post on computing averages using local aggregation.