Unit testing your Kafka code is incredibly important. It’s transporting your most important data. As of 0.9.0 there’s a new way to unit test with mock objects. Refactoring Your Producer First of all, you’ll need to be able to change your Producer at...
I’m often asked what I think will happen to Big Data over the next five to ten years. From a Developer’s point of view, they’re asking if investing their time in becoming a Data Engineer will pay off. We’re going to see a continuing maturity of...
You’re considering a change to become a Data Engineer. Why should you do it? Why shouldn’t you do it? Let’s consider some reasons. Should There is a major shortage of qualified Data Engineers. There is a high demand and low supply of qualified Data...
Today’s blog post comes from a question from a subscriber on my mailing list. The question come from G.P.: I need to gain a hands on understanding of these technologies. I’m going to have to build some demonstration pilots before I would get any traction....
I always encourage companies to break down their Big Data projects into smaller pieces. I call this process crawl, walk, run. There is an interesting corollary to this process. Some companies get stuck at the crawl phase and don’t progress on to the walk and run...
Last week, I gave two talks about Strata+Hadoop World. These talks covered some of the up and coming technologies in Big Data. I describe Strata as the Super Bowl of Big Data conferences. This is where you’ll find the best minds talking about the present and...
