by Jesse Anderson | Aug 10, 2016 | Blog, Business, Data Engineering |
Companies and individuals often come into Big Data thinking everything is cheap. After all, the entire stack is open source, right? Well, some things are cheap and some things are more expensive. Software One of the important distinctions with Hadoop is that it... by Jesse Anderson | Jul 27, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
Some of the contenders for Big Data messaging systems are Apache Kafka, Google Cloud Pub/Sub, and Amazon Kinesis (not discussed in this post). While similar in many ways, there are enough subtle differences that a Data Engineer needs to know. These can range from nice... by Jesse Anderson | Jul 20, 2016 | Blog, Business, Data Engineering |
Kafka 0.10 is out. Here are the changes that developers need to know about. Here is the new URL to the Kafka 0.10 JavaDoc. KafkaConsumer The KafkaConsumer had a minor change to that allows you to specify a maximum number of messages to return. You can set this by... by Jesse Anderson | Jul 13, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
Apache Beam just had its first release. Now that we’re working towards the second release, 0.2.0-incubating, I’m catching up with the committers and users to ask some of the common questions about Beam. Each committer and user is sharing their own opinion... by Jesse Anderson | Jul 6, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
I had a conversation with another person in the Big Data field. We were discussing whether the Data Engineers would become a more common job title and migrate out of Silicon Valley. I told him yes. Big Data is downright complicated on many levels. There are too many... by Jesse Anderson | Jun 29, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
A common question beginners ask about Hadoop are the technical skills needed to get started. This helps level set what skills you need before you embark on a big data journey. For developers and administrators, I divide up the skills as those that required and those...
