by Jesse Anderson | Aug 17, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
Today’s blog post comes from a question from a subscriber to my mailing list. The question come from André M.: How good a Big Data strategy can be defined by someone that doesn’t know the technology behind it? (By knowing I don’t mean being able to... by Jesse Anderson | Aug 10, 2016 | Blog, Business, Data Engineering |
Companies and individuals often come into Big Data thinking everything is cheap. After all, the entire stack is open source, right? Well, some things are cheap and some things are more expensive. Software One of the important distinctions with Hadoop is that it... by Jesse Anderson | Jul 27, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
Some of the contenders for Big Data messaging systems are Apache Kafka, Google Cloud Pub/Sub, and Amazon Kinesis (not discussed in this post). While similar in many ways, there are enough subtle differences that a Data Engineer needs to know. These can range from nice... by Jesse Anderson | Jul 20, 2016 | Blog, Business, Data Engineering |
Kafka 0.10 is out. Here are the changes that developers need to know about. Here is the new URL to the Kafka 0.10 JavaDoc. KafkaConsumer The KafkaConsumer had a minor change to that allows you to specify a maximum number of messages to return. You can set this by... by Jesse Anderson | Jul 13, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
Apache Beam just had its first release. Now that we’re working towards the second release, 0.2.0-incubating, I’m catching up with the committers and users to ask some of the common questions about Beam. Each committer and user is sharing their own opinion... by Jesse Anderson | Jul 6, 2016 | Blog, Business, Data Engineering, Data Engineering is hard |
I had a conversation with another person in the Big Data field. We were discussing whether the Data Engineers would become a more common job title and migrate out of Silicon Valley. I told him yes. Big Data is downright complicated on many levels. There are too many...
