Q and A: How can I tear out Informatica and MySQL and put in Big Data?

Today’s blog post comes from a question from a subscriber on my mailing list. The question come from G.P.:

I need to gain a hands on understanding of these technologies. I’m going to have to build some demonstration pilots before I would get any traction. I’m the VP of Analytics, so the engineering team think little of my opinion on data pipeline design, so I am just going to have to literally build them the example before I can tear Informatica and MySQL out of their cold hands 🙂 

Great question. Let’s unpack some of the questions you have:

• How can I get a management-level understanding of Big Data technologies?
• How can I convince the engineering team to change directions since I’m in the analytics side?
• How can I tear Informatica and MySQL away from the engineering team?

How can I get a management-level understanding of Big Data technologies?

I used to have, Director, VP, and CxO level people come to my hardcore development classes. They would spend four days learning about Big Data, but the class was focused on development. They got value from the first half-day, but nothing from the rest of the time. There simply wasn’t a class that taught the technical side of things for management.

Worse yet, there wasn’t a class that taught management how to run a successful Big Data project. There wasn’t anything that taught how to gain actual value from the data.

I took this to heart and created a course specifically for management. I called it the Business of Big Data. This is a class I normally run at on site at companies with their CxOs, VPs of Engineering, and other management folks. From time to time, I partner with other companies like O’Reilly Media and run the class online.

How can I convince the engineering team to change directions since I’m in the analytics side?

First, I’m going to play devil’s advocate. Then, I’m going to answer your question.

Let’s assume you don’t have Big Data problems and that’s why your engineering team isn’t responding. That’s one of the biggest and most important themes of my Business of Big Data course. Using Big Data on small data problems isn’t just overkill; it’s vastly more expensive and time consuming. If you make this mistake, you could waste a million dollars. In the course, I help walk you through the steps to not make this mistake.

Now let’s assume you’re right; it is a Big Data problem. I have a hunch that this issue is more of a people problem than a technical or business process problem. I’m going to guess your engineering team is making the mistake of thinking that a data engineer is the same thing as a DBA. At some of the companies I’ve taught at, their data warehousing team is responsible for their Big Data initiatives. In my experience, that’s a big mistake.

What you may need to do, is to find the programmers within your organization that actually understand Big Data. This is important because not everyone in the organization will be able to understand it fully.

It’s unfortunately common for the programming side of the company to have to end around the DBA/Data Warehousing side of the company to get their Big Data initiatives going.

How can I tear Informatica and MySQL away from the engineering team?

This is another issue where you may be hitting a people problem. If your engineering team is mostly DBAs, every problem is a nail that can only be hammered with SQL.

If you’re hitting Big Data problems, start looking for the manifestations and write them down. Every time you want to do an analytic and the engineering team says no, write down what you wanted to do and why they said no. If you’re hitting Big Data problems, they’ll consistently say the database can’t handle that query. They’ll say the query will take too long and slow everything else down. They’ll say it will cost too much in upgrades.

With this list, you can start identifying the highest value or simplest use case for a POC (proof of concept). This will be a great way to do an apples to apples comparison of the two systems. If the database can’t do it and the Big Data framework can, you have an easy technical decision. From there, it becomes a business decision on the value of the data and the value of being limited by your imagination instead of the database’s limitations.

One quick note, MySQL still has a place in your company and Big Data isn’t replacing it. One of the reasons you want qualified Data Engineers, is because they can tell you when to use Big Data or small data solutions.

My Recommendations

Here’s generally what I’d do:

• Attend the Business of Big Data class online (if possible) and get feel for Big Data and what it’s about.
• Decide if you really have Big Data problems. See if this really is right for your use case and company. We evaluate that during the class.
• If you do, we could start a focused engagement where we’d evaluate your team and technology. In the engagement we’d run the business class for the management folks and the technical class for engineering team. I have a convincing enough resume to help engineering teams see the light, no matter how stubborn they are. It’s part of what we do with engagements that’s different than other companies’ training.