- Many companies are utilizing data mesh as a common methodology.
- Choosing a methodology requires understanding that there is no one-size-fits-all solution.
- Homegrown methodologies are prevalent, combining elements from various established methodologies.
- Implementing methodologies fully can be challenging, especially in data-related fields.
- Companies often struggle to fully implement agile methodologies, sometimes resorting to a waterfall approach.
The results and analysis from my 2023 Data Teams Survey left a few open questions. Let’s revisit these questions with some answers.
Methodologies and Size of Company
Figure 1 – Methodologies Broken Down By Size of Company Using Them
We see a few commonalities across different company sizes, as shown in Figure 1. One striking commonality is that so many companies are using data mesh.
I find it difficult to believe that these smaller companies have issues that require a data mesh approach. I’m also curious if the small companies have enough people to implement the data mesh approach properly.
As you choose a methodology, remember that there are no magic bullets or a single formula to get it right.
As you choose a methodology, remember that there are no magic bullets or a single formula to get it right.
As you choose a methodology, remember that there are no magic bullets or a single formula to get it right. You can’t just approach this like moving your shopping cart through the store and pulling things off the shelf. Each of these methodologies has essential pros and cons to consider.
Homegrown Methodologies
There were a surprising number of homegrown methodologies that people reported. To better understand what people meant by their homegrown methodology, I contacted people to get more information.
Here are a few anonymous quotes:
“I would say it is a mix of Kanban/Agile/Scrum. The reason being as we have a diverse team with different needs, so we scramble and adjust to be as efficient as we can. I guess you can say we pick the best parts of each methodology whenever we need.”
“For us, the issue is that we have:
- a fully distributed team
- minimal schedule of meetings
- our main access point for deliverables is through GitHub private repos => Azure DevOps
- customer has domain expertise but little exposure to software engineering
- customer is unfamiliar with using GitHub
In the past two years, we’ve had only a handful of people in the [remote] office who actually interacted with GitHub issues. We’re not allowed to see process beyond the Azure boundary, and in some cases it involves transfer by hand of source files from the GitHub private repos into an internal GitLab repo which we’re not allowed to see. But we’ve had to evolve a homegrown process that fits both teams.”
I put homegrown because our methodology is a mix of several of those. Although I work at a company now that has a very immature data process, my experience at other companies at a more mature stage was the same. It seems like the ideas for the major methodologies get adopted in pieces (either what fits or what pieces a manager latches on to) but not fully implemented.
Getting teams to pick one and fully implement it is a really tough task, even when building everything from scratch.
Getting teams to pick one and fully implement it is a really tough task, even when building everything from scratch.
A good example is how many companies try to use agile methodologies, but end up doing it in a waterfall fashion. That isn’t always a bad thing, but getting teams to pick one and fully implement it is a really tough task, even when building everything from scratch. In the software engineering space this is hard, in the data world it seems impossible.
As I am building out the data team in my current role, and have influence over the whole process, I am trying to move towards a dataops methodology, but with limited resources it will likely take a long time to get there. Data engineering is run under the software engineering head, but our projects don’t fit nicely into their sprints. Many of the dataops functions were being handled by the devops team before me, which led to solutions and methods that weren’t optimal for the volume of data we are working with. So with a very small data team it is mostly just trying to organize everything and work on the biggest problems first, while attempting to move toward a more coherent methodology.”
“The workflow was based on a few scrum ceremonies, etc. but without any designated roles, predomimately due to a lack of management prescribed methodology. I have since left the organization which ironically, in addition to the analytics practice I was part of, provided consulting services for Agile implementations. I wasn’t aware of any plans to institute any particular flavor of project management.”
I think these comments highlight a few clear holes in our project management frameworks. We don’t have a ready-made project management framework that slides right into data science or data engineering. I believe that we’ll continue to research what works best.
There isn’t a one-size-fits-all solution for data teams.
As you can see, there isn’t a one-size-fits-all solution for data teams. If you’d like help starting or fixing your team, please contact me.
Frequently Asked Questions (AI FAQ by Summarizes)What is data mesh and why are many companies utilizing it?
Data mesh is a common methodology being used by many companies to manage and organize their data effectively.
Why is it important to understand that there is no one-size-fits-all solution when choosing a methodology?
It's crucial to recognize that different companies have unique needs, and what works for one may not work for another, hence the need for customized solutions.
Why are homegrown methodologies prevalent in many companies?
Homegrown methodologies are popular because they allow companies to combine elements from various established methodologies to create a tailored approach that suits their specific requirements.
Why can implementing methodologies fully be challenging, especially in data-related fields?
Implementing methodologies fully can be challenging in data-related fields due to the complexity of managing and analyzing data effectively.
Why do companies sometimes struggle to fully implement agile methodologies and resort to a waterfall approach?
Companies may struggle to fully implement agile methodologies due to various challenges, leading them to resort to a waterfall approach as an alternative.
Why is transitioning to a dataops methodology a lengthy process with limited resources?
Transitioning to a dataops methodology can be time-consuming and resource-intensive due to the need for significant changes and adaptations within the organization.
Why may project management frameworks not seamlessly fit into data science or data engineering projects?
Project management frameworks may not seamlessly fit into data science or data engineering projects due to the unique nature of data-related work and the specific requirements involved.
Why is there no universal solution for data teams, and why is a unique approach necessary for each team?
There is no universal solution for data teams because each team has its own challenges and requirements, necessitating a customized approach for optimal results.
Why is seeking assistance for starting or improving data teams recommended for those facing challenges?
Seeking assistance for starting or improving data teams is recommended to address challenges effectively and ensure the successful implementation of tailored solutions.
