It’s Time to Change How We Manage Data Teams

As a distributed systems person, I’m used to figuring out how to spread a problem out to the most number of computers possible. Spreading out a problem lets me leverage my resources far better and faster. However, we’re failing to apply this optimization to our own real-world management and innovation.

In business and technology decision-making, we’re centralizing the innovation and strategy onto just a relatively small part – the management team. We aren’t using the individual contributors to spread out the problem to more people to get more ideas, perspectives, and solutions to problems. We’re leaving some of the best and brightest – more importantly, people closest to the data and business problems – out of the conversation.

Much of the thinking and management philosophy comes from industrial age methods. With Agile, we’ve come to realize that we need to involve individual contributors more in planning. We’ll ask them how long something will take (scoping or story points). They can try to add technical tasks (technical debt) to their task list but don’t allow for new features or ideation. This process of ideation is left to a product manager or some other part of management.

Next come the data teams who try to replicate this industrial age top-down innovation. We’re thinking this is the same knowledge worker that needs to be told what to do and ignoring the crucial data aspects that make data teams unique. We’re still missing out on the insights that our individual contributors might have. When are we going to start listening to the people closest to the data? What insights could the data teams closest to the business problem have? If we aren’t listening or even asking these questions, we’re missing out on a considerable amount of brainpower.

While writing Data Teams, I started to find companies who did this. Their management got out of the way, and they started listening to the people closest to the data. I’ve started calling this emergence.

What does emergence look like in the real world? I set out to find that answer during my interview with Stitch Fix’s Brad Klingenberg (Chief Analytics Office) and Eric Colson (Chief Analytics Office Emeritus). Brad said, “The approach we’ve taken is much more like being a gardener, you just want to create circumstances where people can do good work and, occasionally you need to trim a branch back or make room for a new sapling, but generally you’re just trying to get the conditions right to then get out of the way.” You can read the rest of the interview in my Data Teams book. Stitch Fix created a site talking about how they cultivate algorithms.

I recently read David Marquet’s Turn the Ship Around: A True Story of Turning Followers Into Leaders. David’s thesis was that he turned a low-performing submarine by taking their followers (enlisted) and making them leaders. I realize the book about the U.S. Navy and submarines. However, I argue that the mindset shift of turning individual contributors (followers) into leaders is the exact process that data teams need to make in order for emergence to happen. He also shows that this process can work even when the stakes are as high as life and death on a submarine. If you can work with stakes that high, we know it can work for other last risky situations too.

This idea comes out in Aaron Dignan’s Brave New Work. A big theme of the book is that teams need a clear purpose. This purpose comes after better integration of the data teams and communication with a business. The data teams will directly see the value and results of what they’re doing. A direct view into results creates a virtuous cycle that will create more effective teams.

As we come out of the COVID era, we have to leverage our team’s talents to their maximum potential. By using a select few, we aren’t utilizing the entire team to our advantage. We have to start using the whole team for our innovation and strategy. It’s only then that we can use analytics and insights to their highest potential.