Issue #5 - Data’s Growth Problem: Losing the Audience

The data industry is hard to keep up with, and that's a problem

Read time: 7 minutes

Truthfully, data has not and won’t lose people’s attention—between automation, AI and advanced analytics, it is way too captivating and mind-blowing.

But a growing awareness of these data themes is not the same as a growing understanding. In fact, as the popularity of data has continued to skyrocket, understanding has not kept up. And this has BIG implications!

Mo Money Data Mo Problems

Last week, we touched on the growth of data over the decades, from ‘on-prem, centralised data operations’ of the 90s to the ‘cloud-based, augmented with every tool imaginable’ experience we are familiar with today.

Even since 2010, the amount of data generated has grown 60x, with about 328 million terabytes of data created each day in 2023 (probably 25% more in 2024, especially with the buzz around AI). But it isn’t just the amount of data that is staggering, it is the hockey stick curve of the growth.

Source: Exploding Topics

The type of growth you see in the above image is incredibly hard to keep up with.

How can you possibly scale your data capabilities to stay afloat, while doing so in a cost-effective way?

This question is constantly perplexing for business leaders. Organizations take 6 months to even sign off a new tool or project, so it is impossible to keep up with new data & AI trends and tooling.

Data Tooling to Infinity & Beyond

Tackling this growing complexity and lack of understanding is difficult.

With resources already tied up in the day-to-day operations of the enterprise data platform, organizational data leaders often look to their most trusted external resources: consultants and vendors:

1. The Up & Down Consulting Relationship – Strategy and data consultants are often the first port of call. This is a great way to understand what is out there and what is best practice from experts. However, the flip side is over-reliance on consultants for everything data-related. Consulting relationships can become a bad addiction, and cost a lot of money. Use them sparingly, know what you want, and build internally using their advice

2. The Snake Charming Vendor – The other popular option is to find a new tool to solve your problems. As you can see from the MAD Landscape image below, there are an endless amount of data tools, with more emerging every year. And, if you believe them, each vendor knows exactly what you are going through and has the perfect product to solve your data woes. Hence, a lot of organizations have leveraged tools and vendors to help navigate the growing complexity of data. I actually promote using the expertise and experience of vendors, but with anything, take it with a grain of salt because buying new technology is like putting a band-aid on top of a severed limb (it ain’t really going to help).

The MAD Landscape image is truly mad, and the growth in tooling within data has grown exponentially over the past decade since this was first introduced by Matt Turck

---

Forgetting (Or Ignoring) The Foundations

As companies become enamoured with AI-enabled promises from consultancies and fresh out of the box technology fixes from vendors, you see their understanding of the data industry and their ability to navigate it actually decrease in proportion to its growing complexity.

A core aspect of this problem is something that I see in pretty much all my clients—the inability to adequately recognize and deliver on the data basics.

Executive: “Oh let’s just buy a tool to solve this problem”

There are several underlying elements of this I see in my clients all the time:

• Forgoing data governance teams or processes because we have a data catalogue

• Building AI pilot projects with consultancies before figuring out if the data is good enough quality

• Rushing to create analytical dashboards without seeking the business needs

• Doing anything without first ensuring proper data quality

• An organisation model that doesn’t complement the benefits data teams can bring

• Emphasis on Data Engineering and its tools, foregoing the elements that make engineering more robust and better (e.g., architecture, data models, data governance)

All of these examples involve teams prioritizing quick fix initiatives over foundational data activities. It is the idea that, for example, a data catalogue is enough to solve data quality problems; or we need to just get these dashboards and then the teams will adopt them; or we need more engineers to pipe the data even though they don’t know why they are building pipelines.

This is exactly why the vast majority of data teams are missing the bigger picture!

1. They take a siloed view of data, trying to solve front-of-mind problems by addressing 1-2 capabilities without considering the wider picture

2. This is done with a fire-fighting mentality, addressing short-term issues right in front of them and not taking a longer-term strategic approach, leaving the root causes of those immediate issues untouched…

3. At the same time, they are attacking those short-term issues with immediate fixes like tooling or cheap, junior resources, not setting the underlying principles and foundations to create a sustainable solution

What can I say, data teams are in the trenches—they are constantly fighting problems that pop up due to constant demands/ asks from the business without time to think whether they are even heading in the right direction. It is tough, but that’s how the corporate world works. And as the availability of data and the complexity in the industry grows, this will only get worse.

In July, I will do another article on a few of the biggest ‘forgetting the foundations’ barriers based on what I see with companies right now (I tried to fit it into this edition, but it got way too long), so look out for that!

---

How Data Can Avoid Slowly Losing Their Captive Audience

Let’s finish off by returning to the original point, as data growth has continued, companies and executives feel more out of the loop and lack the overall understanding they need to invest in it.

There is no silver bullet answer to how to solve this.

As advanced as our tools have become, there is a growing gap in understanding what data really means for business. Data warehouses, lakes, and AI algorithms are complex, and the people expected to use them often lack the necessary guidance or expertise. After the hype wears off, senior leaders go from the excitement of possibility to the overwhelming reality of implementation.

For data leaders, this lost understanding is a huge problem:

• It hinders potential investment

• It reduces organisational data literacy

• It makes it harder to deliver new projects

• It complicates the ROI calculation of data projects

• It gets in the way of creating a more data-centric culture

There is no silver bullet to solve this, but companies need to recognize why they are in this position, and steer the ship back in the right direction. Instead of thinking with a short-term mindset, they need to plan for the long-term:

• Approach data strategically (including new trends, tools, and fads)

• Understand the actions to operationalize the data strategy

• Hire and build internal capabilities for the long-term

• Shore up the foundations (e.g., data modelling, governance, platform, etc.)

• Declutter the tooling repertoire and be smart about new technology choices

• Constantly reassess the existing issues and what the root causes of those are

Data is going to continue to get more complicated.

At the same time it will continue to get more useful and necessary.

So companies can’t ignore these things any longer if they want the benefits without the incredible headaches, failed starts, poor ROI, inefficient operations, high turnover, etc. that all come with poorly constructed data teams.

And on the flip side, data professionals need to plan for and address these things when they are helping set up and lead teams or even suggesting solutions to existing problems. Scrutiny from senior organizational leaders will increase, and the data teams will need to learn to deal with it, or else lose their captive audience!

Speaking of getting senior leadership buy-in, next week is all about Data Investment—why it’s hard, how to approach it, and steps to pitch it properly. Thanks for reading all and feel free to leave some comments below about your experience!

---

Thanks for the read! Comment below and share the newsletter/ issue if you think it is relevant! Feel free to also follow me on LinkedIn (very active) or Medium (not so active). See you amazing folks next week!