Issue #34 – Top Ten Data Ecosystem Predictions for 2025
Will companies do data better? What capabilities will be hot? What tools will Execs buy? How about AI?
Read Time: 18 minutes (sorry, its worth it though)
It’s that time of year again.
Not Christmas. Not New Years. Not Hanukkah. Not financial year-end.
No, it’s time for people to predict trends in data for the next year.
Predictions are a fickle beast:
Some are industry/ domain specific
Some are droll and predictable
And a lot are self-promotion for companies who want to advertise their products/ industries
Well, you know you don’t get that with The Data Ecosystem.
We tackle everything in data, don’t have corporate ties to marketing budgets, and rely primarily on people like you seeking out the best, non-partisan thought leadership on data a free subscription can buy!
So, without further ado, let’s dig into what I see as the top ten predictions in data for 2025!
Category 1 – Companies’ Data Abilities
1) The Average Company’s Data Maturity Gets to Level 2.5-3 of 5
Every company I work with feels like they are years behind their competitors regarding data & analytics maturity.
To tell the truth, they aren’t, and your organisation probably isn’t either. Most organisations just mention what they want you to hear about; this is how marketing works.
Not to mention, the many data awards and accolades are surface-level and reward member companies that pay the awardee an annual sum.
In reality, most companies are dealing with many of the same pain points you face: data quality issues, lack of communication with the business, little to show for AI investments, etc.
A core activity I do for most Data Strategy projects is a Data Maturity assessment. By ranking each data capability (e.g., engineering, governance, infrastructure, etc.), the company understands its performance and what it needs to do to improve maturity.
Currently, most companies are at a 2, meaning what they do in most data domains is fairly repeatable. Some domains may be a 3, and some may be a 1, but overall, it averages out to about 2.
My prediction for 2025 is that many organisations will get to a 2.5-3.
This means they have gone from having a repeatable approach to data to having more defined processes. So why this jump in maturity?
Most companies I work with have realised they need to get serious about Data Quality and the foundations that underpin analytical insights & data products. It only took 15 years or so…
Technologies are becoming more holistic, and the integration of tools is improving. If your technology works more effectively (especially with other tools), then your output will also improve
Business stakeholders are starting to get it. We love complaining, but business stakeholders expect data to improve their decision-making. They now kind of understand AI (thanks to ChatGPT). They want to use data. A willing workforce makes adoption and usage much easier than anything else I’ve ever seen
Data stakeholders are finding shortcuts that didn’t use to exist. Tools like Orchestra make connecting things easier. Platforms like Databricks make hosting easier. AI technology is making coding and building more efficient. Whatever it is, shortcuts to develop more effectively are out there, and it is about finding the right combo that fits your organisation’s use cases and strategy
Executives invested a lot in data because of the AI boom, and data teams are still riding that wave for two reasons.
The first is budgets: companies investing in data to make AI a reality, which includes putting money into other data domains.
The second is interest: AI offers value that executives and shareholders can’t ignore, making it paramount to put their data department on the right path
Of course, I caveat this all by noting that some companies will remain at a 1, while others might reach a 4; there will always be disparity between organisations.
It’s also worth mentioning that maturity is a sliding scale; as we all evolve, the scale gets harder. We’ve been playing at this entry-level scale for a while, so who knows if the Ad-Hoc level 1 suddenly becomes more difficult to implement.
2) Companies Get Serious About Technology Strategies
My most popular article this past year was about building a data technology strategy. I’ve had multiple inbound leads for consulting services to help set up a data platform, conduct a tech audit or build an overarching data technology strategy (if you are interested, reach out).
Why? Companies are spending a ton of money on their technology. With the AI hype decreasing, data budgets are coming under more scrutiny, requiring data teams to justify their spend, even on technology.
The data sector is too mature now to waste time and money on the wrong tools, and executives realise this. I’ve also seen how hard it is to switch vendors once you commit to a contract or implementation. Finally, so many technologies are out there, making it hard to select the right vendor, especially as tools say they can do so many different things.
Therefore, data leaders must get smarter and more strategic about selecting, investing, and implementing their technology.
Just as Data Strategy is becoming a more common artefact within companies, I see Data Technology Strategies following suit.
3) Value Continues to be Difficult to Quantify
This prediction is a cop-out—it has been true for every year data has been hot.
Value is just hard to quantify. This is for two reasons:
There is often no direct linkage between data spend and company KPIs
Proving the relationship between data and business value takes resources and time, things the data team don’t have access to or want to spend on
Data tools are good at what they do, but few data tools directly drive revenue or cut costs. They lead to information and insights that help business teams do those things, but the improvements are attributed to those business teams (or downstream tools).
Direct is the key point here. Linkages between data initiatives and value aren’t direct so it takes time and effort to assign it.
And teams don’t have that time. They are busy doing data things. Moreover, organisational structures are not set up and operational processes don’t exist to link data activities with business outcomes.
So, if this isn’t a new thing, why am I including it? Why bother?
Well, even though it’s not a new thing, it is important.
There needs to be a renewed focus on defining the tangible benefits data delivers for business teams. Too much money is being spent on data for us not to standardise this process. Moreover, as the AI hype decreases, teams have to justify their spending even more, so business cases need the clarity they currently lack.
Demonstrating the value of data is possible, you just have to invest to do it.
In the next month or so, there will be a co-authored article (with Ryan Brown) on this topic exactly, so subscribe and make sure you read it!
Category 2 – The AI Predictions
I can’t have a top predictions article without mentioning AI, so here are the two that I’m seeing (especially as I do research on my ML/ AI five-part series due to hit the internet in late January & February).
4) AI Re-enters Reality
When ChatGPT came out, one of my old clients immediately set up a new AI team and told their shareholders all the amazing things this team would do. This was despite AI not being on the agenda of their immediate data priorities—they had data quality issues, poorly architected data storage and servicing, and disjointed analytical efforts that didn’t deliver value for business teams.
Unsurprisingly, seven months later that team was shut down.
I’ve said this for a while, if your foundations aren’t there, your AI will fail.
And I expect this to be a HUGE data trend in 2025.
Don’t get me wrong, I believe in the power and usefulness of AI. But most corporations I know don’t have the foundations to get reliable value from AI.
So what is the reality of AI?
Just like Data Science ten years ago, AI needs to find its centre of gravity in organisations and reach for what’s plausible rather than the art of the possible.
Even Gartner agrees, with many AI categories currently at the peak of inflated expectations heading into the trough of disillusionment.
In 2025 I therefore predict AI will reduce its mandate and take two roles in most organisations.
The first is small team experimentation and delivery. Instead of large, ambitious projects, organisations will have smaller initiatives focused on building small, proven AI use cases, potentially on the back of existing data science and analytics work.
The second role will be to train employees to leverage mainstream AI tools like chatbots and AI add-ons. This should be a no-brainer but for the past few years companies have bought AI tools without providing the proper training and change management to get the value from them. Hopefully, 2025 will be the year they invest in democratising accessible AI tools.
Overall, I believe these two approaches to AI are much more sustainable and scalable.
Also, I expect (or hope) that ‘AI-enabled’ marketing is no longer in every SaaS tool’s value proposition…
5) AI is All About Bespoke Insight
Everywhere I look, AI experts (a loose term) are discussing AI Agents.
As defined by Aurimas Griciūnas in his recent article: “an AI agent is an application that uses LLM at the core as it’s reasoning engine to decide on the steps it needs to take to solve for users intent.”
Now, I’m not going to discuss what an AI Agent is or how they will take over the world in 2025 (more on that in my February articles). Instead, I will discuss how AI has philosophically evolved over the past few years and how it will continue to do so.
In two years, the hype has evolved from Large-Language Models (LLMs) to Multi-Modal Models to Retrieval-Augmented Generation (RAG) to AI Agents (or Agentic Models). Each step marks the progress made in AI.
But if we look past the application, what is the bigger business trend?
Simply put, AI is increasingly focused on tailored information focused on driving business-specific action.
LLMs set the base foundation of knowledge by scrapping the entire Internet. Multi-Modal models go beyond language, helping add more context and content. RAGs then add functionality to LLMs by adding domain-specific information, reducing hallucination and improving the business-relevance application of the AI tool. Finally, AI Agents introduce autonomy and decision-making into the mix, helping take action from the base of knowledge available.
My prediction is AI Agents won’t be the last on this list. The evolution we’ve seen reflects a progression from static, wide-ranging knowledge to specific, dynamic functionality. We are entering the point where AI ‘models’ have evolved from passive assistants to becoming active collaborators in solving complex, real-world problems. This problem-solving is becoming more bespoke as we have better information and data to feed the model’s ‘intelligence’.
The only issue? Most data going into these AI tools is poor, limiting their overall effectiveness (more on that in a later prediction).
Category 3 – Data Domains/ Capability Predictions
6) Return of Data Modelling (alongside Data Engineering burnout)
I’ve written about the importance of both data architecture and modelling. I’ve also written about the issues within the Data Engineering industry, with so many Engineers burning out due to the unrealistic demands inherent in their job description.
Maybe I’m caught up in the need for data modelling, so I’m overly bullish on it. But I see its stock rising in 2025. Specifically because:
Data Complexity Requires It – Companies used to bypass data modelling with engineers and tech (e.g., data warehouses, ETL tooling, etc.). But now, the amount of data (and the number of sources generating it) has skyrocketed, creating complexity that most tools/ technologies can’t handle. Not to mention the need to link the business context and the physical data models, I think companies will take a step back to be more strategic about how their data should be modelled.
Data Engineer Frustrations – Data engineers are tired and frustrated at short-term thinking that causes pipeline rework, low-value ticketing solutions and an ever-present divide between their work and the value the organisation sees. Many realise the need to learn about data modelling but don’t necessarily have the time to embed that thinking during their day job. I anticipate some of this frustration to blow over and cause change, just like it did when Data Scientists turned into Engineers due to a lack of analytically ready data.
Joe Reis’s Crusade & New Book –
has been banging the data modelling drum for over a year now and his new book should arrive in 2025. Beyond his newsletter and monthly data therapy sessions, this new book should shed light on thinking about and implementing data modelling to drive data value in organisations. Just think about what The Fundamentals of Data Engineering did for Data Engineering!Tools Exist to Help – This point comes from two angles. The first is that many more focused tools are gaining popularity that solely help companies model their data (e.g., ER/Studio, MySQL, etc.). These companies help track workflows, manage technology integrations, and define data assets/ tables to make it easier to understand what exists where and the value that data is delivering. The second point is that mainstream tools in other technology categories are starting to incorporate data modelling into their product offering. Think Databricks with their Unity Catalog or Orchestra with inherent architecture to create properly structured data flows. This all draws from modelling principles and while it’s not a complete solution, it helps embed some structure where it didn’t exist before
This is a domain and a skillset I’m excited to watch for 2025, and you should be too!
7) Data (and AI) Strategy Popularity Continues to Grow
Okay this is a self-interested trend, but I think Data Strategy is coming into the spotlight.
This is coming from four angles:
Executives Need Clarity on Direction – Data continues to evolve at a breakneck pace with 81% of US executives finding it difficult to keep up. Every Executive I speak to wants to know how to improve their company’s data & AI efforts. While they have been interested in this topic for 5-10 years, they are finally realising that success in data requires a strategic approach (especially after so many AI tooling failures). I’m beginning to see a lot more CMOs, CFOs and COOs banging the data strategy drum to understand how data can help their teams succeed.
Petitions from the Data Team – Data professionals want direction. They need help turning short-term, data activities into beneficial, strategic initiatives. They feel siloed in their teams and frustrated that they aren’t seeing the fruits of their labour. And their voices are getting louder. I don’t expect this to stop. Companies that want to keep their top data talent will need to do more to give them the strategic direction they are requesting. The one caveat I will mention is that data people want the strategy to be actionable, not just a business-focused PowerPoint deck made by pure strategy consultants (this is what you get from many big consultancies because their teams don’t know how to bridge the gap).
Lots of Wasted Data & AI Investment – Unsurprisingly, a lot of AI projects have failed in the past two years, about 80% according to some RAND corporation research. And AI is expensive, so this is a huge loss! I anticipate this wasted investment will lead to more scrutiny and structured spend on AI. The spend might not slow down (companies still recognise the need to do Data & AI), but executives might be a bit smarter where they put their money (i.e., hire a data strategist to build the path to success instead of a data scientist with no guidance on what to do).
AI Strategy – I worked with a company who were caught up in the AI hype and built an AI Strategy. The first recommendation of the AI Strategy? To do a Data Strategy! Realising benefits from AI without first considering data is fundamentally impossible. So, as companies invest in AI strategies, by default, there will be a bigger focus on Data Strategy.
These things accelerate a trend I’ve seen: the increasing complexity of the data world makes strategic thinking and linkage to the business that much more important!
And Data Strategy popularity doesn’t just come as a single domain. Most engineering, analytics, governance, and other projects I talk about with clients have some strategic foundation (or they should). Data teams and professionals realise we can’t build for the sake of building any longer; we need to build with a plan and think more holistically with the long-term in mind!
In the next few months I will finally do an article (or multiple) on Data Strategy. People keep asking me for content recommendations and there are some great books out there (e.g., Simon Aspen-Taylor’s Data & Analytics Strategy for Business, Jordan Morrow’s Be Data Driven), but I still haven’t created my own long-form version of what Data Strategy is and how to do it. As a Head of Data Strategy, and given I’ve been posting about it for 3 years now, its probably time I write that content!
8) Data Governance Gets Real Hot
The most underrated data domain for the 14th year straight????
Drum roll please…
It’s Data Governance!!!
Will this be the year people finally realise its importance? I hope so! People view Data Governance as a bureaucratic service that helps you hit regulatory and compliance requirements. This is a very narrow and short-sighted perspective and thinking this way does a lot of damage to your org’s data capabilities.
As I’ve said before in a past article, Data Governance is about defining how data should deliver value in an organisation while maintaining quality (e.g., accuracy, consistency, compliance, etc.) and being owned by the right business stakeholders. A data governance strategy defines how governance fits in and augments your overall data strategy and, in turn, your business strategy.
Data Governance professionals who approach the job this way are in high demand. Companies are seeing the value in having strong data governance teams who actively set processes, definitions and policies to help the business use data more effectively, while improving the overall quality of data.
So if you want a secure job, go into data governance because it’s going to be a hot domain for years to come!
And check out
’s Data Governance trends. Many of her predictions for 2025 overlap with mine, even though they have a Data Governance focus. I think that alignment just shows how important Data Governance is within the Data Ecosystem!9) Data Quality Moves Upstream
Over the past 4-5 years, there has been a lot of emphasis on data quality.
The first domino to fall was the transfer of budgets from data science to data engineering. Now you have the hype around things like data catalogues, observability, contracts, quality platforms, and everything else.
And now with AI hallucinating or failing due to poor data quality, the attention has only increased.
But what prediction are you even making Dylan? This isn’t a new trend…
I predict that data quality will continue to move upstream beyond what we see today. Everybody talks about the ‘shift-left’ concept. The idea of "shifting left” originates from software development, where testing and quality assurance are moved earlier in the development lifecycle to catch issues before they snowball into costly failures downstream. This same principle is now becoming central to how organisations approach data quality.
In the data world, this means considering data quality, data governance, and security before processing and analysis. Ideally, this makes data management more proactive, involving both the consumer and producer in the design process, adding earlier detection points, and helping create data workflows with improved quality.
Most people talking about shift left work in the data quality or governance domains, and are promoting their own knowledge/ perspective.
I am approaching this evolution from a different angle though.
Trending technologies demonstrate the shift left concept: year after year attention has moved from the right side—consumption—to the left side—requirements setting & design.
This began with the data engineering boom as data scientists realised they needed to build pipelines to clean and deliver data. But people didn’t know what data was being connected so data catalogues got hot. But pipelines continued to break and suddenly data observability became hot. But while people were notified of the errors, they weren’t being fixed or built correctly, leading to the new buzz around data contracts and holistic data quality platforms.
And I don’t see it stopping at Data Contracts or Platforms.
One of my favourite new data tools is Upriver (note I am an advisor, but I am for a reason). Their whole value proposition is about increasing the upstream data quality and transparency by embedding proactive data quality measures directly into data pipelines and systems of record. What does the tool do?
It automates data contracts to ensure schema compliance, DQ measures, and SLAs are easily built between producers and consumers.
Has proactive observability that identifies problems back to the original upstream code (so you know where the problem comes from).
It streamlines the resolution process by defining how to handle each incident.
Improve collaboration between business and engineering teams by measuring customer metrics that business teams care about, helping them set and monitor expectations.
So why is this unique? Because Upriver incorporates the business users’ perspective into the shift-left data quality principle.
And I don’t see that evolution stopping. This is my prediction! Data quality will continue to move upstream to ensure business teams and users are guaranteed it is set (and maintained) from the start.
I don’t think Data Contracts or Catalogues or Observability tools will be the final thing. I think there still is room to better incorporate the business teams’ perspectives into data quality management, and we will see more of the buzz around that in 2025.
Category 4 – Outlier Prediction
10) Databricks IPO Finally Happens
I want to start this off by cautioning that I’m not a market analyst and I’m not a shareholder of Databricks (unfortunately). I also thought of this trend before Databricks issued their Series J investment for $10 billion, bumping their market value to $62 billion (which is insane). After that public offering, I’m less sure it will happen in 2025 (maybe 2026 instead).
But I still want to make this prediction. Why?
Employee Push to Cash Out – With a company worth $62 billion, early employees will want to sell their stock for astronomical prices. Especially after NVIDIA’s surge created hundreds or thousands of new stock-focused millionaires last year, you can bet there will be internal pressure to go public to cash in finally.
Pressure from Cloud Partners – The AWS, GCP, and Microsoft’s dominate the cloud market, but don’t necessarily have a strong footing on the compute and processing tooling. However, you can bet they want one! To create a more single-vendor ecosystem, I could see these three organisations investing more into the Big Query or Redshifts/ Sagemaker tooling which could put pressure on Databricks. An IPO would help fund any necessary investment Databricks would need to make.
The Snowflake Wars – The bidding war for Tabular demonstrated the intense competition between Databricks and Snowflake. Paying $2 billion for it is pretty insane, and probably a reason it is raising again. While it seems funding won’t run out (given their private round was oversubscribed), going public might help give Databricks the edge to continue Snowflake’s downtrend and create market dominance for years to come.
Increasing Data Offering – Databricks is trying to become more holistic in what it delivers (recently breaking into BI product offerings). As a private company with one specific (but strong) value proposition, that is hard to do because people don’t know what else you are about. A public offering helps create transparency into plans, goals, and investment, something that could help with their long-term vision of diversification.
Anyway, this was my fun prediction, and I’m 50/50 on whether I will even be right about it!
There you have it, my 2025 predictions!
I thought about and worked on these for a few weeks (what a fun holiday), so I hope you enjoyed them and they help you think about the year to come. And if you liked any of them, please share them with others!
We will return to regularly scheduled programming next Sunday, and we’ve got a few great topics coming up including Reverse ETL, Scaling a Data Platform, Identifying Data Value and a 5-part series on ML/ AI.
2025 is going to be a great year for the Data Ecosystem so stay tuned and thanks for reading/ subscribing!
Thanks for the read! Comment below and share the newsletter/ issue if you think it is relevant! Feel free to also follow me on LinkedIn (very active) or Medium (increasingly active). And if you are interested in consulting, please do reach out. See you amazing folks next week!
Great read, thanks for the tag ! Totally aligned with your predictions - and especially with the maturity and strategy part, i’m also still working with companies on their data strategy rather than their AI strategy.
Cool stuff.. Great list to look out for 2025.. Moving away from Fancy AI to boring Data Quality.. It might still take time for executive and tech leaders to get back to this fundamental shift…