Issue #46 – Dealing with the Difficulties of Data Ownership
Without proper accountability, all your quality efforts will fall flat
Read time: 9 minutes
There are numerous facets to data quality, making it challenging to keep up.
You may have installed observability tools to detect data issues. And set up contracts and testing to enforce quality standards. And perhaps your engineering team is building better pipelines every day.
Yet, data quality problems persist. Reports are questioned, dashboards are mistrusted, and business decisions remain shaky.
Blame may go back to the engineers or the tools, but the true problem is a lack of ownership.
And unfortunately, that is usually both a business and data problem (even though the data team gets most of the blame)
Over the last three articles, we've examined the modern-day data quality landscape, from the root causes of poor quality to detecting issues through observability and preventing problems with contracts and testing.
However, there's one critical element we haven't fully addressed: who is actually responsible for the data?
From a philosophical perspective, it is actually fascinating how many data concepts ultimately circle back to the foundational People, Process, and Technology framework. Even within data quality, there are distinct roles for each part of this framework, something that can be overlaid in the three components of a data quality platform:
Detection (Observability) is now primarily technology-driven
Enforcement (Contracts & Testing) is fundamentally process-driven
Ownership relates directly to people and establishing clear accountability (although has a lot of process baked in as well)
Only with all three working in harmony can you achieve sustainable data quality. And since ownership creates the crucial bridge between data teams and business stakeholders, it's the lynchpin that transforms quality data into business value.
Let's explore why ownership matters so much and how to establish it effectively within your organisation.
The Ownership Gap in Modern Data Ecosystems
Most data professionals weren’t around (or weren’t working full-time) 30 years ago when technology and the internet were making their way into the everyday functioning of businesses.
Back in the day, it was much simpler; people knew who had what information, systems were fewer, datasets were smaller, and sharing information happened organically in a more ad hoc and informal way. Or if those things didn’t work quite that well, it wasn’t as big of an issue.
But that is not true today.
The explosion of data volume and complexity has made any manual effort to stay on top of things absolutely impossible. As I’ve mentioned before, there is 5- 6x more data than seven years ago, meaning there are also probably 3- 4x more sources of data. Don’t forget that this also means having to manage and be accountable for just as many tools and technologies.
And that is where the ownership gap comes into play. As Nicola Askham mentions in her blog: “As data piles up, so do the problems. Ownership confusion, unclear responsibilities, quality issues, and compliance risks.”
Why is this?
While data technology has evolved at an extremely rapid pace, our approach to handling and dealing with data has not.
In today’s modern data ecosystem, most companies have some fundamental underlying issues with how the organisation is structured and where data fits in. This isn’t to say that companies are doing things incorrectly; there are just some crucial concepts that have been overlooked.
I’ve listed out four of these below. Interestingly, they flow and are connected to one another, showing how interrelated they are when it comes the underlying challenges of data ownership:
Complex Data & Tech Stacks (including legacy tools) – There are dozens of data-producing technologies in any organisation, each with many users. Most of those technologies don’t have an owner, and neither does the data generated by them. The complexity further arises because none of these tools (especially legacy ones) communicate with one another or share the same definitions of things.
An Outdated Org Structure – Data and technology are never mapped out in organisational structures. Maybe it should be, maybe it shouldn’t be, but there should be some sort of accountability across teams for data and tech. Yet, usually there isn’t. Or it sits with business teams who have no desire to do anything with it, and because of data’s role as a siloed vertical, the data teams are useless in enacting any change.
No Linkage Between Data & Business Teams – As mentioned above, data teams are often siloed within an organisation, but data itself is not. Data feeds business teams and decisions, often flowing from a business-owned tool/ system into the data platform and back to the business team. The ownership, therefore, gets muddied in that transition, especially if data and business teams don’t talk or have an aligned data model.
No Accountability Processes – Building on the previous two points, the back-and-forth nature of data means that accountability is not often clearly defined. This can be established, but without clear processes for determining who is responsible for maintaining data quality, the likelihood of issues arising increases rapidly. And organisations don’t usually take the time to figure this out.
The end result is a lack of ownership and accountability across the organisation regarding data. Only when it is extremely clear are we sure who is responsible for managing and maintaining datasets in a clean state.
This isn’t an impossible task to deal with. A data governance strategy helps. Or by using metadata and assigning governance through that. We will discuss these topics further in subsequent articles.
However, most companies don’t have this yet, resulting in “data edited by many but owned by none,” which leads to a culture of finger-pointing when issues arise.
And why does this matter? Well…
…You can detect problems.
…You can enforce standards.
…But without accountability, there's no intrinsic motivation to maintain quality beyond the immediate task at hand.
And without ownership, nobody is ultimately responsible for the quality of data.
Ownership Embodied in Data Quality Platforms
Modern data quality platforms have evolved far beyond simple observability and monitoring functionality. And while you can’t solve ownership and accountability issues with just technology (it requires strong processes too), it helps clarify roles, automate notifications, and streamline processes to address data quality issues with those who are accountable.
Not too long ago (and probably still), tracking data ownership meant maintaining endless Excel spreadsheets. Teams would document dataset owners, manually update quality statuses, and struggle to keep information current. This usually fell apart after a few weeks/ months of updates.
Now, data quality platforms can integrate ownership directly into their architecture. Each dataset or table has an individual accountable, creating a system of record for data accountability. This metadata can be embedded in data modelling tools or platforms, providing visibility into ownership across the organisation.
And given the fact that data governance teams start with low resources, having some sort of automated ownership tracking reduces manual effort and frees up time for the one person in charge of Data Governance.
This comes in many different forms/ features:
Ownership Registries: Centralised, searchable databases of who owns what data assets, complete with contact information and responsibility boundaries. These would replace Excel spreadsheets that require manual updates.
Issue Assignment and Notification: Smart alerting that routes data issues to the appropriate owners based on lineage and domain knowledge. This directs responsibility to the right team and allows fixes to be implemented earlier, reducing blame across teams.
Accountability Metrics: Dashboards that measure how quickly data owners respond to issues and how well they uphold quality standards, allowing the organisation to set quality standards and track cleanliness for certain data domains
Data Contract Agreements: These formal commitments establish ownership and ensure adherence to defined quality thresholds, with specific Service Level Agreements (SLAs) and measurement criteria, between data producers and consumers.
Among all these things, the contract approach is probably the most effective, as it bridges the technical and business worlds by providing a framework that non-technical stakeholders can understand and commit to. Business leaders grasp the concept of "contracts" instinctively, making this model effective for establishing cross-functional ownership.
This is not the end-all, be-all. Having chatted with multiple people using Data Contracts, getting alignment from the business teams, and their ongoing participation, can prove quite problematic.
Nonetheless, these technology components do provide a basic foundational system to track ownership across datasets, all in an effort to maintain good data quality. The most significant gap is the alignment with business teams and who should own the different datasets.
This challenge results in the business not interacting with data quality platforms or tools, regardless of the effort invested. And this is where Data Governance comes into play!
Bridging to Data Governance
This section will serve as a teaser for Data Governance, as it deserves its own article (or three).
But the topic of Data Ownership is the perfect bridge into the domain.
Essentially, Data Governance is the organisational framework that makes data ownership sustainable and meaningful for the business. And while observability and contracts provide the technical infrastructure for ownership, governance provides the human and process guidance.
This domain dates back to the early 2000s, when companies realised that data was a valuable asset and needed to be governed, especially in light of data breaches, security concerns, and the rise of data privacy regulations.
Its roots, therefore, are more risk and compliance based, which is what Data Governance means to most business and data leaders still.
This is unfortunate because the data industry has evolved so much since the early 2000s. While data breaches, platform security and data privacy are still concerns, there is more to the governance of data than that!
Okay, so how does this all come back to ownership then?
Well, if we define Data Governance, it essentially comes down to how an organisation manages its use of data, using people, processes and technology to ensure individuals have the right level of access to and get the most from their data.
At its core, governance solves the ownership problem by establishing clear accountability structures around the organisation’s data. And while contracts or observability tools are helpful in delineating responsibility for quality management, they don’t get to the core strategic reasons for governing data and the human-oriented approach to ensuring business stakeholders use it to their benefit.
So, in a few words, why is Data Governance so important, especially when it comes to solving these constant ownership issues in data?
Strategy to Use Data – Data Governance and Data Strategy align pretty well. While Data Strategy sets the direction, Data Governance provides the approach to utilise data effectively across both business and data teams, ensuring it is done in a secure and compliant manner.
Clarifies Data Ownership Roles – Identification of data owners and data stewards creates accountability over data domains and datasets, allowing them to ensure that the data is being used properly by the business. This also ensures that interested stakeholders know who to contact when they want to use or access this data.
Bridging the Business Context – I may think of Data Governance (DG) differently than most; instead of focusing on compliance and risk mitigation, I view Data Governance as an enabler of data literacy and accessibility. With Data Owners and Stewards (outlined above), organisations can better "bridge the gap between business units and technical teams” because with their ownership, they can help enable the business to use data effectively. This is why, as a Data Strategist, I love it when organisations have a DG team—it is those individuals who help create a culture of data literacy and usage.
Enabled by Data Quality Tools – Someone (or a team) must be available to utilise these data quality tools and platforms strategically. Realistically, Data Engineering will only get so far, and they won’t think with the business in mind or do the strategic work necessary to determine data accountabilities. The DG team can (and should) establish the structure for ownership, which is then enforced by the data quality tooling or platform.
There are additional benefits to Data Governance, and we will delve into more detail over the next three weeks.
For now the thing to remember is that data quality will never be solved by technology alone!
And this is why our journey from detection to enforcement to ownership is inevitably leading to Data Governance. While observability helps you find problems and contracts help you prevent them, only governance can create the sustainable organisational structures that make ownership meaningful and lasting.
Next week, we'll begin our three-week deep dive into Data Governance Strategy. First, we will explore the role of Data Governance in an organisation. Then, we will get into thinking about governance strategically, outlining a framework to do this. Finally, we will discuss implementing it (which is quite challenging) and what you need to consider to do it successfully. Until then, have a great Sunday!
Thanks for the read! Comment below and share the newsletter if you think it's relevant! Feel free to also follow me on LinkedIn (very active) or Medium (increasingly active). See you amazing folks next week!
Someone has to be responsible for data quality, but the systems and culture has to mean everyone has to feel some responsibility to make sure the input/output you receive is actually useable
Inspiring. Agree with you that data governance is the right umbrella for data ownership. I am more lean to divide ownership into two levels, 1. data production and 2.data use (decision maker, the premium data consumer).