An organization may already have a data governance policy in place. It may already have a document for data inventory, its classification, its source systems, and its respective data owners. Yet the organization may still suffer from poor data governance, causing low trust in data. Why? Because those things may not be operationally connected.

The policy document may say sensitive data must be protected, but classification is not linked to access control to be seamlessly enforced. The governance committee may approve standards, but data assets get created faster than they are documented. The spreadsheet says a dataset exists, but analysts cannot find it, verify its lineage, or know whether it is safe to use. And data owners may be named in the spreadsheet, but incidents or any downstream concerns may not be routed back to the owners. Worst, there are no data owners.

The book Data Governance: The Definitive Guide argues that trust in data increases when (1) someone is accountable for the data, (2) the data are easily observable, and (3) the data are secure from unauthorized access, misuse, or corruption.

That’s Accountability, Observability, and Security. They encapsulates the capabilities the many tools and artifacts that is commonplace in data management and governance practices. Examples includes data contracts, metadata management, data catalog, data lineage, data quality tests, access controls, encryption, etc. Some contributes to more than one. Data lineage and data catalog, for example, not only improves observability by improving data discoverability; they also improve accountability by making data asset ownership more explicit and granular, closer to the logical data structure.

One way to model the interaction between these three capabilities is a self-reinforcing governance loop that continually builds trust in a systemic way:

That is to say, the relationship between these capabilities is cyclical: assigning ownership, with the proper incentives, establishes stronger accountability. It motivates data owners — through checks and balances such as data contracts and SLAs — to be accountable to the curation of metadata and improve documentation, making data easier to discover and observe, increasing observability. It also exposes a weak ownership structure if data issues are rampant. Better observability, in turn, enables stronger security controls and faster detection of misuse or quality issues. Those incidents can then be traced back to owners who should be held accountable — such as by institutional policy — reinforcing responsibilities and driving further improvements to observability and security. Thus, each cycle increases trust in the data.

In the world of digital data abundance, software plays a more important role in supporting this system, seamlessly connecting the three important capabilities in data governance, as I noted in DFCE 2025: Regulation, Innovation, and Friction. Yet paperwork may never be extinct. It’s still required for audits, legal accountability, or C-suite sign-offs, but it won’t be enough to operationalize data governance at scale. Beyond paperwork, however, software helps in making it easier to do the right thing with data, and difficult to do the wrong thing.

The Governance Loop on Software

Generally, data catalogs, used for metadata management, focuses on the observability capabilities (by ingesting technical metadata and curating them) and accountability (by assigning data owners to the ingested metadata). Security, meanwhile, is naturally integrated with the AI and data analytics platform, where data analytics or data engineering works — together with policy enforcements — happens.

To appreciate software’s role in data governance beyond bureaucratic paperwork, in this post I’ll use solutions such as OpenMetadata (as the data catalog) and Snowflake (as the AI and data analytics platform) as examples. Note, though, this isn’t a recommendation of the two solutions. Another powerful competitor to Snowflake is Databricks; and a good curated list of data catalog solutions can be found in the awesome-data-catalogs repository on GitHub.

Accountability

Documenting asset ownership usually means the following: Either an owner or an asset was identified first. An agreement then has to be made on who is accountable for the asset — an asset which should be valuable (otherwise why bother?) — to some stakeholders. Ownership usually lies where it is in his or her interest to keep the asset running, reliable, and sustainable (lest stakeholders can’t extract value and blame the owner).

In the corporate world, this is usually structured by some KPI to incentivize the asset owner to do the “right” thing with the asset. So far, this kind of incentive structure is straightforward and common. Examples abound, such as KPIs on incident resolution timeliness, data privacy compliance rating, external audit rating score, etc.

But generic KPIs are not enough. The data assets must also be measured against their intended use, so that dataset used for operational monitoring should have different definition of “good” data quality from dataset used in sales forecasts, for example. The owners are therefore not only accountable for IT-centric issues; equally important is accounting for data consumer needs — to ensure the data is fit-for-purpose and fit-for-use — which corresponds more strongly to the trust placed in data. Indeed, trust in data is not purely absolute (e.g., no duplicated records); it is also relative to the domain context and intended use (e.g., the records satisfies use case requirements).

In the world of data abundance, however, this effort of establishing the link between data owners (and their responsibilities) with their data assets has become cumbersome. The reason can be explained briefly in terms of the three V’s of big data: Volume, Velocity, and Variety¹.

Increasing use of digital workflows increases the rate of growth of organization’s data footprint. The sheer amount of data volume being generated per day — or even minutes — may overwhelm unprepared organizations answerable to regulatory policies. At the same time, the rate — or velocity — at which data are being created, captured, and processed also increases, each of which proliferates new data assets that stakeholders rely on. Problems and confusion arise when this rate of data creation and usage is faster than the organization’s capability to document, manage, and control its data assets. Finally, different use cases necessitate the adoption of different variety of data storage formats and data processing technologies, from simple fixed-schedule batch-processing of tabular formats to more complicated event-driven unstructured data processing. Combined, the large volume, high velocity, and wide variety of data assets can compound accountability problems.

This brings us to another related problem: How do you define the data ownership boundaries? Where does the line begin that delineates one owner’s assets from the others, so that there’s no ownership overlap? And how do you even begin to draw that line²? The complexity of these questions is further compounded by the three V’s above. High velocity means data ownership boundaries change frequently. Ownership then needs to be frequently reviewed and updated. That means the inventory to keep track of the increasingly growing data assets with wide varieties needs to be kept up-to-date as frequently, too.

You can’t beat this data governance problem in a world of data abundance with more bureaucracy and paperwork. Since you can’t manage what you can’t observe — and what you want to manage scales out quickly in every dimension — we need to increase our capability to observe at scale, both organization-wise (as seen with data ownership concerns) and technology-wise (along the volume, velocity, and variety dimensions).

Observability

The oldest observability mechanism in the book involves writing paper documentation of your data assets and manually performing inventory bookkeeping. That usually translates to cumbersome yearly or bi-annual (because any faster will be exhausting) cross-team efforts of surveying and interviewing different teams to understand the data landscape: What are the databases that are currently running? Which database contains sales data? Which ones contains PII data? Which database gets its data from which other databases? If a specific table is corrupted, which downstream stakeholders are affected? As you can imagine, this approach to observability means such information gets stale quickly, and doesn’t scale well when data footprints grow larger and quicker over the years. Answering each question above may take days or weeks, depending on your three V’s.

Despite the increase in data footprints, it’s not uncommon to see digital things being inventoried manually in an Excel sheets or Word documents. That should change. With regulatory requirements, for example, data catalog plays an important role for data observability.

To remain compliant with PDPO in Brunei, for example, an artifact called Data Inventory Map was introduced. The fundamental reason for its introduction is, of course, to ensure organizations and regulatory bodies have the capacity to observe data assets and their use. The following is an illustration provided by AITI — the supervisory authority behind PDPO:

It captures important metadata about data assets containing personal data: the purpose of collection, the retention period, where they’re stored, the legal basis behind the collection, etc. The problem with the Data Inventory Map isn’t the artifact itself; it’s about keeping a live, granular, and accurate metadata of the data assets.

This is where data governance platforms like OpenMetadata or DataHub comes in handy. They can automatically discover, extract, and ingest the technical metadata — logical and structural information about your data assets, like table schema, view definitions, and lineage — in your data landscape ³.

They generally work as follows. First, once they have read-only access to your data systems (e.g., Oracle and SQL Server RDBMS, or Snowflake data warehouse), they can extract the metadata information from the source systems. For example, on SQL Server, you can find this in the information schema views; on an Oracle database, the data catalog views; on Snowflake, the information schema (aka “data dictionary”) views. The good news is that instead of querying these views yourself — which usually isn’t meant for human consumption — platforms like OpenMetadata already have their own built-in technical metadata ingestion tools (called connectors) that reads from the catalog views and parses them further to enrich the metadata in the governance platforms. The caveat is that if there’s no built-in connector for a source system you want to ingest, you’d have to create a custom one.

Once the data governance platform ingests the technical metadata, the accountability-observability-security activities in the loop above become more streamlined and agile. Data assets inventory can be updated quickly in a matter of seconds or minutes, instead of weeks or months, greatly reducing the data inventory staleness problem. The bottleneck to governance is no longer on extracting information about data in systems, but extracting tacit knowledge in people.

The technical inventory of data assets is also more reliable and accurate because it’s inferred directly from source. Storing the inventory in the platform allows it to be indexed for search via fuzzy words matching or tags, which improves discoverability.

The technical metadata can further be curated with information useful for operationalizing data governance, such as data owners for each data assets, the business glossaries, classifications and tagging, and data quality status. While these tasks are traditionally done by human data stewards, future context-aware AI agents might help stewards speeds up tedious data curation exercises.

In short, the link between the governance contexts of the data — that is, the purpose of collection, legal basis, and intended recipients, etc. — and the technical data assets should be unambiguous and clear. Without data catalog, the link gets blurry, giving rise to ambiguity about data ownership and the actual scope of policy implementation.

With a more precise technical metadata structure, metadata management such as data labeling or classification can become less ambiguous, giving way to transparent traceability of data use and enabling precise policy enforcement, and thus — as the loop suggests — enabling better data security.

Security

If you can’t manage what you can’t observe, it goes without saying that you can’t secure what you can’t manage. In the context of data governance, securing data goes beyond the prevention of data exfiltration and data misuse. It also extends to include data corruption: the degradation of data integrity, such as inaccurate data updates, or simply technical data consistency failures (remember ACID?). Data corruption hence deserves a similar treatment as the former two, because violating any of them can cause considerable harm to the organization.

All these can be done by internal or external actors, whether it was done with malicious intent or otherwise. This is why observability is the key enabler for securing data. With the metadata ingested and properly curated, we know what data exists across our data landscape and how it is classified or labeled (e.g., PII, Confidential, Public, etc.). This observability capability allows organizations to better plan and be more proactive to establish the appropriate level of monitoring and controls to right data with the right security controls.

Let’s expand on how observability through metadata management enables security across the three security controls introduced above. The first is the most obvious one, which is to prevent data exfiltration by malicious actors. This means you want to safeguard access to your data systems, such as your enterprise data warehouse, to the right people with the right level of privilege. To ensure the control is in place, not only you should be able to know who can access your data systems, it is also important to monitor their usage to alert suspicious activities (a sudden overnight spike in database queries by a single user outside office hour should raise some suspicion); usage are, after all, metadata that can be ingested by data governance platforms. Naturally, the access control safeguard happens at the data systems level (e.g., SQL Server, Oracle, and Snowflake) — not the data governance platform such as OpenMetadata or DataHub. How data governance platform complements access control, however, is by making it easier for organizations to design controls commensurate with the classification of data attached to the ingested metadata.

The second control is relatively harder to manage, which is to control against data misuse. For example, you may have submitted an online survey form containing your personal email — with your consent in a checkbox — to allow the researchers to contact you for further inquiries about the research survey. However, privacy laws may prohibit the researchers from using your email for commercial purposes — as that would have been a misuse of your personal data (and a betrayal to your trust!)

Purpose-based access control is harder to manage because of low traceability of data usage from data access provision. Mostly, you’ll find the clues about the purpose the data access — and hence its usage — in some email trails, such as emails about data engineers requesting read-only access to a specific table. Indeed, in most cases, purposes are not a first-class citizen in data platforms. Often, they are only weakly inferred from the combination of usernames and roles, and the grants applied to them.

To tackle this issue, OpenMetadata allows us to represent purpose as metadata with the use of tags or classification attached to catalogued data assets. This can be made even tighter on the data systems level: On Snowflake, for example, access to data is restricted by roles defined not by job description (e.g., SALES_ANALYST), but also by purpose-specific roles (e.g., PURPOSE_REVENUE_REPORTING, PURPOSE_OPERATIONS_MONITORING). These purpose roles can then be combined with row-level access policies, masking policies, and other constraints so that the same data is exposed differently depending on the approved purpose of use.

The third control on data corruption, is one that’s often overlooked in the context of data security. Corrupted data may impair decision-making too, and, in the case of AI/ML models, makes them unreliable. This can be damaging to the organization. Again, observability helps. To tackle data corruption issues, organizations should catch data errors as early as possible, ideally within a defined SLA commensurate with the criticality of the data asset. This is usually done by running queries or scripts — often on schedule — that tests data assets for the expected data profile or quality. This can be as simple as rigid rule-based checks, such as looking for blank values, or as complex as regression analysis using machine learning models.

OpenMetadata, for example, has features that allow data stewards to define and run data quality assertions from its web UI. Test failures notify the stewards — or whoever is interested to subscribe to the notification — when quality assertion fail. If the data asset catalogued in OpenMetadata has an owner (as it should!) stakeholders would know who is accountable to move their team to investigate and fix the reported data issue. As discussed above, this is only possible because of accountability that drives the organization to this level of increased observability.

Remediation that follows the violation of the controls above feeds into accountability, as discussed earlier, in the form of a governance loop. We’ve seen how observability enables security across the three controls, thus increasing security capability. These controls force organizations to answer data ownership scrutiny. Who defines the quality metrics? How do you guarantee SLAs? Are your data consumer needs met? These questions, at least in theory, reinforce better accountability, and in turn, their incentive to do better on observability and security.

Conclusion

Data governance is ultimately a systems problem whose objective is to increase trust in data: Trust that the policy intents are applied where and when they should be applied; trust that data is discoverable, understandable, and fit for intended use; trust that sensitive data is protected from misuse, exfiltration, or corruption; and trust that someone is accountable when something breaks.

By modeling this problem using three underlying reinforcing capabilities — Accountability, Observability, and Security — we can reason and explain why their connectedness is important for operationalizing data governance to increase trust. Without closing the connection gap between them, ambiguity rises, accountability becomes blurry, and conformance checks become more costly to conduct. As we’ve seen, data catalog, by automatically ingesting technical metadata, reduces ambiguity that arises when the data asset inventory does not reliably capture (if at all) granular technical metadata information.

Indeed, data governance succeeds when the system — an amalgamation of people, process, and technology — makes doing the right thing with data easier than doing the wrong thing.