Data quality monitoring: the blind spot above your warehouse
- Published
- 8 min read
In short: data quality monitoring for analytics means continuously checking data reliability across three layers: collection (dataLayer, tags), transport (hits, server-side, CDP) and storage (the data warehouse). Nearly every tool on the market covers the third layer. The most expensive errors are born in the first.
Here's a scene I've watched play out more than once. A data team has a clean warehouse, dbt tests on the critical models, freshness alerts, often a proper data observability tool on top. Everything is green. And the revenue figure on the dashboard still doesn't match the back office. Nobody can explain it, because nobody is looking in the right place. The data isn't broken. It's wrong. Those are two different things, and that difference is the whole subject of this article.
What is data quality monitoring for analytics?
It is the continuous, automated checking of data reliability, as opposed to a point-in-time review. The distinction matters. An audit tells you your data was fine on the day of the audit. Monitoring tells you it stopped being fine on Tuesday at 2pm.
The field conventionally covers six dimensions borrowed from the wider data quality discipline: accuracy, completeness, consistency, freshness, uniqueness and validity. The one to hold on to is the split between validity and accuracy. Valid data respects the expected format and rules. Accurate data correctly describes reality. The tooling market is excellent at the first. Almost nobody addresses the second.
A note on vocabulary, because the terms get used interchangeably and shouldn't be. Data observability refers to the tools that monitor pipelines and tables inside a data warehouse: Monte Carlo, Soda, Elementary, Anomalo, or dbt's native tests. Data quality monitoring in the broad sense includes that, but is not limited to it.
How can data be valid and wrong at the same time?
Because validity is checked against a schema, and accuracy against the real world. A schema has no idea what should have happened.
Take an ecommerce site where, after a checkout redesign, the purchase event stops sending the transaction amount. The destination table keeps filling up. Rows land on schedule, the revenue column exists, it's correctly typed as numeric, it isn't null. It contains zero. The freshness test passes. The not-null test passes. The type test passes. And your analytics revenue has been wrong for three weeks.
Another example, equally mundane: two GTM containers deployed on the same site, so every page_view is counted twice. Volume doubles. No conventional quality test fires, because no test says "there shouldn't be two of these". If anything, a volume alert tuned to catch drops will read the increase as good news.
The nastiest case is the items array going out empty to GA4 after a migration. The structure is there, the event fires, the row exists. It's simply hollow. Ecommerce revenue is understated, and because the drop is gradual rather than abrupt, it reads as a business trend rather than a bug.
Three errors, zero alerts. And these three are only a sample: I've written up seven of them, each with its symptom and its fix, in the data layer errors that skew your GA4 data. That's why quality monitoring can't be reduced to schema tests.
Which layers does data quality actually live on?
There are three, and they have neither the same failure modes nor the same tooling. If the notion of a data layer isn't familiar, that's what the first one is about: the data structure your site exposes for tags to read from.
| Layer | What happens there | Typical failures | Who monitors it |
|---|---|---|---|
| Collection | dataLayer, tags, SDKs, consent mode | Event stops firing, empty parameter, duplicate, regression after release | Almost nobody |
| Transport | Hits to GA4, media pixels, server-side, CDP | Hit sent but never received, broken mapping, loss in the server-side hop | Rarely |
| Storage | Data warehouse, tables, dbt models | Freshness, volume, schema, row duplicates | Well tooled |
The key point: a layer 1 error travels intact all the way to layer 3, and becomes undetectable once it gets there. The warehouse cannot know an event is missing, because a missing event leaves no trace. It's an absence, and absences are hard to monitor.
A layer 3 error, by contrast, is loud. A broken dbt model, a join that explodes, a pipeline that fails: you find out fast. Which is precisely why the tooling grew there and not elsewhere. We built tools where problems made noise, not where they cost money.
What does data observability cover, and where does it stop?
Let's be fair, because this is not an argument against data observability. These tools do exactly what they were designed to do, and they do it well: catching a table that didn't refresh, an abnormal row count, an unannounced schema change, a spike in null values. On complex pipelines with dozens of dependencies they're essential, and I don't know a mature data team operating without something in that category.
The limitation is categorical, not qualitative. These tools reason about what arrived in the warehouse. They compare the present to the past and look for breaks. That leaves two blind spots.
The first: they don't know your business specification. They know you used to receive 10,000 purchase events a day and now receive 6,000. They don't know you should have received 10,000, or why some are missing, or where. The alert says "this is different", not "this is broken, here's the offending event".
The second: they arrive after the fact. Between the release that breaks tracking and the detection of a volume anomaly, you're usually looking at several days, the time it takes for the gap to become statistically visible. On a partial degradation, think weeks. Meanwhile the data is feeding decisions and media spend.
A data observability tool is an excellent seatbelt. It isn't a pre-flight check.
How do you monitor the collection layer?
Three approaches exist, and they are not equivalent.
Manual checks before release. This is where everyone starts: someone opens GTM preview or a debug extension, replays the funnel and verifies. It works, and it does not scale. A serious ecommerce funnel is 40 to 60 events, multiplied by consent states, by mobile and desktop. Nobody redoes that on every deployment, and the QA pass becomes a ritual that gets skipped the moment the schedule tightens.
In-house automated tests. A Playwright or Cypress suite that replays journeys and asserts on dataLayer contents. Technically this is the right instinct. The real cost isn't writing them, it's maintaining them: every CSS selector change, every UI redesign, every renamed event breaks tests. And a test suite that fails for the wrong reasons eventually gets muted. I've seen that ending more often than the other one.
Dedicated platforms. Production monitoring, scheduled crawls, scripted tests in CI/CD, or agents that build the test plan from your tracking plan. The approaches differ substantially, and the right one depends on what you're actually trying to get: an alert when production drifts, a governance audit at scale, or a blocking QA gate before release. We went through those differences in detail in our comparison of automated dataLayer QA tools.
Whichever route you take, one rule holds: the reference has to be your tracking plan, not your traffic history. Comparing the present to the past tells you something changed. Comparing the present to the specification tells you something is wrong. Those are different jobs.
Where should you start from zero?
Don't chase full coverage. It's discouraging and you won't finish. Take this order instead.
- Identify your five critical events. The ones a decision or a budget depends on. In ecommerce that's usually
purchase,add_to_cart,begin_checkout, a user identifier and consent state. Five, not fifty. - Write the expected specification for each. Name, required parameters, value formats. If that document doesn't exist, you have nothing to monitor against. You have habits. This is the prerequisite, not an optional step.
- Institute a check before every release, manual at first if that's what's realistic. What matters is that it's systematic and written down, not that it's sophisticated.
- Automate what repeats. The moment the manual check gets skipped twice for schedule reasons, you have your signal.
- Route the alert to people who will act on it. An alert landing in a channel nobody reads is worth no more than no alert. And an alert with no context, one that says "anomaly detected" without saying what or where, gets ignored after the third occurrence.
Point 5 is the one most teams underinvest in. The value of monitoring isn't detection. It's the action that follows.
Wrapping up
Analytics data quality doesn't live where we've tooled for it. We monitor the warehouse heavily, because that's where problems make noise, and we leave the collection layer bare, which is where the most expensive and most silent errors are born.
If you take one thing away: your quality tests confirm the data is well formed. They don't confirm it's true.
Closing that blind spot is exactly what MayIA° does: agents that replay your user journeys and check every event against your tracking plan, before and after each release. Request a demo.
FAQ
What's the difference between data quality and data observability?
Data quality is the general discipline, covering accuracy, completeness, consistency, freshness, uniqueness and validity. Data observability refers to a family of tools applying some of those principles to the pipelines and tables of a data warehouse. Observability is a tooled subset of data quality, focused on the storage layer.
Are dbt tests enough to guarantee data quality?
No, and that isn't their job. dbt tests validate what's in the warehouse: uniqueness, not-null, accepted values, referential integrity. They cannot detect an event that was never collected, because nothing arrives in the table to signal it.
How do you catch a tracking regression before it hits your reports?
By checking the collection layer at release time, not downstream. Concretely: replay the critical journeys and check every event against the tracking plan specification before the code reaches production. Downstream detection, through a volume anomaly, arrives days later at best.
How long does a tracking error typically go unnoticed?
No market study answers this, and be sceptical of any figure quoted without a methodology. What we measure across our own portfolio, written up in our dataLayer regressions benchmark: two weeks or more on average, except on purchase events, which get fixed within hours. Detection almost always comes through an indirect channel, a discrepancy against the back office or an analysis that doesn't add up.
Do you need a tracking plan to do quality monitoring?
Yes, it's the prerequisite. Without a written specification of what should be collected, a tool can only compare the present to the past and flag variations. It cannot tell you whether collection is correct, only whether it changed.