What Breaks When a Column Changes

The dangerous change to a data source is the small one that nothing rejects. A field gets a new name, a status gets added, prices switch from including VAT to excluding it. Every job still finishes and every dashboard still loads, and a number three steps further on is now wrong. The person who made the change had no way to see what depended on it.

A broken pipeline is the good outcome, because someone is told. This article is about the other outcome, and about the three things that prevent it: knowing what depends on a field before you change it, catching errors before they land in a report, and seeing when a source has stopped keeping up.

Three small changes, three wrong numbers

All three examples are hypothetical, and anyone who has run a marketing warehouse has met a version of each.

A field gets a new name. The store platform ships an update and the discount amount on an order arrives under a different name. The model that builds the orders table still looks for the old one. Depending on how the loader was written, it either fails, which is the lucky case, or it fills the old field with nothing. Net revenue is revenue minus discounts, so net revenue goes up. So does margin, and so does return on ad spend. A number that improves does not get investigated. It gets shared in the Monday meeting.

A status gets added. Operations starts shipping large orders in parts and introduces a status for partially shipped orders. The revenue definition counts orders that are paid or fulfilled, written out as a list of the statuses that existed at the time. Orders with the new status are in neither group. Revenue drops by exactly the share of orders that ship in parts, with no error anywhere, and the paid social team spends a week looking for a campaign problem that does not exist.

VAT is treated differently. A store that starts selling to business customers switches its price export from including VAT to excluding it. In the Netherlands the standard rate is 21 percent, so every euro of revenue in the warehouse is suddenly worth about 17 percent less than the day before (100 divided by 121). The finance report had its own VAT logic and removes the tax a second time. Two reports are now wrong in different directions, and the revenue gap between the store, GA4 and the ad platforms just got a fourth contestant.

Why nothing fails

Count the steps in the first example. The source field feeds the orders model. The orders model feeds the net revenue metric. The metric feeds a dashboard tile, a weekly report, a monitor with a threshold, and a customer segment built on order value. The change happened at step one and the wrong number surfaced at step four, in a tool owned by a different person.

Loaders and transformation jobs check that data arrived and that it has the expected shape. They do not check what it means. A discount of zero is a valid discount. An order with an unfamiliar status is a valid order. A price without VAT is a valid price. Every check that looks at one row at a time passes.

Know what depends on a field before you change it

Data lineage is the record of which tables and models each model reads, and which models, dashboards and reports read it in turn. Read upstream, it answers "where does this number come from". Read downstream, it answers the question that matters before a change: "what uses this".

Lineage kept by hand goes stale within a month. The version worth having is derived from the models themselves, so it is correct by construction, and it does not stop at the last table. A lineage graph that ends at the warehouse tells you which tables depend on a field. You also want the dashboards, the scheduled reports, the monitors and the targets, because those are what people look at.

An impact check is lineage used at the moment of change. You name what you plan to remove or alter, a field, a model or a metric, and you get back the list of things that would break. The best place for it is where the change is proposed. Teams that keep their models in a code repository can run the check on every proposed change, so the reviewer reads "this removes a field used by the margin dashboard and the weekly board report" next to the change itself.

The change still happens. It happens together with the two edits it requires downstream, or with a note to the owner of the report.

Catch errors before they land in a report

Validation belongs at two gates, and most teams only have the second.

The first gate is at the source, before data is mapped into the model. Data mapping is the step where a field from a source is tied to a field in your model: this one is the order date, this one the discount, this one the country. When the source changes, the mapping is what breaks. A good platform notices that a field it expects is missing or has changed type, shows what that field feeds, and asks for a new mapping before it carries on loading.

The second gate is at the model, before a change is saved as the version that runs. A draft should be checked for the things a machine can know for certain: that it reads only tables that exist and that you may read, that every field it names is still there, that it does not depend on itself through another model, and that the warehouse accepts it. BigQuery can do that last part as a dry run, which verifies a query and reports how much data it would read without executing it.

Neither gate knows what your business means by revenue. For that you need tests on the content: revenue is never negative, every order has a status from a known list, the daily total stays within a range of the same weekday last week. The status example above is caught by the second of those tests on the first day, as a failed check instead of a slow decline on a chart.

See when a source falls behind

A source that has stopped updating produces the same symptom as a bad change: a plausible number that is wrong. Yesterday's ad spend is missing because a token expired, the dashboard shows spend down by a third, and the first reaction is to look at the campaigns.

Freshness has to be visible per source, in the place where people read the numbers. "Meta Ads: last loaded Tuesday 06:10" answers the question before anyone asks it. A status page that only the data team opens does not.

The run status of each model belongs next to it. When one model fails, the dashboards built on it show a mix of today and yesterday, and seeing per model whether the last run succeeded explains that in a minute.

Thresholds on the business numbers are the last line. A monitor on net revenue, margin or cost per order tells you something moved, and as we argued in an alert without a why is just noise, it is only useful if the cause can be found. Lineage and freshness are how you find it.

What to ask of your own stack

Five questions, and the answers do not depend on which tools you run.

1. If I rename or remove this field today, can I get a list of the models, dashboards and reports that use it, in under a minute?
2. When a source changes shape, does loading stop and ask, or does it continue with empty values?
3. Can a change to a model be checked before it replaces the version that runs?
4. Does every source show when it was last loaded, where the people reading the numbers can see it?
5. Is there a test that would fail if a new order status appeared tomorrow?

In a stack assembled from separate tools, each of these has an answer, but the answers live in different products. The loading tool knows about source fields. The transformation project knows the lineage between models. The BI tool knows which dashboards exist. Nothing joins them, and the join is the part you need, which is one of the quieter items in the cost of a self built stack.

Where Verity fits

Verity runs this chain as one managed platform in your own BigQuery. The data models in your marketing data warehouse come with lineage: for every model you see what it reads, what reads it, and whether the last run succeeded. A model you add or change is a draft that is validated before it is published, and a person publishes it. Every source shows how fresh it is, and when a field in your own data changes, Verity shows what it affects and walks you through mapping it again. For teams that define metrics in dbt, an impact check reports which dashboards and monitors a proposed change would break before it is merged. The whole chain, from the tag on your site to the number on the dashboard, is described under data quality monitoring.

Frequently asked questions

What is the difference between a schema change and a logic change?

A schema change alters the shape of the data: a field is renamed, removed or gets a different type. A logic change keeps the shape and alters the meaning, such as a new order status or prices that no longer include VAT. Schema changes can be detected automatically when data arrives. Logic changes need tests on the content, because the data still looks valid.

How do I find out which dashboards use a field?

You need lineage that continues past the last warehouse table into the dashboards, reports and monitors. An impact check does the same search against recorded usage and returns the list.

Does validation before publishing prevent wrong numbers?

It prevents one class of them: a model that names a missing field or table, or that the warehouse rejects. It does not know whether revenue should include VAT. Use validation for what can be known for certain, and content tests and monitors for what depends on your definitions.