Why AI Gets Your Numbers Wrong, and How a Semantic Layer Fixes It

When an AI answers a business question with a wrong number, the model is rarely the weak link. Current models write better SQL than most of the people reviewing their output. What they lack is context: nothing in a raw warehouse schema says what revenue means, which table sits at which grain, or that finance quietly excludes test orders from every report. The model fills those gaps by guessing, and a guess rendered as clean SQL looks exactly like knowledge.

The fix is a semantic layer. This article defines the term, walks through the four failure classes that account for most wrong AI answers, and shows how each one disappears when the model queries definitions instead of raw tables.

What a semantic layer is

A semantic layer is a machine-readable contract between your data and anything that queries it. It defines every metric once (net revenue is the sum of paid order subtotals minus refunds, excluding test orders), names the entities (order, customer, session, campaign), fixes the grain of each (one row per order, one row per session), maps business vocabulary to physical columns, and records the joins that are legal between them. BI tools have shipped versions of this idea for decades under names like metrics layer or governed dataset. What changed is the consumer: a semantic layer used to make dashboards consistent, and now it is the thing that keeps an AI honest, because the model requests "net_revenue by month" and the layer compiles the SQL from the definition. The model never gets the chance to improvise.

Without one, the model reads table and column names and infers meaning. That works on a demo schema. On a two-year-old warehouse it produces four specific kinds of wrong.

Four ways AI gets your numbers wrong

Ambiguous metric names

A typical ecommerce warehouse holds Shopify's total_price, subtotal_price, and current_total_price, GA4's purchase_revenue, Meta's attributed conversion value, and a few half-retired revenue columns in old models. All of them are revenue to a language model. They can differ by 30 percent or more, for reasons we unpacked in why Shopify, GA4, and Meta all report different revenue. The model picks one, and the answer inherits whichever definition it happened to land on. Ask twice on different days and you can get two different numbers for the same month.

Wrong grain and fan-out joins

Join an orders table to its line items and every multi-item order appears once per item. Sum a header-level column like total_price over that join and you double or triple count. The SQL is valid, the result is inflated, and nothing errors. Models fall into this constantly because the schema does not declare grain anywhere they can read it.

Time and timezone semantics

"Last month" is a policy decision. Shopify timestamps are UTC, the GA4 export uses the property timezone, and your finance team closes the books on Europe/Amsterdam time. An order placed at 00:30 on July 1 in Amsterdam lands in June in UTC. Add fiscal calendars, and "last month" has three defensible interpretations. The model will pick one without telling you which.

Silent filters

Every mature reporting stack has conventions that live in people's heads: exclude test orders, exclude internal traffic, exclude wholesale, net out refunds. None of them are visible in the schema. An AI querying raw tables applies none of them, so its numbers run consistently high compared to the reports the business already trusts, and it takes a painful reconciliation session to find out why.

What the benchmarks say

The public research points the same direction. On Spider, the long-running academic text-to-SQL benchmark with clean, small, well-named schemas, leading models have scored high for years. On BIRD, a newer benchmark built on larger, messier databases with dirty values and required external knowledge, published accuracy drops sharply and still sits well below human performance. Exact numbers shift with every model release, so treat any specific figure as dated the day it is printed. The stable finding is the gap itself: the same models, moved from clean schemas to realistic ones, lose a large share of their accuracy. And a production marketing warehouse is messier than BIRD.

The same question through a semantic layer

One concrete question shows the failure and the fix: "what was net revenue last month?"

Against raw tables, a model produces something like this:

SELECT SUM(o.total_price) AS revenue
FROM `acme.shopify.orders` o
JOIN `acme.shopify.order_lines` l ON l.order_id = o.order_id
WHERE o.created_at >= '2026-06-01'
  AND o.created_at < '2026-07-01'

Wrong metric (gross incl. tax and shipping), fanned-out join, UTC month boundary, no test-order or refund handling. Four failure classes in six lines.

Against a semantic layer, the model does not write this SQL at all. It issues a request:

metric: net_revenue
grain: month
period: 2026-06

and the layer compiles the query from the governed definition:

SELECT
  SUM(o.subtotal_price) - SUM(COALESCE(r.refund_amount, 0)) AS net_revenue
FROM `acme.core.orders` o
LEFT JOIN `acme.core.refunds` r ON r.order_id = o.order_id
WHERE o.financial_status = 'paid'
  AND o.is_test = FALSE
  AND o.created_at_local >= TIMESTAMP('2026-06-01', 'Europe/Amsterdam')
  AND o.created_at_local < TIMESTAMP('2026-07-01', 'Europe/Amsterdam')

Each failure class dies at a specific point. Ambiguity dies because net_revenue resolves to exactly one definition. Fan-out dies because the layer knows the grain of orders and refuses illegal aggregations across joins. Timezone dies because the calendar policy is part of the definition. Silent filters die because they are no longer silent; they are written down where the compiler applies them every time. The model's job shrinks to the part it is actually good at, understanding the question, and the arithmetic becomes deterministic.

Raw schema access versus semantic layer access

What natural language BI actually means

Conversational BI, also called natural language BI, means asking business questions in plain language and getting answers computed from your governed data. The definition matters because two very different products wear the label. One is a chatbot bolted onto a schema: it writes SQL from raw tables and carries every failure class above. The other is a chat interface in front of a semantic layer, where the language model handles the conversation and the layer handles the math. The semantic layer is the entire difference between the two. If you are evaluating a tool in this category, one question settles it: when I ask for revenue, where does the definition of revenue live? "The model figures it out" is the wrong answer.

Where Verity fits

Verity includes the semantic layer as part of the platform: SQLMesh-modeled data in your own BigQuery, a business glossary on top, and Data Chat answering questions through those definitions rather than around them. The same layer is exposed to Claude and other models through a managed MCP server, which is the architecture we recommend even to teams building it themselves. For how that plays out in production, see Connecting Claude to BigQuery: What Works in the Demo and Breaks in Production.

Frequently asked questions

Is a semantic layer the same as my dbt or SQLMesh models?

No, but they are adjacent. Transformation models produce clean tables; a semantic layer defines what the metrics computed from those tables mean, at which grain, with which filters, and exposes that as a queryable contract. Well-modeled tables make the layer easier to build. They do not replace it, because a model can still aggregate a clean table wrongly.

Will better models make semantic layers unnecessary?

Unlikely. No amount of model intelligence can recover a definition that exists only in your finance team's heads. Smarter models guess more plausibly, which arguably makes unreviewed wrong answers more dangerous, not less.

How much work is it to build one?

For a focused marketing warehouse, the first useful version is small: your ten core metrics, five entities, and the joins between them, written down and enforced. The real cost is organizational, getting finance and marketing to agree on one definition of revenue. That argument is worth having anyway.

Does this apply to tools other than chat?

Yes. Dashboards, scheduled reports, and AI agents calling your warehouse through MCP all inherit the same guarantee: one definition per metric, applied everywhere. Chat is just where the absence of a semantic layer hurts fastest, because nobody reviews the SQL.