Incremental Loads and BigQuery Costs for Marketing Data

Loading only what is new is the cheapest way to keep a marketing warehouse up to date, and on its own it gives you wrong numbers. Marketing data keeps changing after it has been loaded. Ad platforms add conversions to days that are already in your warehouse, and orders get returned weeks after they were placed. A good load fetches what is new plus a window of recent days that may have changed, and stores the result so that BigQuery only reads the days a query asks for.

This guide covers what to fetch again and what to leave alone, why a lookback window is the price of loading incrementally, and what partitioning and clustering do to the bill. For the prices themselves and a worked monthly bill, read what BigQuery actually costs for a marketing team. This guide is about the loading habits that keep that bill small as your history grows.

Full refresh and incremental load

A full refresh fetches everything from the source on every run and replaces what was in the warehouse. An incremental load fetches only what was added or changed since the previous run and adds it to what is already there.

A full refresh is fine for small tables that change everywhere, such as a list of campaigns, a product catalog or a currency table. It is the wrong default for anything with a date and years of history. There is a third variant, append only, which fetches new days and never looks back. It is right for data that truly never changes once written, which in marketing is rarer than it sounds. Everywhere else it produces numbers that are quietly too low.

Marketing data changes after it lands

Three kinds of source data keep moving, each for a different reason.

Ad platform conversions. Google Ads has always reported a conversion on the date of the ad click, by its own description, and the purchase may come days or weeks later. The conversion window, the period after a click in which a conversion still counts, can be set anywhere from 1 to 90 days and is 30 days by default. So the conversions and conversion value for the first of the month can still rise at the end of the month. Other ad platforms attribute conversions in a comparable way. If you load each day once and never look back, your warehouse permanently understates recent performance, and the gap is largest for the products people think about longest.

Analytics exports. The GA4 export to BigQuery writes one table per day, and Google states that it keeps updating a daily table for up to two calendar days beyond its date, and occasionally later. A model that reads yesterday's table at six in the morning and never again has read an unfinished day.

Orders. An order is not finished when it is placed. It gets paid, shipped, partly returned, refunded, sometimes cancelled. A return in March on a January order changes January's net revenue, January's margin, and the value of that customer. Unlike an ad conversion, there is no fixed window: a return policy of thirty days still produces refunds on day forty five when customer service makes an exception.

Choose the lookback window per source

A lookback window is the number of recent days an incremental load fetches again on every run, on top of what is new. Each run replaces those days in the warehouse with the current version from the source.

The right length follows from how the source behaves, so one number for every source is the wrong approach.

For orders, a window on the order date is the wrong tool. Ask the source for every order that was modified since the last run, whatever its order date. That picks up the March return on the January order without fetching three months of orders every night.

For ad platforms, check what the account's conversion windows are set to before choosing. A 30 day lookback against a 90 day conversion window leaves conversions outside the window that never reach the warehouse. And when someone changes attribution settings or uploads offline conversions for older dates, run a one time reload of the affected period. A scheduled full reconciliation, monthly or quarterly, catches whatever the window missed.

What BigQuery charges for

The facts in this section and the next two come from Google's BigQuery pricing page and its cost best practices, read on November 13, 2026. Prices are list prices in US dollars on that date, for the Europe (eu) multi region unless stated otherwise. They differ per region, so check the page for yours.

Loading is the cheap part. Google does not charge for batch loading data into BigQuery when it uses the shared pool of capacity. What you pay for is compute, the work of running queries, and storage, the data at rest. Each has two billing choices, and both interact with how you load.

Compute: on demand or capacity pricing

Every query uses compute, and that includes the nightly job that rebuilds your reporting tables after a load. BigQuery bills it in one of two ways.

On demand charges for the amount of data each query processes. The first 1 TiB per month is free and the list price beyond that is $6.25 per TiB in the EU multi region. Single regions can be higher: the Netherlands region (europe-west4) lists at $7.50 and Frankfurt (europe-west3) at $8.125. Nothing is billed while nothing runs. This is the default for every project.

Capacity pricing charges for processing power over time instead of data read. The unit is the slot, a virtual CPU, and you buy it through one of three BigQuery editions: Standard, Enterprise and Enterprise Plus. Capacity can scale automatically with the work, and according to Google's autoscaling documentation it does so in multiples of 50 slots, billed per second with a one minute minimum by default, and you are charged for the slots that were scaled up, not for the slots a query happened to use. The Standard edition lists at $0.04 per slot hour in the Iowa region (us-central1); slot prices also differ per region. Enterprise and Enterprise Plus add a baseline of always available slots and one or three year commitments at a discount.

For a webshop with a few sources, a nightly run and a handful of dashboards, on demand is the right choice, and usually the whole month fits inside the free TiB. A calculation shows why capacity is the wrong fit at that size. It is an example at the Iowa list price, not a quote: with autoscaling, a single dashboard refresh that wakes up 50 slots is billed for at least one minute, which is 50 slots times $0.04 divided by 60, about three cents. Two hundred refreshes spread over a day come to roughly $6.70, every day, for work that on demand would have done for free.

The other direction, also as an example at those list prices: $6.25 buys one TiB on demand or about 156 slot hours on the Standard edition. A team that processes many TiB every month with queries that finish quickly can get more work out of the same money with capacity.

A marketing or ecommerce team has a reason to switch in three situations. The on demand bill is consistently higher than a small reservation would cost. Finance wants a hard ceiling, which capacity gives because spend cannot exceed the maximum number of slots you set. Or one heavy workload, such as an hourly rebuild of event data, dominates the bill. You do not have to move everything: Google lets you use editions and on demand side by side, per project, so the heavy pipeline can run on capacity while ad hoc analysis stays on demand.

The billing model changes what partitioning and clustering buy you. On demand, every byte a query skips comes straight off the bill. On capacity pricing there is no charge per byte, so skipping data shows up as queries that finish faster and need fewer slots, which lets you run on a smaller reservation. The habits in the rest of this guide pay off under both models, but only on demand shows the saving as its own line.

Storage: active or long term, logical or physical

Storage has two distinctions, and they are independent of each other.

Active and long term

A table or partition that has not been modified for 90 consecutive days moves to long term storage on its own, at a lower price. In the EU multi region, active logical storage lists at $0.02 per GiB per month and long term at $0.01, exactly half. The discount is smaller in some single regions: Frankfurt and Iowa both list $0.023 and $0.016. Nothing changes in speed or availability. Any modification resets the clock.

Google considers each partition of a partitioned table separately, and that is where loading habits come in. Take a webshop with five years of order and ad history. If the tables are fully replaced every night, every byte is modified every night and all five years are billed at the active rate forever. If the tables are partitioned by date and loaded incrementally with a 30 day lookback, only the most recent days are rewritten. Everything older than about four months sits at the long term rate, which for five years of history is nearly all of it.

Logical and physical billing

This choice decides how the size of your data is measured. Google's documentation on datasets describes two storage billing models. Logical billing, the default, charges for the uncompressed size of the data. Physical billing charges for the compressed size on disk. You set the model per dataset, and after a change you have to wait 14 days before you can change it again.

Physical billing has a higher price per GiB. In the EU multi region, active physical storage lists at $0.044 per GiB per month against $0.02 for active logical, and long term at $0.022 against $0.01. That is 2.2 times as much. The ratio is not the same everywhere: Iowa lists $0.04 against $0.023, about 1.7 times. Physical billing also counts bytes that logical billing includes for free. BigQuery keeps changed and deleted data for a time travel window of seven days by default, adjustable down to two, plus a fail safe period of another seven days. Under logical billing that retained data is part of the base rate. Under physical billing it is charged separately at the active rate.

So in the EU multi region, physical billing is only cheaper when the compressed data, with the time travel and fail safe bytes added, is less than 1/2.2 of the uncompressed size, which is under 45 percent, and the tables are not rewritten often. Marketing data with many repeated values, such as channel names, countries and campaign names, tends to compress well, but measure it for your own dataset before switching. Physical billing is more expensive for tables that are replaced in full every night, because each replaced copy stays billable through the time travel and fail safe windows. This is the cost Google warns about when it advises against overwriting tables frequently and recommends loading incrementally instead.

For a small webshop the storage bill is low under either model, and the first 10 GiB per month is free. Leave the default alone until storage is a visible line on the invoice, and make sure the tables are loaded incrementally before you consider physical billing.

Partitioning: read thirty days, not five years

Think of a table as a filing cabinet with all your daily ad results in one drawer, in no particular order. To answer "what did we spend in the last thirty days", someone has to go through every sheet in the drawer.

Partitioning gives every day its own drawer. In Google's words, a partitioned table is divided into segments, called partitions, and when a query filters on the partitioning column BigQuery scans the partitions that match and skips the rest. Skipped partitions are not billed.

Take a dashboard that shows the last thirty days of spend and revenue, on a table holding five years of daily data. Five years is about 1,826 days. Without partitioning, every refresh of that dashboard reads all of them and uses thirty. With the table partitioned by date, it reads thirty drawers, under 2 percent of the table if the days are of similar size. The dashboard shows the same numbers and nobody opening it notices a difference. The difference shows up in the bytes billed, on every refresh, for every viewer.

Partitioning serves the load as well as the dashboard. An incremental run with a 30 day lookback replaces thirty partitions and leaves the other 1,796 untouched, which is also what lets them age into long term storage.

Two details from Google's documentation on querying partitioned tables matter in practice. The date filter has to be a fixed value or a constant expression. If the dates are looked up from another table inside the query, BigQuery cannot skip partitions. And you can switch on a setting that requires a filter on the partitioning column, so a query without a date range is refused instead of billed.

Clustering: sort the sheets inside each drawer

Partitioning decides which drawers are opened. Clustering decides how much of an opened drawer has to be read.

A clustered table keeps its data sorted by the columns you choose, up to four. If the sheets in each daily drawer are sorted by channel and then by country, a question about Meta Ads in the Netherlands goes straight to that section. BigQuery uses the filter in the query to skip the blocks of storage that cannot contain matching rows, and it keeps the sorting up to date in the background at no charge as new data arrives.

Cluster on the things your reports filter on most. For marketing tables that is usually channel, country, store or account, and for customer tables the customer.

One drawback. With partitioning, BigQuery can tell you before a query runs how much it will read. With clustering it cannot give an accurate estimate in advance, because how many blocks it can skip is only known when the query runs. Google's own advice is to prefer clustering when partitions would be small, roughly under 10 GB each, or when you need finer detail than partitioning allows, and to combine both when you want the estimate and the sorting.

A daily run, start to finish

Here is the whole pattern for the ad data of a store that keeps five years of history, described as a hypothetical example.

Each night the load fetches yesterday plus the previous thirty days from each ad platform, because the account's conversion window is thirty days. Those thirty one days replace the same days in the raw table, which is partitioned by date. The orders load asks the store for every order modified since the last run and updates those orders, whatever month they belong to.

The reporting tables are then rebuilt for the days that changed, not for all of history. A summary table with spend, revenue and orders per day, channel and country is partitioned by date and clustered by channel and country. The dashboard reads that summary table with a date filter, never the raw data.

Once a month a wider reload covers the last ninety days, and after any change in attribution settings the affected period is reloaded by hand.

The result is a nightly run that takes the same time in year five as in year one. The dashboard reads thirty partitions of a small table, and the numbers for last month still move when the ad platforms move them.

Questions to ask about any pipeline tool

Reviews of pipeline software tend to compare the number of connectors. For marketing data, how a connector loads matters more than how many there are. Ask these of any tool, including one you built yourself:

• Does it fetch recent days again for ad platforms, and can the window match your conversion window?
• Does it pick up changes to existing orders, such as refunds and returns, or only new orders?
• Does it write partitioned tables into your warehouse, or one growing table?
• Are the reporting tables rebuilt incrementally, or from scratch on every run?
• Is the price based on rows touched? If so, every restated conversion counts again each month.
• Can you reload a period yourself when settings change?

The last two decide the surprises. Pricing per row touched and a long lookback window work against each other, and that tension is worth understanding before you sign. Our comparisons of Verity and Fivetran for marketing data and of Verity and Supermetrics cover where a specialized loading tool is the better choice, which it is when you need a very large connector catalog or a warehouse other than BigQuery.

Where Verity fits

Verity is a managed platform that includes the loading, so these choices are made and maintained for you. Managed marketing ETL loads your store, ad, analytics and email sources into your own BigQuery, incrementally where a source needs it, and works with partitioning and clustering in the tables it builds. The models on top live in your marketing data warehouse, in a Google Cloud project you own, so the BigQuery bill is yours to read line by line. Verity works with a focused set of sources and does not compete on connector count; a connector that is missing is built in house for a new customer. More on the warehouse side under Verity and BigQuery, and on keeping the loaded data trustworthy under data quality monitoring.

Frequently asked questions

What is an incremental load?

An incremental load fetches only the data that was added or changed since the previous run and adds it to what is already in the warehouse. The opposite is a full refresh, which fetches everything again each time. Incremental loads keep run time and query costs flat as history grows.

How long should the lookback window be for ad platforms?

At least as long as the longest conversion window used in the account. Google Ads allows conversion windows from 1 to 90 days, with 30 days as the default. If your window is 30 days, fetch the last 30 days again on every run, and plan a wider reload now and then for anything that arrived later.

Does loading data into BigQuery cost money?

Google does not charge for batch loading data using the shared pool of capacity. You pay for storage once the data is in, and for the queries that turn it into reporting tables. That is why the way you rebuild tables matters more for the bill than the load itself.

Should a small team use on demand or capacity pricing?

On demand. It has a free first TiB of query data every month, bills nothing while nothing runs, and needs no setup. Capacity pricing starts to make sense when the monthly on demand bill is consistently higher than a small reservation would cost, or when you want a hard ceiling on spend.

What is the difference between logical and physical storage billing?

Logical billing, the default, charges for the uncompressed size of your data and includes time travel and fail safe storage. Physical billing charges for the compressed size at a higher price per GiB and bills time travel and fail safe storage separately. It is set per dataset. Physical is cheaper for data that compresses well and is rarely rewritten.

Should I partition or cluster a small table?

For small tables it makes little difference to cost, since the first 1 TiB of queries per month is free on the on demand model. Google suggests clustering over partitioning when partitions would be small, roughly under 10 GB each. Partitioning by date is still useful for incremental loads, because a run can replace recent days without touching the rest.

Do I need to partition the GA4 export myself?

No. The GA4 export already writes a separate table per day, which works like a partition as long as queries name the days they need. What you do have to handle is that recent daily tables are still being updated, so models built on them should process the last few days again on each run.