Get an estimate

Data models

Data models: the unit you actually buy

Interfaces, infrastructure and the work of keeping them reliable are all necessary — and none of them is what you wanted. Tell us which datasets you need; what lands in your warehouse is data your team can put to work.

4 API resources from the HR API become one Employees data model in BigQuery, with the raw resources landed alongside it.HR APIYour warehouseBigQuery/employees/employee-fields/departments/locationsConnectorauth · sync · stateRAW — always includedemployeesemployee_fieldsdepartmentslocationsEmployees data modelemployeesone fixed price
The Employees data model needs 4 API resources (/employees, /employee-fields, /departments, /locations). You ask for Employees; we work out the resources. Both the Employees dataset and the raw resources behind it land in your BigQuery. Table naming varies by warehouse — the principle does not.

Types of data models

Six shapes of data, and what each one makes you decide

Almost every dataset a business asks for is one of six shapes. Which one it is decides more about the work than the industry, the vendor or the API ever will — because each shape forces a decision the source does not make for you.

Master data

The things your business is about.

One row per real-world thing, with a version history. The same customer record at three valid-from dates, the most recent marked current.one row per thing, versioned by valid-fromcustomerC-10422024-012025-032026-02currentcurrent state, or every version?

Customers, employees, products, suppliers, sites. One row per real-world thing, a small number of rows, and a long life. The hard part is never the volume — it is that the thing changes and the source may or may not remember what it used to be. A customer moves country; a product is renamed; an employee changes department.

What it forces you to decideWhether you need today's picture, the full history, or both. History means versioned rows and a valid-from date, which is a different dataset from the current-state one — and often worth having as both.

Transaction data

Things that happened, with a number attached.

One row per thing that happened, appended and never edited, each referring back to the master record it belongs to.one row per thing that happened2026-02-031,200.002026-02-07340.502026-02-119,910.00appended, never editedcustomerC-1042

Orders, invoices, payments, absences, timesheet entries. One row per occurrence, appended and never edited, each pointing back at the master records it belongs to. Volumes are larger by orders of magnitude and they only grow.

What it forces you to decideHow far back the first load goes, and whether the source can be asked what changed since. Backfill and incremental support are two of the heaviest factors in the assessment, and a transactional dataset is where both bite.

Time series data

One value, measured repeatedly.

Three observations. Asked for the value on the 10th, the answer is the value last observed on the 7th, held until the next observation on the 18th.103718heldvalueday

A price, a valuation, an FX rate, a meter reading, an index. The source reports it when it moves, not on the grid your report wants to read. Everything difficult about this shape is in the gaps between observations.

What it forces you to decideWhat the value was between two observations. Held, interpolated or extrapolated are three different answers from the same data — and interpolation is a derived field, not a mapping.

Event data

Something happened, and there is a lot of it.

A high-volume append-only stream arriving in bursts, including one event delivered twice, removed by de-duplicating on the event id.append-only, and it arrives in burstssame event twiceat-least-once — de-duplicate on event id

Clicks, logins, sensor readings, webhook deliveries, status changes. Append-only like transactions, but higher volume, arriving in bursts, and usually with no useful way to ask the source for a second opinion about what it sent you.

What it forces you to decideWhat counts as the same event, and how late one may arrive and still be counted. Delivery is at-least-once, so the model de-duplicates on an event id — which means the source has to provide one.

Reference data

The small lists everything else joins to.

A small, rarely changing list of codes and their meanings, joined to by many other datasets.small, rarely changes, joined to by everythingCHFSwiss francEUREuroUSDUS dollarGBPSterlingordersinvoicespaymentsyour codes, or the published standard?

Currencies, countries, units, tax codes, status codes, industry classifications. A handful of rows that change once a year and are read by every other dataset you own. Often the cheapest data model to build and the most annoying to get wrong, because an error propagates everywhere at once.

What it forces you to decideWhose list wins. Your own codes and the published standard rarely agree exactly, and deciding that in the model is much cheaper than discovering it in a report.

Document data

One record, nested several levels deep.

One nested document per record, flattened into a parent table and a child table — a choice about grain, not a mechanical conversion.one nested document per recordordercustomerlines[]sku, qtyflattenordersorder_lineswhich arrays become tables of their own?

A contract, an application form, a configuration payload, an API response that carries its own children — arrays inside objects inside arrays. The source hands over one document per record and considers the job done.

What it forces you to decideWhich nested arrays become tables of their own and which stay inside the row. That is a grain decision, and a different grain is a different data model, so it changes what you are buying rather than only how it is built.

A shape is not a price. Two datasets of the same shape can classify differently — that depends on how many resources they need, how well the source answers “what changed”, and how much has to be derived. The ladder below is where that is settled.

Three rules

What is and is not included

Three consequences of pricing by dataset rather than by endpoint — and the three places connector pricing usually misleads people.

One model may use many endpoints

One data model may use many API endpoints. You don't pay per endpoint. We work out which resources a dataset needs and how they join. A list of resources appears in your estimate as evidence for the classification, never as line items.

Raw source data is always included

You get the model and the raw source data behind it. Every resource read to build a model is also landed in raw form, at no extra cost, so you are never dependent on our interpretation of your data. Physical table naming varies by warehouse, so we promise the principle, not specific table names.

A difficult source does not make a model cost more

Awkward authentication, bad pagination and hostile rate limits move the connector into the complex band, once. They never raise the type of a data model on it. Otherwise you would pay for the same authentication problem repeatedly.

Interpretation

A price is not a series until someone decides what happened in between

Time series deserve their own section, because the gaps between observations are where the modelling decisions actually are — and where a dataset quietly stops being a copy of the source.

Say the source gives you three observations — a value on the 3rd, the 7th and the 18th. Your report asks what it was on the 10th. The source never said. There are three honest answers, each correct somewhere and wrong somewhere else, and the same three observations produce three different curves.

Three observations. Asked for the value on the 10th, the answer is the value last observed on the 7th, held until the next observation on the 18th.103718heldvalueday

Last known value

It stays as last observed until the next observation. Right for prices, contract rates and fixings — things that hold until someone changes them. Every number in the column traces back to something the source actually said.

Three observations. Asked for the value on the 10th, the answer is a value part-way between the 7th and the 18th, assuming it moved evenly between them.103718assumedvalueday

Interpolated

Assume it moved evenly between the two observations. Right for a continuous physical quantity read intermittently — a meter, a temperature, a tank level. Wrong for a price: the price did not drift on the 10th, it simply was not observed.

Three observations. Asked for the value on the 24th, after the last observation, the answer continues the last observed direction. Nothing in the source supports it.243718modelledvalueday

Extrapolated

Past the last observation, on the direction it was last heading. Nothing in the source supports it. It is a forecast wearing the same column name as a fact, which is the most expensive mistake on this page.

We default to last known value

It is the only one of the three that never invents a number. If a different rule is right for your data, we will use it — and write down which one, and why, next to the dataset.

A smooth daily curve is business logic

Interpolating between observations is a derived field, not a mapping, and it puts the data model at Advanced or above. Worth knowing before you ask for “just a daily series”, because that phrase is doing a lot of work.

We do not extrapolate by default

If you ask for it, it lands in a separately named column, marked as modelled, never mixed into the observed one. You can always tell which numbers came from the source.

Two dates, not one

The day the value applied to and the day the source told you are different, and sources revise the past — a price for the 7th restated on the 20th. If your reports must not change retrospectively, you need both dates, and that has to be decided before the first load, not after.

The ladder

Four types, and what puts a dataset in each

Classification is a documented judgement made from your API's documentation, with the reason written down. If we get it wrong in your favour, that is our problem.

TypeTypicallyDelivery
SimpleThree come with a Base ConnectorOne resource, flat records, little or no transformation. Reference and lookup data, or a single entity with no dependencies.2–5 business days
StandardA straightforward business entity. Normal reference data, simple relationships, standard incremental sync, limited transformation.5–10 business days
AdvancedMultiple dependent resources, complex relationships or nested structures, harder incremental sync, significant transformation.10–20 business days
CustomSubstantial bespoke business logic, unusual reconciliation, more than one API, or exceptional technical requirements. Individually assessed.20–40 business days

Afterwards

When something is a change and when it is a new model

Worth knowing before you ask, because the answer decides the price.

A new field on a dataset you already havePart of keeping it working. No charge under a support plan
A new API resource inside an existing data modelA fixed fee, smaller than a data model. See the price list
A new datasetA new data model, priced by its type
A second source systemA new Base Connector. The DevOps line is zero — your environment already exists

Tell us the datasets, not the endpoints

Name the questions you want the warehouse to answer. We will come back with the models, the types, the reasons and a price.

Get an estimate