This is placeholder content written to demonstrate the article layout. Replace it with a real technical article.
The single most expensive property of an API is not SOAP, and not OAuth2. It is having no way to ask for only what changed.
What you are left with
Full extract, every run. Correct, simple, and it moves the cost onto the customer’s cloud bill forever. For 50,000 records daily this is fine and probably the right answer. For 20 million it is not.
Client-side change detection. Fetch everything, hash each record, compare against the previous run’s hashes, write only the differences. This works, and it is real engineering: the hash state has to be stored somewhere, kept in step with the destination, and survive a partial run.
A proxy for change. Sometimes an id is monotonic, or there is a created_at even without an
updated_at. Then you can get new records incrementally and only need a periodic full pass to
catch updates. This is usually the best compromise and it depends entirely on the API.
The trap
An API with updated_at on some endpoints and not others. It looks incremental, the first
implementation looks correct, and the endpoints without it silently stop reflecting updates. Six
weeks later somebody notices a record that changed in the source and never changed in the
warehouse.
Score each endpoint separately. “Does this API support incremental sync?” is the wrong question; “which endpoints do?” is the right one.
Why it belongs in the estimate
This is one of the sixteen factors in our assessment, and one of the highest-weighted, because discovering it during the build changes the architecture rather than the effort. An API with no incremental capability at meaningful volume is escalated to Advanced automatically — not as a pricing tactic, but because a full-extract design with change detection is a different connector from a watermark query.