EXAMPLE: The API said 10,000 records. There were 41,000.

This is placeholder content written to demonstrate the article layout. Replace it with a real technical article.

Most connector bugs are not crashes. They are a run that completes successfully, reports success, and quietly returns a subset of the data. Pagination is where this happens most often.

The three ways pagination lies

It caps without telling you. You request page after page, the API keeps returning results, and then it returns an empty page at record 10,000 — not because there are 10,000 records, but because that is the maximum the endpoint will traverse. Nothing in the response says so.

It shifts under you. Offset pagination over a table that is being written to will skip and duplicate rows. If records are inserted while you are on page 3, page 4 starts in the wrong place.

It resets. A cursor that expires mid-traversal and returns an error you retry successfully — from the beginning.

How to detect it

Cheap and worth doing on every connector:

# Illustrative, not production code
seen = 0
pages = 0
while cursor:
    batch, cursor = fetch(cursor)
    seen += len(batch)
    pages += 1
    if pages > MAX_EXPECTED_PAGES:
        raise SuspiciousTraversal(pages, seen)

# Independent check: does the API agree with us?
if (reported := fetch_count()) is not None and reported != seen:
    raise CountMismatch(reported, seen)

The second half matters more than the first. Ask the API how many records it thinks there are, and compare. Where an endpoint cannot answer that, a bounded independent re-extract is the fallback — and if neither is possible, completeness cannot be proven, which is a scoping decision rather than a detail.

Why this belongs in the price

An hour of this work on every endpoint is the difference between a connector and a script that looks like one. It is also the least visible part of the job, which is exactly why it gets skipped when a connector is built in an afternoon.