Get an estimate

Connector as a Service

We build the connectors that don't exist

A connector is not an API call. It is everything that has to be true for data to arrive in your warehouse reliably, again tomorrow, without anyone watching. How much that takes depends entirely on the source.

Complexity

Not all connectors are equal

The same words — “we'll build a connector for that API” — can mean an afternoon or a month. The difference is not how hard we are willing to work. It is what the source forces on us. Move along the scale to see what changes.

A source is fetched once, and what comes back is stored.SourceGETConnectorWarehouse

Fetch

Some connectors really are simple.

A published file, a page of data, one request that returns everything. The connector runs on a schedule, takes what is there, and writes it down. There is no state to keep, because the next run does not need to know anything about the last one.

  • Runs and finishes
  • Nothing to remember
  • Retry by running it again
An API is read page by page; the connector saves where it got to before writing to the warehouse.REST APIGETpage 1page 2page 3ConnectorCheckpointWarehouse

Stateful

This is what most connectors are.

A normal production REST connector authenticates, walks through pages, stays inside the rate limit, handles the responses that are not what the documentation promised, and keeps track of where it got to. That last part is what separates it from a script.

  • Authentication and pagination
  • Incremental sync and backfill
  • Remembers where it got to
A planner splits the work into independent units, three workers run them at the same time, and all of them write to the warehouse.SourcePlannerdivides the workWorker 1Worker 2Worker 3Warehouse

Parallel

Some sources cannot be read one page at a time.

Thousands of accounts, one request per record, or ten years of history to load before the first useful report. A planner works out what needs doing and splits it into pieces that can run independently; several workers run them at once, coordinating so the source is never asked for more than it allows.

  • Work divided into independent units
  • Shared rate-limit budget
  • Resumes mid-run
A connection is held open to the source, events arrive continuously, and a buffer absorbs them before the warehouse.Sourceheld openConnectoreventsBufferabsorbs the burstsWarehouse

Continuous

Not everything is a scheduled request.

Some sources push to you and never stop: a socket, a streaming endpoint, an event feed. The connector holds the connection open, notices when it drops, reconnects without losing its place, discards messages it has already seen, and buffers bursts so a busy hour upstream does not become a failure downstream.

  • Connection lifecycle and reconnects
  • Duplicate handling
  • Absorbs bursts

Most connectors sit in the middle two. A few really are a single scheduled request, and we will say so rather than dress one up. A few need work planning, several workers and a coordinated rate-limit budget, because reading the source any other way would take days.

Reliability

The devil is in the detail

Building a data loader is one thing. Keeping it running is another, and the second job never appears in the estimate for the first. A script that calls an API works until the day the source is busy, or does not recognise where the call came from, or rotates a token overnight. Here are eight of those days, and what a connector does about each one.

The source pushes back

Ask for too much and a source will say so.

Four attempts against a rate-limited source: three are refused, and the connector waits longer after each one before the fourth succeeds.The source is refusing to be rushedSource[429][429][429][200]Connectortime

A 429, a quota header, or answers that simply get slower. The connector reads that and waits — longer each time, shorter as things recover — rather than retrying straight into the same wall. Ignoring back-pressure is how an integration gets throttled to a crawl, or has its key pulled by somebody who assumed it was a runaway script.

The source does not know you

Some APIs only accept traffic from addresses they already trust.

One API and one connector. A call from an unpredictable address is refused; calls from the single allowlisted address are accepted, and webhooks are delivered only to that same address.APIany address[403]API calls[200]webhooks[200]Connector203.0.113.10

That does not work if the connector goes out from whatever address it is handed that morning. We give it a fixed outbound address you can pass to the source's owner or to your own security team — and where a source insists on being reached from a particular place, the traffic is routed to come from there.

Updates during pagination

This is the quietest bug in the whole business.

The connector reads page one and page two. The system then updates a record that was on page three, which moves it to the top of page one — behind the read cursor. Page three no longer contains it, so the row is never read, and every request returned 200.Updates during paginationSystemAPIConnectorp1moved row lands here, already readp2record updatednewest first, so it moves to p1p3gone from here. nothing errored

Rows change position between one page and the next, and you skip records that were never deleted — or read others twice. Nobody gets an error; the row is simply not there. Counting pages cannot fix it, so the connector orders by something stable and carries a cursor, and what it has already read cannot move underneath it.

Nothing reliably says what changed

Incremental sync needs the source to answer “what changed since?”

An employee record's own timestamp stays at 09:00 while its address changes at 11:00, so a query for everything changed since 10:00 does not return it.Employee 42updated_at 09:00└ Addresschanged 11:00ask: changed since 10:00Employee 42 is not returned

Many cannot. A record's own timestamp may not move when a child record does, and deletes often leave no trace at all. We work out what the source can honestly be asked — and where the answer is “nothing dependable”, we say so and schedule a periodic full re-read rather than ship a cursor that quietly drifts.

The API changes shape

A field turns from a string into an object.

A field the connector was built against as text starts coming back as an object, so the connector stops rather than writing the wrong shape.Built against"manager": "A. Smith"Now returns"manager": { "id": 42 }Stop. Do not write this into the warehouse.

An endpoint is versioned out; a vendor sunsets something with six weeks' notice. The connector notices when responses stop matching what it was built against and stops, rather than writing the wrong thing into your warehouse. Who fixes it depends on what changed, and we say which in writing.

The credentials move

A connector that authenticates only at start-up works until the first rotation.

A token is valid, then expires and is refused, then is refreshed and is valid again.tokenvalidexpiresrefusedrefreshnew tokenvalidRead at run time, never baked into the image

Tokens expire, refresh tokens rotate, secrets get changed by somebody doing exactly the right thing. Credentials are read at run time from wherever you keep them, refreshed when the source says they are stale, and never baked into the image — so rotating one is a change you make, not a redeployment you ask us for.

There is nowhere safe to test

Some sources have a sandbox that behaves like production. Many do not.

A sandbox behaves differently from production, so the connector is proven against recorded real responses and a narrow live slice.Sandboxnot the sameProductionthe only truthSo the connector is proven againstrecorded responsesa live slice

That decides how the connector is built and how it is safely changed later: what can be exercised against recorded real responses, what has to run against a live account, and what is proven on a narrow slice first. It ships with those tests, and with its environments kept apart, so changing it later is not an act of faith.

When a request fails anyway

Not every failure deserves the same response.

Treating them alike is what turns one bad record into a stalled pipeline — or a revoked credential into a night of silent retries.

ResponseWhenWhat happens
RetryThe API says you are going too fastWaits for as long as it was asked to, then carries on where it left off.
RetryA request times out, or the API returns an error it should notTries again, backing off each time, a bounded number of times.
ContinueOne record is malformed and cannot be readSets that record aside where you can see it, and keeps going with the rest.
StopCredentials are rejected, or the source changed shapeStops and raises it. Retrying cannot fix either, and continuing would land wrong data.
StopThe source has been unreachable for too longStops and raises it, with the last good checkpoint intact.

None of it is silent. A connector that stops tells you it stopped, and one that set a record aside tells you which one — because a pipeline that fails quietly is worse than one that fails loudly. What a connector reports is the same whether or not you buy a support plan.

It does not start again from zero

As it works, the connector records where it got to. Depending on the source that might be a timestamp, a page token, an event id or a list of finished work items — which one is our problem, not yours.

What it means for you is simple: an interrupted load of ten million records resumes near where it stopped. Nobody re-runs it over a weekend, and the source is not asked for everything a second time.

A long run fails part-way through. The next run resumes from the last checkpoint instead of starting again from the beginning.Records to loadfailedcheckpointresumes here
The connector records where it got to as it goes. When a run stops — a failure, a deployment, a source that went down — the next one continues from that point rather than reloading everything.

Freshness

Batch or continuous is a requirement, not a preference

Both are normal. The right answer is the simplest one that meets the business need, and that is a question about your reporting, not about our engineering.

Batch

The connector wakes up, does the work and stops — nightly, hourly, every fifteen minutes. Nothing runs in between, which makes it cheaper to run, simpler to reason about and easier to re-run when something upstream was wrong.

This is the right answer far more often than people expect. A dashboard read each morning does not need a connection held open all night.

Continuous

Events are handled as they arrive, because waiting for the next scheduled run is not acceptable — a fraud check, an operational alert, a queue somebody is working from.

It costs more to run and more to operate: something is always connected, and everything that can drop has to be able to recover. Worth it when the requirement is real.

Whose problem is it

You describe the data requirement. We engineer the connector that fits it.

You tell us

  • The source, and how to get into it
  • The datasets you want, in business terms
  • Roughly how much data there is
  • How fresh it needs to be
  • What breaks if it stops

We decide

  • Where it runs and how it is scheduled
  • Whether the work is divided, and how
  • How progress is recorded and recovered
  • How failures are retried, set aside or raised
  • Whether it is batch or continuous

Those decisions are written down and handed over with the connector — the source, the infrastructure definitions and the documentation, in your accounts, in your cloud. You are not buying access to something we run. See what you get.

Sources and destinations

What we build against

Any of these sources, into any of these warehouses. The pairs below are the shapes we have written up; they are examples, not a menu.

Sources

  • REST
  • GraphQL
  • SOAP
  • Private / internal
  • Vendor / partner
  • Custom / legacy

Destinations

  • BigQuery
  • Snowflake
  • Databricks

REST API → BigQuery

A documented REST API landing in BigQuery on a schedule, with incremental sync and a tested backfill.

3 data models included. Then: Employees, Customers, Orders +1

REST API → Snowflake

A REST API landing in Snowflake through a staged load, with incremental sync and a tested backfill.

3 data models included. Then: Employees, Absences, Products

REST API → Databricks

A REST API landing in Databricks via object storage and Auto Loader, as Delta tables you own.

3 data models included. Then: Employees, Events, Customers

SOAP API → your warehouse

A SOAP or legacy XML interface, still the system of record, landing in a modern warehouse.

3 data models included. Then: Shipments, Invoices, Reference data

Private or internal API → your warehouse

Your own service, or a partner's, behind a VPN or an allowlist — where a SaaS connector cannot reach.

3 data models included. Then: Pricing, Entitlements, Usage

Tell us about one API

Send the documentation link. You get a complexity tier, a recommended architecture, a fixed price and a delivery window — free, usually the same day. If a vendor connector would serve you better, we will say that instead.

Get an estimate