Change data capture (CDC) is how Hightouch detects which rows changed between sync runs, so a sync sends only the difference instead of the whole dataset.
| Audience | Data teams configuring or troubleshooting syncs |
| Prerequisites | A model with a unique primary key. |
Without CDC, every run would resend your entire dataset, which means more API calls, slower syncs, and wasted downstream processing. With CDC, a sync sends only what changed since the last run.
How CDC works
Each time a sync runs, Hightouch compares the current query results against a stored snapshot of the previous run, called the diff file, and sends only the rows that differ:
- Query the model against your source.
- Compare the results to the previous run's diff file, matching rows by primary key.
- Classify each row as added, changed, or removed.
- Send those changes to the destination, along with any rows that failed on the previous run.
- Save a new diff file for the next comparison.
CDC compares only mapped model columns. The exception is custom destinations, such as HTTP Request, which consider every column.
CDC depends on a unique primary key
The primary key is how CDC identifies a row from one run to the next. If primary keys are missing or duplicated, Hightouch can't reliably tell an added row from a changed one. See primary key requirements.
Changing a model's primary key breaks CDC's record identity. If you select a different primary key column, reset CDC for every sync that depends on the model. If you keep the same column but change how its values are calculated, records may be added or removed in your destination, depending on the sync configuration.
CDC by sync mode
The sync mode determines which detected changes become destination operations:
- Insert — sends only rows whose primary key is new since the previous run.
- Update and Upsert — send rows whose mapped values changed.
- Archive and All (sometimes called Mirror) — don't use CDC. Every row is sent on every run, and these modes don't store requests in the live debugger.
For the full mode reference, see sync modes and change data capture.
Reset or reprocess CDC
Some changes require Hightouch to rebuild CDC state before the next run behaves as you expect. Reset CDC, full resync, and clear-and-fill are run behaviors, not sync modes.
| When this happens | What to do | Effect |
|---|---|---|
| You change a model's primary key | Reset CDC | The next run captures a fresh baseline and sends nothing. |
| You change field mappings | Nothing; it's automatic | Hightouch reprocesses the model results on the next run. |
| A sync hasn't run in over 30 days | Reset CDC or full resync | The diff file has expired, so CDC has no baseline to compare against. |
| You need to rebuild destination state | Full resync | Processes every current row as if it were newly added. |
For the consequences and click paths of each action, see the Syncs overview.
When CDC runs
CDC happens after Hightouch receives the query results and before it writes to the destination. While the sync status shows Querying, Hightouch is waiting for query results, computing the diff, or saving the new diff file.
Where CDC runs
By default, Hightouch computes CDC on its own infrastructure, called the basic engine. For supported sources, the Lightning Sync Engine computes CDC directly in your warehouse, which speeds up syncs at higher volumes.
Where diff files are stored
Hightouch stores the previous run's query results to compute the next diff. By default, diff files live in an encrypted, Hightouch-managed bucket and are retained for 30 days. Some plans let you bring your own bucket to keep diffs in your own infrastructure.
If a sync doesn't run for more than 30 days, its diff file expires. The next run needs a reset CDC or a full resync to establish a new baseline.
Why reverse ETL uses difference-based CDC
Hightouch compares full before-and-after query results, a method called difference-based CDC. Warehouses can't produce CDC logs for arbitrary SQL queries or dbt models, so log-based CDC isn't possible when reading from them.
This is the reverse of traditional ETL. OLTP databases such as Postgres and MySQL log incremental changes natively, and ETL tools use those logs when loading data into a warehouse. Because Hightouch sends data out of the warehouse, it diffs query results instead.
FAQ
Does Hightouch keep historical diff files?
No. Hightouch stores only the most recent diff file and compares against it.
What happens if a sync fails?
CDC retries the failed rows on the next run. See retries.
What happens if I stop syncing for more than 30 days?
The diff file in managed storage expires. Reset CDC or run a full resync to rebuild the baseline.
What happens if I change my mappings?
Changing mappings can trigger a full reprocess on the next run. See field mapping updates.