How to Fix Airbyte CDC Syncs That Stop or Duplicate Rows

October 28, 2026

CDC syncs are quiet when they break. No error email, no failed job — the destination table just stops gaining rows, and you find out three weeks later when a report is wrong.

The three failures below account for nearly every broken Postgres CDC pipeline I’ve had to fix: the WAL was truncated before the slot read it, the slot leaked and filled the disk, or the destination got duplicates because of how it handles updates. Each has a different detection signal.

Why did my CDC sync silently stop?

The most common cause is wal_level. Change data capture for Postgres requires wal_level = logical. The default on many managed instances is replica, which supports replication slots but not the logical decoding Airbyte uses.

Check:

SHOW wal_level;
SHOW max_replication_slots;
SHOW max_wal_senders;

You want logical, and max_replication_slots above the number of active slots. Changing wal_level requires a database restart — on RDS and Aurora that’s a reboot, not a maintenance window resize.

The second silent-stop cause is that the slot already exists from a previous connection attempt. Airbyte creates a slot named airbyte_slot_ followed by the connection id; if a deleted connection left one behind, the new connection’s initial snapshot may fail to create it and the connector retries indefinitely.

SELECT slot_name, active, restart_lsn FROM pg_replication_slots;

Deactivate (or delete) stale slots before recreating the connection. A slot with active = false and a very old restart_lsn is holding WAL and doing nothing.

Why does my database disk keep filling up?

A replication slot tells Postgres “don’t discard WAL past this point”. If nothing consumes the slot, WAL accumulates forever.

An Airbyte connection that’s paused, deleted without cleanup, or failing to read leaves the slot open. On a write-heavy database, a leaked slot can add 50-200 GB in a week, and the first symptom is the primary running out of disk.

Detect it before it’s an outage:

SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag
FROM pg_replication_slots
ORDER BY lag DESC;

Anything above 10 GB of lag deserves an alert. Above 100 GB on a small instance, you’re hours from a disk-full event.

Airbyte won’t clean up abandoned slots for you. Audit quarterly:

  1. List slots.
  2. Match against active Airbyte connections.
  3. Drop any slot with active = false and no matching connection.

Dropping a slot that’s still needed means the next sync does a full resync. That’s recoverable. A full disk on the primary is not.

Why does the destination have duplicate rows?

CDC emits an insert event for every insert, but when a row is updated you get a second event with the same primary key. Append them into MergeTree and the destination now holds two versions of the same row.

The fix is destination-side deduplication:

CREATE TABLE orders (
  id UInt64,
  status String,
  updated_at DateTime64(3),
  _airbyte_extracted_at DateTime64(3),
  _airbyte_meta String
) ENGINE = ReplacingMergeTree(_airbyte_extracted_at)
ORDER BY id -- primary key

ReplacingMergeTree keeps the row with the highest version column — use Airbyte’s _airbyte_extracted_at, not the source’s updated_at, because clocks lie and batch order matters.

The catch: deduplication happens at merge time, not at insert time. Until the merge settles you’ll see duplicates in query results. Add FINAL to the query:

SELECT count() FROM orders FINAL;

FINAL is slow on large tables — it forces a merge-on-read. The alternative is an AggregatingMergeTree with argMax, or simply grouping in the query. For dashboards reading a few million rows, FINAL is acceptable; on 500 million it isn’t.

If your source does hard deletes, CDC needs the slot’s retention to see them and the destination needs a deletion strategy — ReplacingMergeTree has no opinion about deleted rows. The usual approach is a soft-delete column with a downstream filter, plus a periodic full-refresh reconciliation.

What should I monitor on an Airbyte CDC pipeline?

Four checks, cheapest to most useful:

Check Where Alert when
Slot lag bytes Postgres query above > 10 GB
Sync duration trend Airbyte job history 2x the 7-day median
Destination max(cursor) age Destination SQL Older than 2x sync frequency
Row count delta Source vs destination count() Off by > 0.5%

The last one catches failures the first three miss — a sync that “succeeds” with a schema mismatch will still emit jobs on time.

Airbyte has built-in connection notifications (webhook or email on failure) since 0.50. Turn them on, but read the trigger carefully: they fire on job failure, not on “job succeeded with 0 records emitted”. A connector that filters everything out because a schema change renamed a column reports success.

Is CDC worth the complexity?

For a table that gets updated, yes. Xmin-based incremental sync misses rows whose updated_at isn’t maintained, and full refresh on a large table is wasteful.

For an append-only event table that never updates rows, CDC is unnecessary work — plain incremental append with a monotonic cursor is simpler and has none of the slot risk.

The honest trade: CDC gives you deletes and sub-minute change visibility, and charges you a replication slot you must monitor forever. If you can’t monitor that slot, use the simpler mode.

Quick Summary

  • CDC requires wal_level = logical and a database restart to change it.
  • Leaked replication slots accumulate WAL and can fill a primary’s disk; audit slots quarterly.
  • Deduplicate in the destination with ReplacingMergeTree(_airbyte_extracted_at) and query with FINAL.
  • Alert on slot lag above 10 GB and on destination cursor age exceeding twice the sync frequency.
  • Built-in notifications fire on failure, not on “succeeded with zero records” — check row counts separately.
  • Append-only tables don’t need CDC; skip it and avoid the slot risk.</}
Tags: