Debezium's heartbeat is a hack around a missing message

A database that streams its changes to other systems, which is called change data capture, keeps its log until each consumer acknowledges a position in it: "I have everything up to here." Postgres is such a database. Its log is the write-ahead log (WAL), and a replication slot is where it remembers each consumer's last acknowledged position [1, § WAL disk space consumption]. Debezium, a common consumer, runs on Kafka Connect, Kafka's framework for moving data in and out of other systems, and Kafka Connect stores a position only on a record, so by default Debezium acknowledges only positions it has written as records [1, § Streaming changes]. Point it at a table that rarely changes, on a server that changes all the time, and it has nothing to write, so it never acknowledges, and the log grows until the disk is full.1 The standard fix is a heartbeat: a record written on a timer so there is something to acknowledge with. It works. It is also a sentinel, a record whose meaning is "there is no record", and that is why it feels wrong.

An acknowledgment is one number, a position, and it covers the whole prefix of the log behind it. The simplest way to send that number is to attach it to the records. Each record Debezium writes carries the position it came from; Postgres calls a position an LSN, and Kafka Connect stores it as the record's offset [1, § Streaming changes]. Once Kafka Connect has stored it, Debezium acknowledges that position to Postgres.2 While every stretch of log the consumer reads turns into records, the acknowledged position keeps up with the log.

It stops keeping up as soon as the consumer reads log that produces no records. Number the log's entries 1 to 100, and capture only table A. Entries 1 to 10 are changes to A; they become records, and the acknowledged position reaches 10. Entries 11 to 100 are changes to table B, or to another database on the same server, or entries that are not row changes at all. The consumer reads them and has no record to write. Postgres even tells it where the log ends: every keepalive message carries "the current end of WAL on the server" [3, § Primary keepalive message]. So the consumer knows it has everything up to 100. By default it does not acknowledge that without a record, and Postgres keeps entries 11 to 100 indefinitely. Debezium's documentation describes this case [1, § WAL disk space consumption].3

Postgres's side of the conversation already has the right shape. Its replication protocol has a progress message, one that reports a position without carrying records: the consumer's acknowledgment, a status update, can be sent "at any time", records or no records [3].4 An acknowledgment never has to pretend to be a record. Kafka Connect's interface has no such message. The poll() method that Kafka Connect calls on Debezium returns a List<SourceRecord> [5, § poll], and the position is a field on a record. There is no way to say "no new records, but I have everything up to 100", so inside Kafka Connect a position can only travel disguised as a record.

What the consumer can acknowledgeThree copies of one log with entries 1 to 100, where only entries 1 to 10 are changes to the captured table. In panel 1, records only, the consumer has read to the end of the log at 100 but has acknowledged only 10, and entries 11 to 100 are kept on disk. In panel 2, a heartbeat record is written alongside the 10 real records and the acknowledgment reaches 100. In panel 3, a status update with no record moves the acknowledgment to 100.1 · Records onlyend of log, 100Postgres's logWhat the consumer sends11–100 kept on diskacknowledged 1010 recordsno record to carry 1002 · Heartbeatend of log, 100Postgres's logWhat the consumer sendsacknowledged 10010 records+ 1 heartbeat record3 · Progress messageend of log, 100Postgres's logWhat the consumer sendsacknowledged 10010 records+ 1 status update to Postgres
One log, entries 1 to 100; only 1 to 10 are changes to the captured table. With records only, the consumer has read to 100 but can acknowledge only 10, so Postgres keeps 11 to 100. A heartbeat and a progress message both move the acknowledgment to 100; only the heartbeat needs a record to do it.

The heartbeat works within that limit. With heartbeat.interval.ms set, Debezium periodically writes a record to a dedicated topic.5 The record carries the latest position, Kafka Connect stores it, and the acknowledged position moves. When the database Debezium reads is itself idle, Debezium is not even woken up, and heartbeat.action.query has to write a real row into a real table in that database so that a change exists to capture [1, § heartbeat.action.query]. Each heartbeat is a record that means "nothing happened", the same move as a function returning null to mean "no result" because its return type has no other way to say it.

The proper fix has been on file for ten years. Randall Hauch, who started Debezium [6], opened KAFKA-3821 in 2016: "Allow Kafka Connect source tasks to produce offset without writing to topics" [7]. It is still open [7]. KIP-910, a Kafka Improvement Proposal, proposed the missing message, a SourceTask.updateOffsets() method, and rejected the alternative of a record with a null topic, key or value because of its confusing "null semantics" [8, § Rejected Alternatives]. Its authors saw the sentinel and refused it. But one of the first objections in the 2023 discussion, from Chris Egerton, asked why Kafka Connect should add the feature "when the problem it addresses is already solved" by Debezium's heartbeats [9], and the thread went quiet that September [10]. The workaround's existence became the argument against the fix.

Egerton's objection deserves an answer, and Debezium's own documentation gives it. The heartbeat is off by default: heartbeat.interval.ms is 0, and "the default behavior is that the connector does not send heartbeat messages" [1, § heartbeat.interval.ms]. When the database is idle, the fix is a change to that database: the consumer has to run a query that writes something, either a row in a heartbeat table added to the tables Postgres streams or, since Postgres 14, a message written straight into the log [1, § WAL disk space consumption; 4]. So the workaround costs two things the problem never called for. A consumer whose job is to read the database has to write to it. And because the fix is off by default and depends on each database's version, privileges and settings, every team that runs Debezium has to discover the problem and set up the fix itself.6 A progress message would need none of it.

Debezium has since added a way around the heartbeat, and it shows what the missing message costs. In lsn.flush.mode=connector_and_driver, a Technology Preview, the Postgres driver acknowledges the end-of-log position that keepalives report, "to prevent WAL growth on low-activity databases" [1, § lsn.flush.mode]. That acknowledgment goes around Kafka Connect, so the slot can move past the position Kafka Connect stored, and after a restart a setting decides which of the two to believe; trusting the slot "skips replaying events between the stored offset and the slot position" [1, § offset.mismatch.strategy]. One fact, how far the consumer has read, now lives in two places that nothing keeps in step: a dual write, the problem change data capture exists to remove [12, pp. 501–503]. And the mode still turns the heartbeat on, every ten minutes, so that Kafka Connect's copy does not fall too far behind [1, § lsn.flush.mode].

That answers why the heartbeat feels wrong. Postgres's protocol can carry "I have everything up to 100" in one status update. To pass the same acknowledgment through Kafka Connect, Debezium has to invent a record, a topic to hold it, and sometimes a row in a table nobody reads. The heartbeat is not wrong because it is a workaround; it is wrong because an acknowledgment was never a record, and the interface gave it nowhere else to go except around the interface itself.

The pattern is not specific to change data capture. When an interface gives its users no way to say something they need to say, they say it with something it does allow, a sentinel in an ordinary message, and everyone who reads that message inherits the exception. Progress messages are easy to leave out, because a design pictures a busy stream, and only a quiet one needs them. The systems that met quiet streams added them: TCP sends a bare acknowledgment when it has no data to carry one, and stream processors send watermarks, progress messages that travel forward with the records instead of back.7

Wherever an acknowledgment is what lets something be freed, it has to be able to travel on its own. While records flow, piggybacking hides the gap; only a quiet moment exposes it.

Footnotes

  1. max_slot_wal_keep_size caps how much log Postgres keeps for one consumer; past the cap the log is removed, and that consumer can no longer continue [2, § max_slot_wal_keep_size]. It is not a fix for a quiet consumer; it protects the database from a dead one, which no acknowledgment can rescue. ↩

  2. Debezium's documentation puts it from the other side: without stored offsets, Debezium "does not have an opportunity to send the latest retrieved LSN to the database" [1, § heartbeat.interval.ms]. ↩

  3. Gunnar Morling traced it on an idle Amazon RDS database whose log grew until it filled 200 GiB [4]. ↩

  4. Every message in the protocol starts with a type byte: w for log data and k for a keepalive from Postgres, r for the consumer's status update [3]. ↩

  5. __debezium-heartbeat.<prefix> by default [1, § topic.heartbeat.prefix]. ↩

  6. John Ousterhout's rule is to pull complexity downwards: a module should handle its own hard cases rather than pass them to everyone who uses it, and a configuration parameter is the usual way of passing them up, since each user then has to work out the right value [11, § 8.2]. Here the cost arrives as what he calls an unknown unknown, the worst of his three symptoms of complexity: nothing tells a team the setting exists until the disk is filling [11, § 2.2]. ↩

  7. Having the message is not enough if it is generated only from records. Flink derives watermarks from records, so it documents idle sources that hold back every watermark [13, § Dealing With Idle Sources], and on the KIP-910 thread John Roesler noted that Kafka Streams' stream time stalls the same way when an input goes quiet [14]. ↩

References

  1. Debezium connector for PostgreSQL [HTML]
    Debezium Authors. Debezium Documentation 3.7.

  2. Replication [HTML]
    PostgreSQL Global Development Group. PostgreSQL 18 Documentation.

  3. Streaming Replication Protocol [HTML]
    PostgreSQL Global Development Group. PostgreSQL 18 Documentation.

  4. The Insatiable Postgres Replication Slot [link] [archived]
    Morling, G., 2022. Gunnar Morling.

  5. SourceTask [HTML]
    Apache Software Foundation. Apache Kafka 3.9 API.

  6. Debezium Evolving [link] [archived]
    Hauch, R., 2017. Debezium Blog.

  7. Allow Kafka Connect source tasks to produce offset without writing to topics [link] [archived]
    Hauch, R., 2016. Apache Kafka Jira.

  8. KIP-910: Update Source offsets for Source Connectors without producing records [link] [archived]
    Rao, S., 2023. Apache Kafka Wiki.

  9. Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records [HTML] [archived]
    Egerton, C., 2023. dev@kafka.apache.org.

  10. Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records [HTML] [archived]
    Rao, S., 2023. dev@kafka.apache.org.

  11. A Philosophy of Software Design
    Ousterhout, J., 2021. Yaknyam Press.

  12. Designing Data-Intensive Applications
    Kleppmann, M. and Riccomini, C., 2025. O'Reilly Media.

  13. Generating Watermarks [link]
    Apache Software Foundation. Apache Flink 2.3 Documentation.

  14. Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records [HTML] [archived]
    Roesler, J., 2023. dev@kafka.apache.org.