freenode
Databases & Infrastructure

PostgreSQL offline checksum flips can break or desync standbys

A regression from online checksums work, plus a long-standing gap in pg_checksums, can leave replicas verifying pages they never checksummed or silently diverge from the primary.

Offline changes to PostgreSQL data checksums can leave standbys in the wrong verification state, in one case making a replica refuse connections after replaying a primary checkpoint that turns checksum checking on without the standby’s own pages ever having been rewritten.

Bertrand Drouvot reported the problems on pgsql-hackers after hitting them while working on related checksum tooling. One failure path is a regression tied to online checksum support: a primary that enables checksums offline can emit checkpoint WAL that causes a standby to start verifying checksums it never applied, yielding fatal invalid-page errors on connect. A second path mixes online enablement with a later offline disable on a stopped standby; after restart the standby still believes checksums are on.

Daniel Gustafsson, working with Zsolt Parragi and Tomas Vondra, framed two distinct problems. Correct offline changes in a replicated cluster no longer behave as they did before online checksums, because checkpoint replay could overwrite a node’s local offline state. Separately, mismatched checksum state across primary and standby has existed in all supported versions since pg_checksums was introduced; the docs mention the risk only briefly, and tools such as pg_rewind become dangerous when nodes disagree.

Hard failure on mismatch forces a full rebuild from backup, so the near-term direction is to stop adopting checksum state from checkpoint replay, apply only dedicated checksum WAL records, and log a warning when states diverge. That improves on the silent ignore in versions 14 through 18, but does not make mixed online and offline procedures safe in every lockstep stop-and-rewrite scenario.

With PostgreSQL 19 close to release candidate, developers are wary of large control-file and WAL design changes. Discussion has turned toward documenting that offline and online transitions use different durability models and should not be mixed casually, and toward longer-term work so offline pg_checksums changes are visible to recovery instead of rewriting the data directory with no WAL trail.