pgsql: Fix data loss in logical replication.

From: Amit Kapila <akapila(at)postgresql(dot)org>
To: pgsql-committers(at)lists(dot)postgresql(dot)org
Subject: pgsql: Fix data loss in logical replication.
Date: 2025-04-24 05:10:37
Message-ID: E1u7or7-001Vsz-0R@gemulon.postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-committers

Fix data loss in logical replication.

This commit is a backpatch of commit 4909b38af0 for 13.

Data loss can happen when the DDLs like ALTER PUBLICATION ... ADD TABLE ...
or ALTER TYPE ... that don't take a strong lock on table happens
concurrently to DMLs on the tables involved in the DDL. This happens
because logical decoding doesn't distribute invalidations to concurrent
transactions and those transactions use stale cache data to decode the
changes. The problem becomes bigger because we keep using the stale cache
even after those in-progress transactions are finished and skip the
changes required to be sent to the client.

This commit fixes the issue by distributing invalidation messages from
catalog-modifying transactions to all concurrent in-progress transactions.
This allows the necessary rebuild of the catalog cache when decoding new
changes after concurrent DDL.

The fix for 13 is different from what we did in branches 14 and above,
such that for 13, the concurrent DDL changes (from DDL types mentioned
earlier) will be visible for any newly started transactions. To make them
visible in concurrent transactions, we need to introduce a new change type
REORDER_BUFFER_CHANGE_INVALIDATION, already present in branches 14 and
greater. We decided not to take the risk of a bigger change and fix the
issue partially in 13.

Reported-by: hubert depesz lubaczewski <depesz(at)depesz(dot)com>
Reported-by: Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com>
Author: Shlok Kyal <shlok(dot)kyal(dot)oss(at)gmail(dot)com>
Author: Hayato Kuroda <kuroda(dot)hayato(at)fujitsu(dot)com>
Reviewed-by: Zhijie Hou <houzj(dot)fnst(at)fujitsu(dot)com>
Reviewed-by: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
Reviewed-by: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Tested-by: Benoit Lobréau <benoit(dot)lobreau(at)dalibo(dot)com>
Discussion: https://postgr.es/m/de52b282-1166-1180-45a2-8d8917ca74c6@enterprisedb.com
Discussion: https://postgr.es/m/CAD21AoAenVqiMjpN-PvGHL1N9DWnHSq673bfgr6phmBUzx=kLQ@mail.gmail.com
Discussion: https://postgr.es/m/CAD21AoAhU3kp8shYqP=ExiFDZ9sZxpFb5WzLa0p+vEe5j+7CWQ@mail.gmail.com

Branch
------
REL_13_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/247ee94150b6fe8906da51afadbedf8acf3c17cf

Modified Files
--------------
contrib/test_decoding/Makefile | 2 +-
.../expected/invalidation_distrubution.out | 20 +++++++
.../specs/invalidation_distrubution.spec | 32 +++++++++++
src/backend/replication/logical/reorderbuffer.c | 64 +++++++++++++++++++---
src/backend/replication/logical/snapbuild.c | 63 +++++++++++++++++----
src/include/replication/reorderbuffer.h | 4 ++
6 files changed, 164 insertions(+), 21 deletions(-)

Browse pgsql-committers by date

  From Date Subject
Next Message Christoph Berg 2025-04-24 10:21:06 Re: extension_control_path and "directory"
Previous Message Michael Paquier 2025-04-24 03:23:47 pgsql: psql: Fix assertion failures with pipeline mode