Re: Start logical decoding from any lsn present in the wal segment

On 29 June 2017 at 16:32, sanyam jain <sanyamjain22(at)live(dot)in> wrote:
> Hi,
>
> Currently logical decoding finds a consistent point by generating a snapshot
> and stream changes after consistent point.I want to change this behaviour to
> something which stream changes from a lsn in the past provided its present
> in wal segment. Can this behaviour be implemented

No, it can't. Not without infinite catalog bloat, anyway.

To perform logical decoding we need catalogs that contain historic
data in the form of dead tuples preserved in the heaps. Say you ALTER
TABLE to add a column and drop one. There'll be a new pg_attribute row
for the new column, and the old pg_attribute row will get marked with
attisdropped by writing a new row version.

When we're doing logical decoding and examining WAL from before the
ALTER TABLE, we need to see the old state of the catalogs. So a
historic snapshot is created with an (xmin,xmax) range that means it
won't see the new pg_attribute row, and it'll see the old row-version
of the dropped attribute where attisdropped is not set. So it'll see a
table definition that matches how it was at the time the WAL it's
looking at was generated.

If you could only see the latest table definition from pg_catalog
you'd have no way to correctly decode such WAL.

That's what all the catalog_xmin business in the code is about.

There are some other complications too, around finding safe points in
the WAL stream where running xacts are precisely known; see
xl_running_xacts and the snapshot builder.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services