Re: prevent immature WAL streaming

At Wed, 25 Aug 2021 18:18:59 +0000, "Bossart, Nathan" <bossartn(at)amazon(dot)com> wrote in
> On 8/25/21, 5:33 AM, "alvherre(at)alvh(dot)no-ip(dot)org" <alvherre(at)alvh(dot)no-ip(dot)org> wrote:
> > On 2021-Aug-24, Bossart, Nathan wrote:
> >> Another interesting thing I see is that the boundary stored in
> >> earliestSegBoundary is not necessarily the earliest one. It's just
> >> the first one that has been registered. I did this for simplicity for
> >> the .ready file fix, but I can see it causing problems here.
> >
> > Hmm, is there really a problem here? Surely the flush point cannot go
> > past whatever has been written. If somebody is writing an earlier
> > section of WAL, then we cannot move the flush pointer to a later
> > position. So it doesn't matter if the earliest point we have registered
> > is the true earliest -- we only care for it to be the earliest that is
> > past the flush point.
>
> Let's say we have the following situation (F = flush, E = earliest
> registered boundary, and L = latest registered boundary), and let's
> assume that each segment has a cross-segment record that ends in the
> next segment.
>
> F E L
> |-----|-----|-----|-----|-----|-----|-----|-----|
> 1 2 3 4 5 6 7 8
>
> Then, we write out WAL to disk and create .ready files as needed. If
> we didn't flush beyond the latest registered boundary, the latest
> registered boundary now becomes the earliest boundary.
>
> F E
> |-----|-----|-----|-----|-----|-----|-----|-----|
> 1 2 3 4 5 6 7 8
>
> At this point, the earliest segment boundary past the flush point is
> before the "earliest" boundary we are tracking.

We know we have some cross-segment records in the regin [E L] so we
cannot add a .ready file if flush is in the region. I haven't looked
the latest patch (or I may misunderstand the discussion here) but I
think we shouldn't move E before F exceeds previous (or in the first
picture above) L. Things are done that way in my ancient proposal in
[1].

https://www.postgresql.org/message-id/attachment/117052/v4-0001-Avoid-archiving-a-WAL-segment-that-continues-to-t.patch
+ if (LogwrtResult.Write < firstSegContRecStart ||
+ lastSegContRecEnd <= LogwrtResult.Write)
+ {
<notify the last segment>

By doing so, at the time of the second picutre, the pointers are set as:

E F L
|-----|-----|-----|-----|-----|-----|-----|-----|
1 2 3 4 5 6 7 8

Then the poiter are cleard at the time F reaches L,

F
|-----|-----|-----|-----|-----|-----|-----|-----|
1 2 3 4 5 6 7 8

Isn't this work correctly? As I think I mentioned in the thread, I
don't think we don't have so many (more than several, specifically)
segments in a region [E L].

[1] https://www.postgresql.org/message-id/20201216.110120.887433782054853494.horikyota.ntt%40gmail.com

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center