Re: Changing the state of data checksums in a running cluster

On 3/13/25 17:26, Tomas Vondra wrote:
> On 3/13/25 13:32, Daniel Gustafsson wrote:
>>> On 13 Mar 2025, at 12:03, Tomas Vondra <tomas(at)vondra(dot)me> wrote:
>>>
>>> ...
>>>
>>> This also reminds me I had a question about the barrier - can't it
>>> happen a process gets to process multiple barriers at the same time? I
>>> mean, let's say it gets stuck for a while, and the cluster happens to go
>>> through disable+enable. Won't it then see both barriers? That'd be a
>>> problem, because the core processes the barriers in the order determined
>>> by the enum value, not in the order the barriers happened. Which means
>>> it might break the expected state transitions again (and end with the
>>> wrong local value). I haven't tried, though.
>>
>> Interesting, that seems like a general deficiency in the barriers, surely
>> processing them in-order would be more intuitive? That would probably require
>> some form of Lamport clock though.
>>
>
> Yeah, that seems non-trivial. What if we instead ensured there can't be
> two barriers set at the same time? Say, if we (somehow) ensured all
> processes saw the previous barrier before allowing a new one, we would
> not have this issue, right?
>
> But I don't know what would be a good way to ensure this. Is there a way
> to check if all processes saw the barrier? Any ideas?
>

Actually, scratch this. There already is a way to do this, by using
WaitForProcSignalBarrier. And the XLOG_CHECKSUMS processing already
calls this. So we should not see two barriers at the same time ...

>>>>> One issue I ran into is the postmaster does not seem to be processing
>>>>> the barriers, and thus not getting info about the data_checksum_version
>>>>> changes.
>>>>
>>>> Makes sense, that seems like a pretty reasonable constraint for the barrier.
>>>
>>> Not sure I follow. What's a reasonable constraint?
>>
>> That the postmaster deosn't process them.
>>
>
> OK, that means we need a way to "refresh" the value for new child
> processses, similar to what my patch does. But I suspect there might be
> a race condition - if the child process starts while processing the
> XLOG_CHECKUMS record, it might happen to get the new value and then also
> the barrier (if it does the "refresh" in between the XLogCtl update and
> the barrier). Doesn't this need some sort of interlock, preventing this?
>
> The child startup would need to do this:
>
> 1) acquire lock
> 2) reset barriers
> 3) refresh the LocalDataChecksumValue (from XLogCtl)
> 4) release lock
>
> while the walreceiver would do this
>
> 1) acquire lock
> 2) update XLogCtl value
> 3) emit barrier
> 4) release lock
>
> Or is there a reason why this would be unnecessary?
>

I still think this might be a problem. I wonder if we could maybe
leverage the barrier generation, to detect that we don't need to process
this barrier, because we already got the value directly ...

FWIW we'd have this problem even if postmaster was processing barriers,
because there'd always be a "gap" between the fork and ProcSignalInit()
registering the new process into the procsignal array.

regards

--
Tomas Vondra