Re: FSM Corruption (was: Could not read block at end of the relation)

On Tue, Feb 27, 2024 at 11:34:14AM +0100, Ronan Dunklau wrote:
> - happens during heavy system load
> - lots of concurrent writes happening on a table
> - often (but haven't been able to confirm it is necessary), a vacuum is running
> on the table at the same time the error is triggered
>
> Then, several backends get the same error at once "ERROR: could not read
> block XXXX in file "base/XXXX/XXXX": read only 0 of 8192 bytes", with different

What are some of the specific block numbers reported?

> has anybody witnessed something similar ?

https://postgr.es/m/flat/CA%2BhUKGK%2B5DOmLaBp3Z7C4S-Yv6yoROvr1UncjH2S1ZbPT8D%2BZg%40mail.gmail.com
reminded me of this. Did you upgrade your OS recently?

On Fri, Mar 01, 2024 at 09:56:51AM +0100, Ronan Dunklau wrote:
> I think I may have missed something on my first look. On other affected
> clusters, the FSM is definitely corrupted. So it looks like we have an FSM
> corruption bug on our hands.

What corruption signs did you observe in the FSM? Since FSM is intentionally
not WAL-logged, corruption is normal, but corruption causing errors is not
normal. That said, if any crash leaves a state that the freespace/README
"self-correcting measures" don't detect, errors may happen. Did the clusters
crash recently?

> The occurence of this bug happening makes it hard to reproduce, but it's
> definitely frequent enough we witnessed it on a dozen PostgreSQL clusters.

You could do "ALTER TABLE x SET (vacuum_truncate = off);" and see if the
problem stops happening. That would corroborate the VACUUM theory.

Can you use backtrace_functions to get a stack track?

> In our case, we need to repair the FSM. The instructions on the wiki do work,
> but maybe we should add something like the attached patch (modeled after the
> same feature in pg_visibility) to make it possible to repair the FSM
> corruption online. What do you think about it ?

That's reasonable in concept.