'CLUSTER' in one database prevents running it in two others on the same database cluster (PG15.2)

Hi all,

I am running into a weird issue I haven't encountered before.

This is PostgreSQL 15.2 (Ubuntu 15.2-1.pgdg22.04+1) running on Ubuntu
22.04.2 LTS, with PostGIS (POSTGIS="3.3.2 4975da8" [EXTENSION]
PGSQL="150" GEOS="3.10.2-CAPI-1.16.0" PROJ="8.2.1" LIBXML="2.9.13"
LIBJSON="0.15" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)").

Now the PostgreSQL database cluster contains multiple databases. In one
of these, an OpenStreetMap 'Planet' size database, a CLUSTER operation
is running against a very large table of buildings (> 400M records).
This runs fine, and I can see the actual progress in pgAdmin by
inspecting the 'pg_stat_progress_cluster' view. So far everything seems OK.

However, as part of two secondary Python scripts running that do all
kinds of batch operations against spatial tables (generalization and
such) in two other databases on the same database cluster, at the very
end of the processing, these initialize 'CLUSTER' operations as well.
Both of these now fail with a:

"missing chunk number 0 for toast value X in pg_toast_Y"

type error.

After these errors, I see the CLUSTER command listed as well in the
PostgreSQL main log, but the ERROR line mentioned above is listed just
above the actual CLUSTER statement in the log. No other anomaly / error
is visible there. The CLUSTER command just fails though on the two
secondary databases. The time displayed in the log is exactly the same
for these two log lines one after another, also the number between the
square brackets listed immediately after the "CET" of the time are the
same for the same database. The pg_toast_Y table names mentioned in the
errors differ by the way between the two databases.

If I inspect the table that is mentioned in the CLUSTER command in the
log, the very first one that is going to be CLUSTERed in the secondary
databases, the table can be displayed fine in DBeaver, even spatially.
Checking the geometries with PostGIS's 'ST_IsValid' also shows them to
be fine, no errors in the geometries.

Everything else seems OK, database cluster is still up&running, all
tables in all databases can be viewed and accessed in DBeaver and pgAdmin.

In all other rare cases where I have seen the "missing chunk number 0
for toast value X in pg_toast_Y" type error, the generalization
processing already failed somewhere half-way, never at the end when
CLUSTER is initiated. The error therefore seems weird to occur at that
point.

I have now been able to reproduce this twice, with two tiny database's
filled with just Liechtenstein data, while the Planet database is still
in the stage of CLUSTERing the huge building table.

Obviously, it will be interesting to see what happens if the CLUSTER
operation on the Planet database is finished, but this will still take
many hours, especially since other large tables need to be CLUSTERed in
the same database, and I do not want to break off this process right now
after already gotten this far.

I will report back once it is finished, but does anyone have a clue why
this might happen?

The system has plenty of free space by the way (4x2TB NVMe), and the
database runs on a professional workstation with ECC RAM, and used RAM
also doesn't seem an issue.

Marco