Multi master disjoint cluster

Hi all,

tl;dr
I have a simple question,
given a choice if I can write the same data to two databases in parallel,
should I opt for primary / replica setup or multi writer/master setup. This
setup has the ability to make use of kafka consumer groups (like two
replication slots each having their own lsn offsets) to write to both db
node pairs in parallel via the application layer.

The churn of data is really high, there is a lot of wals generated, around
500gb/hr.

If I go with primary/replica, (lr not feasible)

I need to ensure both are on the same major version. Upgrades are tricky
(we don't have qa) so we just have option to schema dump for upgrade
compatibility. Data, we trust postgresql for that :). (I wish we had zfs
everywhere but no )

Any excl table blocking operations, (although with later versions there are
very less blocking operations) can impact queries on replica as well
(excluding delay settings).

Corruption can cause downtime (we have tons of them and raids to protect
them) so if replica is having issues, we can zero the pages on the replica
and do some operations if we isolate the problem pages, else resync the
replica from primary. But if primary is having some issues, we copy data
from replica to disk and copy in to primary after truncating etc. Some
downtime but not a lot. (I am not expert at data recovery) and mostly rely
on amcheck, dd, and raid checks.

We don't use pitr (too many wals × 58) or delayed replication as we can't
afford more servers.

ddl deploys are guaranteed by replication. So no need to try 2pc like stuff
at app layer. (Although apps use deploy tools to ensure eventually the ddls
are consistent and idempotent)

Basically primary/replica relieves the app to think what is there on the
primary is also on the replica eventually, so there can be source of truth.

But with multi writers, any app mishandling like bug in catching exception
etc can result in diversion and no more mirrored setup.
We need to have checks/reconciliation to ensure both write nodes in pair
have almost similar data at the end of the day so we can trust this setup
independent of any app mistakes.

But if app layer gets robust, we have almost no downtime in reads and
writes, we can have both nodes on different versions, (w/o logical
replication) can query both nodes real time, no real replication lag issues
, conflicts etc, can upgrade like blue green, canary test some changes on
one if needed etc.

Am I making sense at all? Or I am sounding confused, and I don't know the
difference between primary/replica vs multi writer. This is not bdr like
thing, they don't really need each other unless we are into some recovery.

My point is, we have 58 such primary/replica shards (each 10tb+)
(consistent hashing at app layer, no fdw) and there is no scope of
downtime for reads, so any issue like post upgrade performance degradation
(if any) gives me chills. and we have no qa to test real data.

There are too many dimensions to shard on and aggregations need to run
across the shards (Yes there is no scope of data isolation).