Re: -1/0 virtualtransaction

On Wed, Apr 28, 2021 at 2:25 AM Mike Beachy <mbeachy(at)gmail(dot)com> wrote:
> On Tue, Apr 27, 2021 at 2:56 AM Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at> wrote:
>>
>> Not sure, but do you see prepared transactions in "pg_prepared_xacts"?
>
> No, the -1 in the virtualtransaction (https://www.postgresql.org/docs/11/view-pg-locks.html) for pg_prepared_xacts was another clue I saw! But, it seems more or less a dead end as I have nothing in pg_prepared_xacts.
>
> Thanks for the idea, though.

There is another way to get a pg_lock that shows -1/0 there: if we run
out of SERIALIZABLEXACT objects, we transfer the locks of the oldest
committed transaction to a single SERIALIZABLEXACT object called
OldCommittedSxact, and it has an invalid virtual xid. You can see
this if you recompile with TEST_SUMMARIZE_SERIAL defined to force that
path, and then run three overlapping transactions like this:

tx1: BEGIN;
tx1: SELECT * FROM t WHERE id = 42;
tx2: BEGIN;
tx2: SELECT * FROM t WHERE id = 999;
tx1: COMMIT;

Even though it comitted, at this point we still see tx1's locks, along
with tx2's.

tx3: BEGIN;
tx3: SELECT * FROM t WHERE id = 1234;

At this point we see tx1's locks still but their vxid has changed to -1/0.

> I still need to put more effort into Tom's idea about SIReadLock hanging out after the transaction, but some evidence pointing in this direction is that I've reduced the number of db connections and found that the '-1/0' locks will eventually go away! I interpret this as the db needing to find time when no overlapping read/write transactions are present. This doesn't seem completely correct, as I don't have any long lived transactions running while these locks are hanging out. Confusion still remains, for sure.

But do you have lots of short overlapping transactions so that there
is never a moment where there are zero transactions running?

As mentioned, locks (and transactions, and conflicts) hang around
after you commit. That's because things that your finished
transaction saw can cause transactions that are still running to
become uncommittable, by forming part of a cycle. The rules for when
the locks can be freed change once you reach the degraded "summarised"
mode, though, due to lack of bookkeeping space. Not sure of the exact
details without rereading the source code 42 times with much coffee,
but it's something like: summarised locks can only be freed at times
when there are zero active read/write serializable transactions,
because that's when "CanPartialClearThrough" advances, while in the
normal case they can be freed as soon as there are no SSI snapshots
that were taken before it committed, because that's when
"SxactGlobalXmin" advances.

The number of SERIALIZABLEXACT objects is (max_connections +
max_prepared_transactions) * 10. So, you could try increasing
max_connections (without increasing the actual number of connections)
to see if you can get to a point where you don't see these invalid
virtual xids, and then maybe it'll be able to clean up locks more
aggressively.