Fix memory counter update in reorderbuffer

Hi,

I found a bug in the memory counter update in reorderbuffer. It was
introduced by commit 5bec1d6bc5e, so pg17 and master are affected.

In ReorderBufferCleanupTXN() we zero the transaction size and then
free the transaction entry as follows:

/* Update the memory counter */
ReorderBufferChangeMemoryUpdate(rb, NULL, txn, false, txn->size);

/* deallocate */
ReorderBufferReturnTXN(rb, txn);

However, if the transaction entry has toast changes we free them in
ReorderBufferToastReset() called from ReorderBufferToastReset(), and
end up subtracting the memory usage from zero. Which results in an
assertion failure:

TRAP: failed Assert("(rb->size >= sz) && (txn->size >= sz)"), File:
"reorderbuffer.c"

This can happen for example if an error occurs while replaying
transaction changes including toast changes.

I've attached the patch that fixes the problem and includes a test
case (the test part might not be committed as it slows the test case).

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com