pgsql: Avoid stack overflow in ShowTransactionStateRec()

Avoid stack overflow in ShowTransactionStateRec()

The function recurses, but didn't perform stack-depth checks. It's
just a debugging aid, so instead of the usual check_stack_depth()
call, stop the printing if we'd risk stack overflow.

Here's an example of how to test this:

(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "SET log_min_messages = 'DEBUG5'; SAVEPOINT sp;") | psql >/dev/null

In the passing, swap building the list of child XIDs and recursing to
parent. That saves memory while recursing, reducing the risk of out of
memory errors with lots of subtransactions. The saving is not very
significant in practice, but this order seems more logical anyway.

Report by Egor Chindyaskin and Alexander Lakhin.

Discussion: https://www.postgresql.org/message-id/1672760457.940462079%40f306.i.mail.ru
Author: Heikki Linnakangas
Reviewed-by: Robert Haas, Andres Freund, Alexander Korotkov

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/6f38c43eb135f5d6b873924180562d8334d6e31d

Modified Files
--------------
src/backend/access/transam/xact.c | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)