Re: 012_subtransactions.pl vs clang -fsanitize=undefined

Thomas Munro <thomas(dot)munro(at)gmail(dot)com> writes:
> On Mon, Jul 25, 2022 at 8:59 AM Justin Pryzby <pryzby(at)telsasoft(dot)com> wrote:
>> I found that -fsanitize causes the test to fail, going back to REL_10_STABLE,
>> for any clang in:

> Yeah I've seen this too... it'd be good to figure out how to fix it:
> https://www.postgresql.org/message-id/CA%2BhUKGLDA-GuQKRvDF3abHadDrrYZ33N9e4DEOGwKH3JqdYSCQ%40mail.gmail.com

Yeah, reproduces here too with RHEL8's clang 13.0.1. I also see
that the failures are due to "stack depth exceeded" errors from that
recursive hs_subxids() function. As best I can tell, the stack depth
failure is entirely honest:

(gdb) p stack_base_ptr
$1 = 0x7ffd92032100 "\360[\\\006"
(gdb) p $sp
$2 = (void *) 0x7ffd91e305a0
(gdb) p 0x7ffd92032100 - 0x7ffd91e305a0
$3 = 2104160

I can get at most 82 recursion levels without failure.
With my normal build it can get to 678 levels before dying.

I think what's happening is just that this build configuration
eats stack extravagantly. Maybe it keeps all locals on the stack
instead of in registers? I'm too lazy to check out the assembly
code.

I thought for a moment of rewriting hs_subxids() to iterate instead
of recurse, but it doesn't look like that's tremendously feasible in
plpgsql --- the only way to make nested subtransactions is to recurse.
It could probably be done from the client by issuing a series of
SAVEPOINT commands, but not nearly as elegantly.

regards, tom lane