init_htab causes SIGFPE (or worse) due to miscalculation for large nbuckets

From: Jeff Davis <pgsql(at)j-davis(dot)com>
To: pgsql-bugs(at)postgresql(dot)org
Subject: init_htab causes SIGFPE (or worse) due to miscalculation for large nbuckets
Date: 2012-12-09 19:01:55
Message-ID: 1355079715.3896.14.camel@jdavis
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

If nbuckets is around 2^30+1, which can happen if work_mem is very high,
then:

nbuckets = 1 << my_log2(lnbuckets);

ends up as 1 << 31, which is negative, leading to a SIGFPE on my
machine, but I think it can also lead to an infinite loop or a crash
(after corrupting the HASHHDR).

The only simple way I can reproduce this is with gdb:

1. attach gdb to a session
2. set a breakpoint in ExecInitRecursiveUnion and continue
3. execute in session:
set work_mem='100GB';
with recursive r (i) as
(select 1 union select i+1 from r where i < 10)
select * from r;
4. (gdb) set node->numGroups = (1 << 30) + 1
5. (gdb) continue

I think we should just cap nbuckets at 1 << 30 in init_htab.

There was a previous fix here:

http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=299d1716525c659f0e02840e31fbe4dea3

But it assumed that init_htab could deal with INT_MAX. In practice,
work_mem will usually be the limiting factor anyway, but not if it's set
high.

Regards,
Jeff Davis

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Thomas Kellerer 2012-12-09 22:44:13 Re: BUG #7748: "drop owned by" fails with error message: "unrecognized object class: 1262"
Previous Message Thomas Kellerer 2012-12-09 17:04:31 Re: BUG #7748: "drop owned by" fails with error message: "unrecognized object class: 1262"