Re: Bug? ExecChooseHashTableSize() got assertion failed with crazy number of rows

> -----Original Message-----
> From: David Rowley [mailto:david(dot)rowley(at)2ndquadrant(dot)com]
> Sent: Wednesday, August 19, 2015 9:00 AM
> To: Kevin Grittner
> Cc: Kaigai Kouhei(海外 浩平); pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] Bug? ExecChooseHashTableSize() got assertion failed with
> crazy number of rows
>
> On 19 August 2015 at 08:54, Kevin Grittner <kgrittn(at)ymail(dot)com> wrote:
>
>
> Kouhei Kaigai <kaigai(at)ak(dot)jp(dot)nec(dot)com> wrote:
>
> > long lbuckets;
>
> > lbuckets = 1 << my_log2(hash_table_bytes / bucket_size);
>
> > Assert(nbuckets > 0);
>
> > In my case, the hash_table_bytes was 101017630802, and bucket_size was
> 48.
> > So, my_log2(hash_table_bytes / bucket_size) = 31, then lbuckets will
> have
> > negative number because both "1" and my_log2() is int32.
> > So, Min(lbuckets, max_pointers) chooses 0x80000000, then it was set
> on
> > the nbuckets and triggers the Assert().
>
> > Attached patch fixes the problem.
>
> So you changed the literal of 1 to 1U, but doesn't that just double
> the threshold for failure? Wouldn't 1L (to match the definition of
> lbuckets) be better?
>
>
>
>
> I agree, but I can only imagine this is happening because the maximum setting
> of work_mem has been modified with the code you're running.
>
> hash_tables_bytes is set based on work_mem
>
> hash_table_bytes = work_mem * 1024L;
>
> The size of your hash table is 101017630802 bytes, which is:
>
> david=# select pg_size_pretty(101017630802);
>
> pg_size_pretty
> ----------------
> 94 GB
> (1 row)
>
> david=# set work_mem = '94GB';
> ERROR: 98566144 is outside the valid range for parameter "work_mem" (64 ..
> 2097151)
>
Hmm. Why I could set work_mem = '96GB' without error.

It was described in the postgresql.conf.

postgres=# SHOW work_mem;
work_mem
----------
96GB
(1 row)

> So I think the only way the following could cause an error, is if bucket_size
> was 1, which it can't be.
>
> lbuckets = 1 << my_log2(hash_table_bytes / bucket_size);
>
>
> I think one day soon we'll need to allow larger work_mem sizes, but I think there's
> lots more to do than this change.
>
I oversight this limitation, but why I can bypass GUC limitation check?

--
NEC Business Creation Division / PG-Strom Project
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>