Re: [COMMITTERS] pgsql: Introduce dynamic shared memory areas.

On Mon, Dec 5, 2016 at 11:37 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Introduce dynamic shared memory areas.
>
> gaur finds another problem with dsa.c: it uses SIZE_MAX which AFAICS
> is not required to exist by POSIX.
>
> We could adopt the timezone code's workaround
>
> #ifndef SIZE_MAX
> #define SIZE_MAX ((size_t) -1)
> #endif
>
> but I don't find that particularly nice, and in any case I wonder
> why we would want to use SIZE_MAX and not MaxAllocHugeSize. The
> overflow hazards that motivate that not being the max possible
> value seem relevant here too.

It's not quite the same thing, because control->max_total_segment_size
is a total of the memory used by all allocations plus the associated
bookkeeping overhead, not the amount of memory used by a single
allocation. On 64-bit systems, MaxAllocHugeSize is clearly fine,
because it's some unreasonably large value that nobody's ever going to
hit anyway (or at least not until systems with multiple exabytes of
memory start to become common). On 32-bit systems, IIUC,
MaxAllocHugeSize is just under 2GB. It's pretty hard to imagine
allocating more than 2GB of storage in a DSA on a 32-bit system,
especially because with the current value of
DSA_NUM_SEGMENTS_AT_EACH_SIZE and the changes introduced by
88f626f8680fbe93444582317d1adb375111855f mean that such systems are
limited to about (4 * 1MB) + (4 * 2MB) + (4 * 4MB) + (4 * 8MB) + (4 *
16MB) + (4 * 32MB) + (4 * 64MB) + (8 * 128MB) = ~1.5GB of allocations
per DSA anyway. That could be fixed, though, in several different
ways, and then the 2GB limit you're proposing to impose would become
relevant, at least if the system is using something like a 3GB/1GB
user/kernel split rather than 2GB/2GB. It's probably still not a
practical limitation, though, and even if it is it may not be one we
want to care very much about. I think the bigger issue is whether
there's any theoretical justification for using MaxAllocHugeSize to
limit anything other than the size of an individual allocation; to me,
that looks arbitrary.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company