Re: Changing shared_buffers without restart

On Fri, Apr 18, 2025 at 7:25 PM Dmitry Dolgov <9erthalion6(at)gmail(dot)com> wrote:
> > On Thu, Apr 17, 2025 at 03:22:28PM GMT, Ashutosh Bapat wrote:
> >
> > In an offlist chat Thomas Munro mentioned that just ftruncate() would
> > be enough to resize the shared memory without touching address maps
> > using mmap and munmap().
> >
> > ftruncate man page seems to concur with him
> >
> > If the effect of ftruncate() is to decrease the size of a memory
> > mapped file or a shared memory object and whole pages beyond the
> > new end were previously mapped, then the whole pages beyond the
> > new end shall be discarded.
> >
> > References to discarded pages shall result in the generation of a
> > SIGBUS signal.
> >
> > If the effect of ftruncate() is to increase the size of a memory
> > object, it is unspecified whether the contents of any mapped pages
> > between the old end-of-file and the new are flushed to the
> > underlying object.
> >
> > ftruncate() when shrinking memory will release the extra pages and
> > also would cause segmentation fault when memory outside the size of
> > file is accessed even if the actual address map is larger than the
> > mapped file. The expanded memory is allocated as it is written to, and
> > those pages also become visible in the underlying object.
>
> Thanks for sharing. I need to do more thorough tests, but after a quick
> look I'm not sure about that. ftruncate will take care about the memory,
> but AFAICT the memory mapping will stay the same, is that what you mean?
> In that case if the segment got increased, the memory still can't be
> used because it's beyond the mapping end (at least in my test that's
> what happened). If the segment got shrinked, the memory couldn't be
> reclaimed, because, well, there is already a mapping. Or do I miss
> something?

I was imagining that you might map some maximum possible size at the
beginning to reserve the address space permanently, and then adjust
the virtual memory object's size with ftruncate as required to provide
backing. Doesn't that achieve the goal with fewer steps, using only
portable* POSIX stuff, and keeping all pointers stable? I understand
that pointer stability may not be required (I can see roughly how that
argument is constructed), but isn't it still better to avoid having to
prove that and deal with various other problems completely? Is there
a downside/cost to having a large mapping that is only partially
backed? I suppose choosing that number might offend you but at least
there is an obvious upper bound: physical memory size.

*You might also want to use fallocate after ftruncate on Linux to
avoid SIGBUS on allocation failure on first touch page fault, which
raises portability questions since it's unspecified whether you can do
that with shm fds and fails on some systems, but it let's call that an
independent topic as it's not affected by this choice.