Re: logical decoding / rewrite map vs. maxAllocatedDescs

Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com> writes:
> While investigating an issue with rewrite maps in logical decoding, I
> found it's pretty darn trivial to hit this:
> ERROR: 53000: exceeded maxAllocatedDescs (492) while trying to open
> file "pg_logical/mappings/map-4000-4eb-1_60DE1E08-5376b5-537c6b"

> This happens because we seem to open the mapping files and keep them
> open until the next checkpoint, at which point we fsync + close.

It sounds like whoever wrote that code was completely impervious to the
API spec for AllocateFile():

* Note that files that will be open for any significant length of time
* should NOT be handled this way, since they cannot share kernel file
* descriptors with other files; there is grave risk of running out of FDs
* if anyone locks down too many FDs.

Perhaps these files should be accessed through fd.c's VFD infrastructure
rather than ignoring it. Reinventing it is especially not the way to go.

> Furthermore, the situation is made worse by opening the same file
> repeatedly, under different file descriptor.

That's just stupid :-(

Note that the limit on maxAllocatedDescs is based on the kernel limit
on how many files a process can have open, so assuming you can increase
it is not going to end well.

regards, tom lane