Re: Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS

Le samedi 8 juin 2013 19:06:58, Tom Lane a écrit :
> Recently we had a gripe about how you can't read very many files
> concurrently with contrib/file_fdw:
> http://www.postgresql.org/message-id/OF419B5767.8A3C9ADB-ON85257B79.005491E
> 9-85257B79(dot)0054F6F1(at)isn(dot)rtss(dot)qc(dot)ca
>
> The reason for this is that file_fdw goes through the COPY code, which
> uses AllocateFile(), which has a wired-in assumption that not very many
> files need to be open concurrently. I thought for a bit about trying to
> get COPY to use a "virtual" file descriptor such as is provided by the
> rest of fd.c, but that didn't look too easy, and in any case probably
> nobody would be excited about adding additional overhead to the COPY
> code path. What seems more practical is to relax the hard-coded limit
> on the number of files concurrently open through AllocateFile(). Now,
> we could certainly turn that array into some open-ended list structure,
> but that still wouldn't let us have an arbitrary number of files open,
> because at some point we'd run into platform-specific EMFILES or ENFILES
> limits on the number of open file descriptors. In practice we want to
> keep it under the max_safe_fds limit that fd.c goes to a lot of trouble
> to determine. So it seems like the most useful compromise is to keep
> the allocatedDescs[] array data structure as-is, but allow its size to
> depend on max_safe_fds. In the attached proposed patch I limit it to
> max_safe_fds / 2, so that there's still a reasonable number of FDs
> available for fd.c's main pool of virtual FDs. On typical modern
> platforms that should be at least a couple of hundred, thus making the
> effective limit about an order of magnitude more than it is currently
> (32).
>
> Barring objections or better ideas, I'd like to back-patch this as far
> as 9.1 where file_fdw was introduced.

I just wonder about this statement that you removed:
* Since we don't want to encourage heavy use of those functions,
* it seems OK to put a pretty small maximum limit on the number of
* simultaneously allocated descs.

Is it now encouraged to use those functions, or at least that it seems less
'scary' than in the past ?
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation