From: | KONDO Mitsumasa <kondo(dot)mitsumasa(at)lab(dot)ntt(dot)co(dot)jp> |
---|---|
To: | Andres Freund <andres(at)2ndquadrant(dot)com> |
Cc: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Amit Langote <amitlangote09(at)gmail(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Postgres General <pgsql-general(at)postgresql(dot)org> |
Subject: | Re: [HACKERS] Bottlenecks with large number of relation segment files |
Date: | 2013-08-06 11:10:14 |
Message-ID: | 5200D996.5020409@lab.ntt.co.jp |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general pgsql-hackers |
(2013/08/06 19:33), Andres Freund wrote:
> On 2013-08-06 19:19:41 +0900, KONDO Mitsumasa wrote:
>> (2013/08/05 21:23), Tom Lane wrote:
>>> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
>>>> ... Also, there are global
>>>> limits to the amount of filehandles that can simultaneously opened on a
>>>> system.
>>>
>>> Yeah. Raising max_files_per_process puts you at serious risk that
>>> everything else on the box will start falling over for lack of available
>>> FD slots.
>> Is it Really? When I use hadoop like NOSQL storage, I set large number of FD.
>> Actually, Hadoop Wiki is writing following.
>>
>> http://wiki.apache.org/hadoop/TooManyOpenFiles
>>> Too Many Open Files
>>>
>>> You can see this on Linux machines in client-side applications, server code or even in test runs.
>>> It is caused by per-process limits on the number of files that a single user/process can have open, which was introduced in the 2.6.27 kernel. The default value, 128, was chosen because "that should be enough".
>
> The first paragraph (which you're quoting with 128) is talking about
> epoll which we don't use. The second paragraph indeed talks about the
> max numbers of fds. Of *one* process.
Yes. I have been already understood like that.
> Postgres uses a *process* based model. So, max_files_per_process is about
> the the number of fds in a single backend. You need to multiply it by
> max_connections + a bunch to get to the overall number of FDs.
Yes, too. I think max_file_per_process seems too small. In NoSQL, it was
recommended large number of FD. However, I do not know whether it is really
enough in PostgreSQL. If we use PostgreSQL with big data, we might need to change
max_file_per_process, I think.
Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center
From | Date | Subject | |
---|---|---|---|
Next Message | ascot.moss@gmail.com | 2013-08-06 11:16:51 | Re: psql: FATAL: the database system is starting up |
Previous Message | Haribabu kommi | 2013-08-06 10:52:52 | Re: psql: FATAL: the database system is starting up |
From | Date | Subject | |
---|---|---|---|
Next Message | Florian Weimer | 2013-08-06 11:19:55 | Re: Bottlenecks with large number of relation segment files |
Previous Message | Tomonari Katsumata | 2013-08-06 11:05:01 | Re: Should we remove "not fast" promotion at all? |