Re: Why we are going to have to go DirectIO

Hi,

On 2013-12-03 10:44:15 -0800, Josh Berkus wrote:
> I don't know where we'll get the resources to implement our own storage,
> but it's looking like we don't have a choice.

As long as our storage layer is a s suboptimal as it is today, I think
it's a purely detractory to primarily blame the kernel.

We
* cannot deal with large shared_buffers, the dirty-buffer scanning is far to
expensive. The amount of memory required for locks is pretty big, and
every backend carries around a pretty huge private array for the
buffer pins.
* do not have scalability in pretty damn central datastructures like
buffer mapping.
* Our background eviction mechanism doesn't do anything in lots of
workloads but increase contention on important data structures.
* Due to the missing efficient eviction, we synchronously write out data
when acquiring a victim buffer most of the time. That's already bad if
you have a kernel buffering your writes, but if you don't...
* Due to the frequency of buffer pins in almost all workloads, our
tracking of the importance of individual buffers is far, far too
volatile.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services