Re: StrategyGetBuffer optimization, take 2

Performance testing this patch is a real bugaboo for me; the VMs I have to
work with are too unstable to give useful results :-(. Need to scrounge up
a doner box somewhere...

On Tue, Aug 13, 2013 at 12:26 AM, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>wrote:

> Merlin Moncure wrote:
> On Wed, Aug 7, 2013 at 11:52 PM, Amit Kapila
> <amit(dot)kapila(at)huawei(dot)com> wrote:
> >>> -----Original Message-----
> >>> From: pgsql-hackers-owner(at)postgresql(dot)org [mailto:pgsql-hackers-
> >>> owner(at)postgresql(dot)org] On Behalf Of Merlin Moncure
> >>> Sent: Thursday, August 08, 2013 12:09 AM
> >>> To: Andres Freund
> >>> Cc: PostgreSQL-development; Jeff Janes
> >>> Subject: Re: [HACKERS] StrategyGetBuffer optimization, take 2
> >>>
> >>> On Wed, Aug 7, 2013 at 12:07 PM, Andres Freund
> <andres(at)2ndquadrant(dot)com>
> >>> wrote:
> >>> > On 2013-08-07 09:40:24 -0500, Merlin Moncure wrote:
>
> >>> I have some very strong evidence that the problem is coming out of the
> >>> buffer allocator. Exhibit A is that vlad's presentation of the
> >>> problem was on a read only load (if not allocator lock, then what?).
> >>> Exhibit B is that lowering shared buffers to 2gb seems to have (so
> >>> far, 5 days in) fixed the issue. This problem shows up on fast
> >>> machines with fast storage and lots of cores. So what I think is
> >>> happening is that usage_count starts creeping up faster than it gets
> >>> cleared by the sweep with very large buffer settings which in turn
> >>> causes the 'problem' buffers to be analyzed for eviction more often.
> >
> >> Yes one idea which was discussed previously is to not increase usage
> >> count, every time buffer is pinned.
> >> I am also working on some of the optimizations on similar area, which
> you
> >> can refer here:
> >>
> >>
> http://www.postgresql.org/message-id/006e01ce926c$c7768680$56639380$@kapila@
> >> huawei.com
>
> > yup -- just took a quick look at your proposed patch. You're
> > attacking the 'freelist' side of buffer allocation where my stripped
> > down patch addresses issues with the clocksweep. I think this is a
> > good idea but more than I wanted to get into personally.
>
> > Good news is that both patches should essentially bolt on together
> > AFAICT.
>
> True, I also think so as both are trying to reduce contention in same area.
>
> > I propose we do a bit of consolidation of performance testing
> > efforts and run tests with patch A, B, and AB in various scenarios. I
> > have a 16 core vm (4gb ram) that I can test with and want to start
> > with say 2gb database 1gb shared_buffers high concurrency test and see
> > how it burns in. What do you think?
>
> I think this can mainly benefit with large data and shared buffers (>
> 10G), last year also I had ran few tests with similar idea's but
> didn't get much
> in with less shared buffers.
>
> > Are you at a point where we can
> > run some tests?
>
> Not now, but I will try to run before/during next CF.
>
>
> With Regards,
> Amit Kapila.
> EnterpriseDB: http://www.enterprisedb.com
>