Re: zLinux Load Testing Experience

On Wed, May 1, 2013 at 1:21 PM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
> On Wed, May 1, 2013 at 11:34 AM, Andrew Hastie <andrew(at)ahastie(dot)net> wrote:
>>
>> On 01/05/13 15:34, Merlin Moncure wrote:
>>
>> On Wed, May 1, 2013 at 8:01 AM, Andrew Hastie <andrew(at)ahastie(dot)net> wrote:
>>
>> On 30/04/13 20:46, Merlin Moncure wrote:
>>
>> On Tue, Apr 30, 2013 at 12:26 PM, Jeff Janes <jeff(dot)janes(at)gmail(dot)com> wrote:
>>
>> On Tue, Apr 30, 2013 at 8:28 AM, Andrew Hastie <andrew(at)ahastie(dot)net>
>> wrote:
>>
>> I'm currently working on a project porting an application from RedHat
>> Linux on Intel onto IBM zLinux. Our application requires PostgreSQL at
>> version 9.n, so the PostgreSQL binaries have been built using the
>> standard
>> build tools from source. Everything appears run correctly. However as
>> part
>> of performance testing, our IBM and Linux SysProgs have been "poking
>> around"
>> using strace and have reported the following (which they think is an
>> error
>> condition) when hooking up to the postmaster processes:-
>>
>> read(3, 0x3ffff875ee0, 16) = -1 EAGAIN (Resource temporarily
>> unavailable)
>> poll([{fd=3, events=POLLIN}, {fd=6, events=POLLIN}], 2, 200) = 0
>> (Timeout)
>> read(3, 0x3ffff875ee0, 16) = -1 EAGAIN (Resource temporarily
>> unavailable)
>> poll([{fd=3, events=POLLIN}, {fd=6, events=POLLIN}], 2, 10000) = 0
>> (Timeout)
>> ... repeated many times
>>
>> That does not look like the postmaster process. It looks like probably
>> the
>> background writer process.
>>
>> It is normal, and doesn't explain high CPU utilization.
>>
>> yeah: we're probably a couple of steps in front of deep system
>> profiling. Helpful things to provide to help diagnose would be:
>>
>> *) 'explain analyze' of the queries that are eating cpu
>> *) more details about the hardware -- how many cpu, etc.
>> *) better definition of 'perceived high CPU utilisation'
>> *) some correlating performance tests, expecially cpu bound pgbench
>> tests (pgbench -S)
>>
>> merlin
>>
>>
>> I'm not sure how much experience the community has on tuning PostgreSQL
>> running on RedHat which in turn is hosted on an IBM mainframe under VM
>> (using zLinux). So I'm happy to start posting further details and benchmark
>> results and see where we go. Should I be moving this thread over into the
>> pg-performance list, or is pg-general the right place?
>>
>> certainly performance. and yes, zLinux is less well traveled. Did
>> you compile postgres from source? Did you confirm that there is a
>> native spinlocks implementation and it is being used?
>>
>> merlin
>>
>> Did you compile postgres from source? - Yes (I need PG v9.n as v8.n shipped
>> with RedHat Ent6 does not have several v9 specific features we need).
>>
>> Did you confirm that there is a native spinlocks implementation and it is
>> being used? - I believe so as no errors or warnings logged during the build.
>> Is there a simple way to check whether spin-locks are running native?
>>
>> I've started looking at several articles covering pgbench and running some
>> initial tests, so I plan to start a new thread on pg-performance in the next
>> day or so.
>>
>> Thanks for the advice so far - Appreciated :-)
>
> I can't remember off the top of my head if configure forces you to
> specifically unset spinlocks to get through a build on a non-hardware
> spinlock platform. Point being: the interesting stuff happens during
> configure, not build.
>
> Check the contents of src/include/pg_config.h and look for this line:
> #define HAVE_SPINLOCKS 1
>
> to see if you have hardware spinlocks.

Just a follow up here since I'm about to go on vacation and will be
out of pocket for the next several days. If you do indeed find out
that you are using non TAS spinlocks, and are suspicious that this is
causing your load issues, and are feeling experimental, and are using
gcc to compile postgres, and have determined that the
HAVE_GCC_INT_ATOMICS macro is set, I'd maybe consider hacking s_lock.h
to use the gcc __sync_lock_test_and_set variant of TAS (see around
line 300) in s_lock.h.

merlin