Re: Some performance degradation in REL_16 vs REL_15

"Anton A. Melnikov" <a(dot)melnikov(at)postgrespro(dot)ru> writes:
> I can't understand why i get the opposite results on my pc and on the server. It is clear that the absolute
> TPS values will be different for various configurations. This is normal. But differences?
> Is it unlikely that some kind of reference configuration is needed to accurately
> measure the difference in performance. Probably something wrong with my pc, but now
> i can not figure out what's wrong.

> Would be very grateful for any advice or comments to clarify this problem.

Benchmarking is hard :-(. IME it's absolutely typical to see
variations of a couple of percent even when "nothing has changed",
for example after modifying some code that's nowhere near any
hot code path for the test case. I usually attribute this to
cache effects, such as a couple of bits of hot code now sharing or
not sharing a cache line. If you use two different compiler versions
then that situation is likely to occur all over the place even with
exactly the same source code. NUMA creates huge reproducibility
problems too on multisocket machines (which your server is IIUC).
When I had a multisocket workstation I'd usually bind all the server
processes to one socket if I wanted more-or-less-repeatable numbers.

I wouldn't put a lot of faith in the idea that measured pgbench
differences of up to several percent are meaningful at all,
especially when comparing across different hardware and different
OS+compiler versions. There are too many variables that have
little to do with the theoretical performance of the source code.

regards, tom lane