Re: pgbench logging broken by time logic changes

On Wed, Jun 16, 2021 at 2:59 PM Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> wrote:

> I'm unhappy because I already added tap tests for time-sensitive features
> (-T and others, maybe logging aggregates, cannot remember), which have
> been removed because they could fail under some circonstances (eg very
> very very very slow hosts), or required some special handling (a few lines
> of code) in pgbench, and the net result of this is there is not a single
> test in place for some features:-(
>

I understand your struggle and I hope I was clear about two things:

-I am excited by all the progress made in pgbench, and this problem is an
integration loose end rather than a developer failure at any level.
-Doing better in this messy area takes a team that goes from development to
release management, and I had no right to complain unless I brought
resources to improve in the specific areas of the process that I want to be
better.

I think the only thing you and I disagree on is that you see a "first issue
in a corner case" where I see a process failure that is absolutely vital
for me to improve. Since the reality is that I might be the best
positioned person to actually move said process forward in a meaningful
long-term way, I have every intention of applying pressure to the area
you're frustrated at. Crunchy has a whole parallel review team to the
community one now focused on what our corporate and government customers
need for software process control and procedure compliance. The primary
business problem I'm working on now is how to include performance review in
that mix.

I already know I need to re-engage with you over how I need min/max numbers
in the aggregate logging output to accomplish some valuable goals. When I
get around to that this summer, I'd really enjoy talking with you a bit,
video call or something, about really any community topic you're frustrated
with. I have a lot riding now on the productivity of the PostgreSQL hacker
community and I want everyone to succeed at the best goals.

There is no problem with proposing tests, the problem is that they are
> accepted, or if they are accepted then that they are not removed at the
> first small issue but rather fixed, or their limitations accepted, because
> testing time-sensitive features is not as simple as testing functional
> features.
>

For 2020 Crunchy gave me a sort of sabbatical year to research community
oriented benchmarking topics. Having a self contained project in my home
turned out to be the perfect way to spend *that* wreck of a year.

I made significant progress toward the idea of having a performance farm
for PostgreSQL. On my laptop today is a 14GB database with 1s resolution
latency traces for 663 days of pgbench time running 4 workloads across a
small bare metal farm of various operating systems and hardware classes. I
can answer questions like "how long does a typical SSD take to execute an
INSERT commit?" across my farm with SQL. It's at the "works for me!" stage
of development, and I thought this was the right time in the development
cycle to start sharing improvement ideas from my work; thus the other
submissions in progress I alluded to.

The logging feature is in an intermediate spot where validating it requires
light custom tooling that compares its output against known variables like
the system time. It doesn't quite have a performance component to it.
Since this time logic detail is a well known portability minefield, I
thought demanding that particular test was a pretty easy sell.

That you in particular are frustrated here makes perfect sense to me. I am
fresh and ready to carry this forward some distance, and I hope the outcome
makes you happy