Re: How can we make beta testing better?

On Tue, Apr 15, 2014 at 5:47 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:

> Hackers,
>
> I think 9.3 has given us evidence that our users aren't giving new
> versions of PostgreSQL substantial beta testing, or if they are, they
> aren't sharing the results with us.
>
> How can we make beta testing better and more effective? How can we get
> more users to actually throw serious workloads at new versions and share
> the results?
>
> I've tried a couple of things over the last two years and they haven't
> worked all that well. Since we're about to go into another beta testing
> period, we need something new. Ideas?
>

I think it boils down to making it really easy to create a workload
generator. Most companies have simple single-threaded regression tests for
functionality but very few companies have good parallel workload generators
which reflect activities in their production environment.

A documented beta test process/toolset which does the following would help:
1) Enables full query logging
2) Creates a replica of a production DB, record $TIME when it stops.
3) Allow user to make changes (upgrade to 9.4, change hardware, change
kernel settings, ...)
4) Plays queries from the CSV logs starting from $TIME mimicking actual
timing and transaction boundaries

If Pg can make it easy to duplicate activities currently going on in
production inside another environment, I would be pleased to fire a couple
billion queries through it over the next few weeks.

#4 should include reporting useful to the project, such as a sampling of
queries which performed significantly worse and a few relative performance
stats for overall execution time.