Re: 8.5 release timetable, again

Hi,

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
>> We have regression tests.  They could and should be expanded.  That's a
>> developer job, and we can start working on that now.  But this
>> discussion was about what to do during beta.  And I think during beta
>> you want to test PostgreSQL against a large set of real applications.
>> But we could try to clarify how to actually do that in an organized
>> way.

Exactly, and I think that what we're missing here is a simple tool for
our users to check a new PostgreSQL release against their existing
application.

We already know how to either log all queries and analyze the log files
(CSV makes it easier, pgfouine parses them too) or to have a fe/be
protocol proxy to record application SQL traffic (tsung recorder does
that).

What we miss is a tool to run the captured queries through both versions
of PG and report any resultset mismatch, of course with a way to account
for ordering issues (but we've seen people rely on the ordering when
they don't give an order by clause, then bug the lists about it if a new
release changes it).

> What I want to do is address the concern about too much of any given
> year being consumed by beta and CommitFest. I'm not sure I know how
> to do that though.

To do this I think we *need* to provide beta tester a validator kind of
tool which they can use even without having an application unit test
suite. And a way to easily report success or failure, and in case of
success, a notion of their tests coverage (which is reduced to the list
of PG features the application exercices, but an automated way to
extract the information while running the tool would allow for hackers
friendly categories).

Anyone interrested into starting a project and coding the tool we so
much need?

Regards,
--
dim

PS: of course provided with such a tool, I'd run it for several
applications as soon as possible, which might include every alpha
release.

PPS: I'll be happy to participate into such a project, but I seem to
have a rather full plate already and a shrinking OpenSource free time,
so waiting for me to get there won't help until it's about too late.