Re: Two weeks to feature freeze

> * clean the source, destination directories
> * pull latest CVS tip down.

Why tip? Lets simply update the current source tree to the most current
of whatever branch they had checked out initially.

Running it on older stable branches is just as useful.

> * record environment / installed packages

Virtually impossible, especially if people take to installing some items
by hand (we want to test them as well). Recording the configure output
on the other hand would be quite useful.

> * loop - on different options ( w/ or w/o krb5, w/ or w/o ssl, etc. )
> o make clean
> o configure with sets of options
> o compile
> + log messages
> + analyze errors ( perhaps gather statitistics:
> warnings, failures, notices, etc.)

Any warnings, failures, etc. aside from what is in the 'known' file
should be reported -- makes sense.

> o (run / install) if successful
> o run tests
> + output results (perhaps to HTML)
> + compare results with expected
> + record differences if any | gather aggregate information

Standard regression tests should suffice. Any non-ignored errors
reported.

> o uninstall / clean up

Skip this. Cleanup prior to the run. If something is wrong, we may
need additional information from the environment.

> * end loop
>
> Perhaps there could be an occasion where the test would be able to put
> in a corrupt WAL or a corrupt table to do regression tests for recovery
> of errors.

These would be interesting extensions to the standard regression suite
-- but perhaps require a flag

> Of course, these are just ideas and I'm not sure how practical it is to
> do any of them. I just am really concerned about the uninstall/clean up
> phase and how that can be done in an orderly fashion. Unless the
> process can start from a clean state again, then it won't be valid. At

I've not tried, but if PostgreSQL can do an 'out of tree' compile it
could make it much easier. That said, poor cleanup would be a result of
a broken make clean.

> one point I had even given thought, vainly, to purchasing VMWare for
> such an occasion. Suggestions?

Skip VMWare. Find a few volunteers to cron the script (I'll volunteer).

I think we should replace Bruce's pgtest script with this one -- with an
argument to accept the email address to report to for FAILING cases.
Success isn't very interesting if it runs regularly.

--
Rod Taylor <rbt(at)rbt(dot)ca>

PGP Key: http://www.rbt.ca/rbtpub.asc