In-core regression tests for replication, cascading, archiving, PITR, etc.

Hi all,

The data replication bug causing data corruption on hot slaves found
lately (http://wiki.postgresql.org/wiki/Nov2013ReplicationIssue) is
causing a certain amount of damage among the users of Postgres, either
companies or individuals, and impacts a lot of people. So perhaps it
would be a good time to start thinking about adding some dedicated
regression tests for replication, archiving, PITR, data integrity,
parameter reloading, etc. The possible number of things or use-cases
that could be applied to that is very vast, but let's say that it
involves the manipulation of multiple Postgres nodes and structures
(like a WAL archive) defining a Postgres cluster.

The main purpose of those test would be really simple: using a given
GIT repository, a buildfarm or developer should be able to run a
single "make check" command that runs those tests on a local machine
in a fashion similar to isolation or regression tests and validate
builds or patches.

I imagine that there would be roughly two ways to implement such a facility:
1) Use of a smart set of bash scripts. This would be easy to implement
but reduces pluggability of custom scripts (I am sure that each
user/company has already its own set of scenarios.). pg_upgrade uses
something similar with its test.sh.
2) Use a scripting language, in a way similar to how isolation tests
are done. This would make custom test more customizable.
Here is for example an approach that has been presented at the
unconference of PGcon 2013 (would be something different though as
this proposal does not include node manipulation *during* the tests
like promotion):
https://wiki.postgresql.org/images/1/14/Pg_testframework.pptx
3) Import (and improve) solutions that other projects based on
Postgres technology use for those things.

In all cases, here are the common primary actions that could be run for a test:
- Define and perform actions on a node: init, start, stop, promote,
create_archive, base_backup. So it is a sort of improved wrapper of
pg_ctl.
- Pass parameters to a configuration file, either postgresql.conf,
recovery.conf, or anything.
- Launch SQL commands to a node.
Things like the creation of folders for WAL archiving should be simply
harcoded to simplify the life of developer... As well, the facility
should be smart enough to be allow the use of custom commands that are
the combination of primary actions above, like for example the
possibility to define a sync slave, linked to a root node, is simply
1) create a base backup from a node, 2) pass parameters to
postgresql.conf and recovery.conf, 3) start the node.

Let me know your thoughts.
Regards,
--
Michael