Changeset Extraction v7.0 (was logical changeset generation)

Hello Everyone,

I am pleased to announce the next version of the the changeset
extraction feature.

There's more changes than I can remember, but that's what comes to my
tired mind:
* Initial userlevel docs (including an example session!).
* generalization of the "replication slot" system to also work for
streaming rep, although the user interface for that is mostly missing.
* don't remove WAL still required by a replication slot, be it a slot for
changeset extraction or streaming rep.
* New output plugin interface with one _PG_output_plugin_init dlsym()ed
function filling out the callbacks.
* Renaming of the init and _cleanup output plugins to startup and shutdown.
* Simplification of the prepare_write/write interface for output plugins
(no need to specify LSNs anymore).
* Renaming of the changeset extraction operations to
create_replication_slot/drop_replication_slot/start_replication
... logical.
* moving the SQL changeset functions from a contrib module into core
* Addition of peeking functions for changeset extraction.
* revised error messages
* revised comments
* ...

I've followed Robert's wishes with generalizing the replication slot
interface to not only work for changeset generation, but also streaming
rep - not sure whether that was the right choice, it's been more work
than I expected blocking things a bit but we're there now....
There's no clientside support included except as in the pg_receivexlog
hack attached as the last patch, but I also have tested it via streaming
rep and it mostly works (minus a hot_standby_feedback bug, will report
tomorrow).

What I think is missing:
* The user docs need more work, even though we're in a much better state
than before.
* Replication slots are stored in binary files. I think it might make
sense to store them as text files instead, for easier
extensibility. Especially since we want to use them for streaming rep,
I am pretty sure new attributes will soon come. I don't think it's
critical enough performancwise to store them in binary.
* Contrary to what Robert and I'd discussed I've named the SQL functions
outputting changes decoding_slot_(get|peek)_[binary_]changes instead of
decoding_stream_* - I can change that, but the SQL functions don't
actually support streaming, so I thought that might be
confusing. Opinions?
* Robert complained earlier about the way historical catalog snapshots
are forced in tqual.c - I don't really know yet what the better way
would be here.
* Some functionality probably needs to move between the patches - it's a
bit hard to see where the best boundaries are.

The sources are in my git tree at:
http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=summary
branch xlog-decoding-rebasing-remapping.

The last two patches are *not* indendet to be actually applied, but are
useful for testing.

If you want to test, you'll need a clean initdb, set wal_level=logical
and max_replication_slots>0. There's a example SQL session showing how
things can be used....

Testing, Review, Questions welcome!

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services