Re: rmgr hooks and contrib/rmgr_hook

On Sat, 2008-09-13 at 10:59 +0300, Heikki Linnakangas wrote:

> The importance of the WAL will only increase as more people start to
> use it for PITR, replication etc.

Agreed.

> The 2nd use case, however, I find pretty unconvincing. I can't think of
> a good example of that. Anything that needs to define its own resource
> manager is very low-level stuff, and probably needs to go into the core
> anyway.

New indexes are a big one, but I listed others also.

Indexes have always been able to be added dynamically. Now they can be
recovered correctly as well.

Other data structures can be maintained by trigger code that writes new
types of WAL. That was always possible before, now they can be
recoverable too.

If we have extensible functions, triggers, indexes, why not WAL? What is
the problem with making WAL extensible? It carries no penalty at all for
standard WAL records, since the internal design for WAL already caters
for exactly this.

> So, let's focus on the 1st use case.

No, lets look at both...you can't just wave away half the use cases. If
you look at all of the use cases the argument for doing it externally
quickly falls apart since it severely limits what can be achieved.

> I think a better approach for that
> is to implement the filters as external programs, like pglesslog. It
> allows more flexibility, although it also means that you can't rely on
> existing backend functions to manipulate the WAL. I'd love to see a "WAL
> toolkit" on pgfoundry, with tools like the filter to only restore a
> single database, pglesslog, a WAL record viewer etc. A while ago, you
> also talked about replacing the Slony triggers in the master with a tool
> that reads the WAL; another good example of an external tool that needs
> to read WAL. The toolkit could provide some sort of a framework and
> common user interface to read and write WAL files for all those tools.

This patch provides exactly the toolkit you describe, just internally.

As you point out, doing it other ways means you can't access internal
functions easily and can't maintain internal data structures correctly
either. So doing it externally is *not* a substitute and this is not a
simple discussion of include/exclude from core.

I'm lost as to why suggesting we limit the functionality is going to be
a good thing? If external tools really are so good, then we can do that
*as well*.

But this is only a plugin API, so the tools will be developed externally
anyway.

> A while ago, you
> also talked about replacing the Slony triggers in the master with a tool
> that reads the WAL

Writes the WAL you mean? Slony triggers could write data to WAL rather
than log tables and the slon daemon can be implemented as an rmgr
plugin. Or many other options.

> Another tool like that is pglesslog, although that
> one couldn't actually be implemented with these hooks.

Sounds like we'll want to integrate that into synch replication some
how, so suggestions as to how to do that welcome - if you're not already
doing it via some other plugin in synch rep code?

> I'm sure there's
> more tricks like that people would find useful, if the tools existed.

Agreed. So lets make them exist.

If there's an argument against doing this, I've not heard it made
clearly by anybody. When we discussed it first on hackers there was no
objection, so I wrote the patch. If people want to see this blocked now,
we need some good reasons.

I've got nothing riding on the acceptance of this patch, I just think
its a good thing. That's why I deprioritised it. If there's some hidden
threat to national security or whatever, tell me off list.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support