Re: Pluggable Indexes

On Fri, 2009-01-23 at 16:49 -0500, Tom Lane wrote:
> Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
> > On Fri, 2009-01-23 at 10:33 -0500, Tom Lane wrote:
> >> Right, the WAL-record-processing API is not really at issue, since it's
> >> been proven internally to the core code. My concern is with the other
> >> part, namely exactly how are we going to identify and install additional
> >> rmgrs.
>
> > The patch is just
> > * a hook in StartupXLOG to allow loading arbitrary code into Startup
> > * some slight redefinition of RmgrTable to allow arbitrary code to add
> > or modify the contents of that table of functions. (Being able to modify
> > the table is an not necessary for index extensions, but is for other
> > uses).
> > * some safeguards people requested
>
> Well, that really seems to just prove my point. You've defined a hook
> and not thought carefully about how people will use it.

This was originally proposed on 19 August and a patch submitted to the
September commit fest.
http://archives.postgresql.org/pgsql-hackers/2008-08/msg00794.php

After about 30 emails of technical rebuttal we have a list of possible
uses that can't be done sensibly any other way.
* WAL filtering
* Recovery when we have buggy index AMs, yet without losing data
* Pluggable indexes
* Extracting user data from WAL records (very challenging though)

Those uses require the ability to both add to *and* modify all of the
RmgrTable entries. If this was just for pluggable indexes then the API
probably would look a little different, but it's not. The simplicity of
the hook proposal says nothing about the careful thought behind it, it
just relates to the wide variety of beneficial uses.

At any point there we might have hit serious problems with the patch,
but we didn't. I've done my best to cover the objections raised with
code or suggested control mechanisms, so I'm not expecting anyone to
agree with my first musings.

> The main thing
> that I can see right now that we'd need is some way to determine who
> gets which rmgr index. (Maybe community assignment of numbers ---
> similar to what we've defined for pg_statistic kind codes --- is fine,

http://archives.postgresql.org/pgsql-hackers/2008-08/msg00916.php

> or maybe it isn't; in any case we need an answer for that before this
> hook can be considered usable.) Furthermore, maybe that's not the only
> problem. I'd feel a lot better about this if the hook patch were done
> in parallel with development of actual WAL support in an actual external
...

I agree we need an external module and I learned that lesson from the
earier API proposal you mentioned. The supplied WAL filter plugin was/is
a valid use for this and, as discussed, is the only practical way of
doing WAL filtering. As I said, am happy to make a few mods to make that
more acceptable.

I've deferred on this patch sometimes because of my other work, but also
because I sensed there might be some feeling that people thought this
was a threat to the project from some commercial usurpation (e.g. like
InnoDB). I asked to be contacted off-list if that was the case but
nobody has, so I have assumed this to be a decision based on technical
merit alone. After considering all that has been said I feel this idea
has merit. Yes, we need more and better plugins and this patch is the
seed for those.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support