Re: Pluggable storage

On Sat, Jul 15, 2017 at 5:14 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Thu, Jun 22, 2017 at 9:30 AM, Alexander Korotkov
> <a(dot)korotkov(at)postgrespro(dot)ru> wrote:
> > If #1 is only about changing tuple and page formats, then could be much
> > simpler than the patch upthread? We can implement "page format access
> > methods" with routines for insertion, update, pruning and deletion of
> tuples
> > *in particular page*. There is no need to redefine high-level logic for
> > scanning heap, doing updates and so on...
>
> That assumes that every tuple format does those things in the same
> way, which I suspect is not likely to be the case. I think that
> pruning and vacuum are artifacts of the current heap format, and they
> may be nonexistent or take some altogether different form in some
> other storage engine.

I think that pruning and vacuum are artifacts of not only current heap
formats, but they are also artifacts of current index AM API. And this is
more significant circumstance given that we're going to preserve
compatibility of new storage engines with current index AMs. Our current
index AM API assumes that we can delete from index only in bulk manner.
Our payload to index key is TID, not arbitrary piece of data. And that
payload can't be updated.

InnoDB isn't much like the PostgreSQL heap, and
> neither is SQL Server, IIUC. If we're creating a heap format that can
> only be different in trivial ways from what we have now, and anything
> that really changes the paradigm is off-limits, well, that's not
> really interesting enough to justify the work of creating a heap
> storage API.

My concern is that we probably can't do anything that really changes
paradigm while preserving compatibility with index AM API. If you don't
agree with that, it would be good to provide some examples. It seems
unlikely for me that we're going to have something like InnoDB or SQL
Server table with our current index AM API. InnoDB utilizes
index-organized tables where primary and secondary indexes are versioned
independently. SQL Server utilizes flat data structure similar to our
heap, but MVCC implementation also seems very different.

I think in general there are two ways dealing with out index AM API
limitation. One of them is to extend index AM API. At least, we would
need a method for deletion of individual index tuple (for sure, we already
have kill_prior_tuple but it's just a hint for now). Also, it would be
nice to have arbitrary payload to index tuples instead of TID, and method
to update that payload. But that would be quite big amount of work.
Alternatively, we could allow pluggable storages to have their own index
AMs, and that will move this amount of work to the pluggable storage side.

The thing which we would evade is storage API, which would be invasive like
something changing paradigm, but actually allowing just trivial changes in
heap format. Mechanical replacement of heap methods with storage API
methods could lead us there.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company