Re: On columnar storage

On 06/14/15 17:22, Alvaro Herrera wrote:
> Robert Haas wrote:
,,,
>> Second, there's the idea of a way of storing tuples that is different
>> from PostgreSQL's usual mechanism - i.e. a storage manager API. I
>> understand your concerns about going through the FDW API so maybe
>> that's not the right way to do it, but it seems to me that in the end
>> you are going to end up with something awfully similar to that + a
>> local relfilenode + WAL logging support. I'm not clear on why you
>> want to make the column store API totally different from the FDW API;
>> there may be a reason, but I don't immediately see it.
>
> I just don't see that the FDW API is such a good fit for what I'm trying
> to do. Anything using the FDW API needs to implement its own visibility
> checking, for instance. I want to avoid that, because it's its own
> complex problem. Also, it doesn't look like the FDW API supports things
> that I want to do (neither my proposed LateColumnMaterialization nor my
> proposed BitmapColumnScan). I would have to extend the FDW API, then
> contort my stuff so that it fits in the existing API; then I will need
> to make sure that existing FDWs are not broken by the changes I would
> propose elsewhere. Round peg, square hole is all I see here. All in
> all, this seems too much additional work, just to make to things that
> are really addressing different problems go through the same code.
>
> You're correct about "local WAL logging". We will need a solution to
> that problem. I was hoping to defer that until we had something like
> Alexander Korotkov's proposed pluggable WAL stuff.

Probably worth mentioning we don't expect the column store API to be
used only by extensions, but even by code from within the core which can
use the current WAL infrastructure etc.

--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services