Re: On columnar storage

On 2015-06-14 17:36, Tomas Vondra wrote:
>
> On 06/14/15 17:22, Alvaro Herrera wrote:
>> Robert Haas wrote:
> ,,,
>>> Second, there's the idea of a way of storing tuples that is different
>>> from PostgreSQL's usual mechanism - i.e. a storage manager API. I
>>> understand your concerns about going through the FDW API so maybe
>>> that's not the right way to do it, but it seems to me that in the end
>>> you are going to end up with something awfully similar to that + a
>>> local relfilenode + WAL logging support. I'm not clear on why you
>>> want to make the column store API totally different from the FDW API;
>>> there may be a reason, but I don't immediately see it.
>>
>> I just don't see that the FDW API is such a good fit for what I'm trying
>> to do. Anything using the FDW API needs to implement its own visibility
>> checking, for instance. I want to avoid that, because it's its own
>> complex problem. Also, it doesn't look like the FDW API supports things
>> that I want to do (neither my proposed LateColumnMaterialization nor my
>> proposed BitmapColumnScan). I would have to extend the FDW API, then
>> contort my stuff so that it fits in the existing API; then I will need
>> to make sure that existing FDWs are not broken by the changes I would
>> propose elsewhere. Round peg, square hole is all I see here. All in
>> all, this seems too much additional work, just to make to things that
>> are really addressing different problems go through the same code.
>>
>> You're correct about "local WAL logging". We will need a solution to
>> that problem. I was hoping to defer that until we had something like
>> Alexander Korotkov's proposed pluggable WAL stuff.
>
> Probably worth mentioning we don't expect the column store API to be
> used only by extensions, but even by code from within the core which can
> use the current WAL infrastructure etc.
>

I think it would be even ok if the initial implementation had
restrictions in a way that extensions can't use it. I think we want
implementation in the core first, support for extensions can come later.
But it's still worth it to have an API since single implementation
probably won't fit every purpose (see indexes).

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services