Re: [HACKERS] Pluggable storage

On Fri, Feb 16, 2018 at 9:56 PM, Alexander Korotkov <
a(dot)korotkov(at)postgrespro(dot)ru> wrote:

> Hi, Haribabu!
>
> On Mon, Feb 5, 2018 at 2:22 PM, Haribabu Kommi <kommi(dot)haribabu(at)gmail(dot)com>
> wrote:
>
>>
>> To integrate the columnar store with the pluggable storage API, I found
>> that
>> there are couple of other things also that needs to be supported.
>>
>> 1. Choosing the right table access method for a particular table?
>>
>> I am thinking of adding a new table option to let the user select the
>> correct table
>> access method that the user wants for the table. HEAP is the default
>> access
>> method. This approach may be simple and doesn't need any syntax changes.
>>
>> Or Is it fine to add syntax "USING method" to CREATE TABLE similar like
>> CREATE INDEX?
>>
>> comments?
>>
>
> Sure, user needs to specify particular table access method when creating a
> table. "USING method" is good for me. I think it's better to reuse
> already existing syntax rather than reinventing something new.
>

Yes, I also felt the same. Updated patches included this implementation.

>
>
>> 2. As the columnar storage needs many other relations that are needs to be
>> created along with main relation.
>>
>
> That's an interesting point. During experimenting on some new index
> access methods I also have to define some kind of "internal" relations
> invisible for user, but essential for main relation functionality. I've
> made them in hackery manner, and I think legal mechanism for them would be
> very good.
>
>
>> As these extra relations are internal to the storage and shouldn't be
>> visible
>> directly from pg_class and these will be stored in the storage specific
>> catalog tables. A dependency is created for the original table as these
>> storage
>> specific tables must be created/dropped/altered whenever there is a change
>> with the original table.
>>
>
> I think that internal relations should be visible from pg_class, otherwise
> it wouldn't be possible to define dependencies over them. But they should
> have different relkind, so they wouldn't be messed up with regular
> relations.
>

OK.

> Is it fine to add new API while creating/altering/drop the table to get
>> the control?
>> or to use only exiting processutility hook?
>>
>
> I believe that we need some generic way for defining internal relations,
> not hooks. For me it seems like useful feature for further development of
> both index access methods and table access methods.
>

How about a new relkind of RELKIND_EXTERNAL ('e') or something similar
types and these can be created internally by the extensions that needs
extra relations apart
from main relation. These type of relations can have storage, but these
cannot be selected directly by the user using SQL commands.

I will try the above approach.

> During developer meeting [1], I've proposed to reorder patches so that
> refactoring patches go first and API introduction patches go afterwards.
> That would make possible to commit some of refactoring patches earlier
> without necessary committing API in the same release cycle. If no
> objection, I would do that reordering.
>

Yes, that will be easy to review the API patches once the refactoring
patches went in.

> BTW, EnterpriseDB announces zheap table access method (heap with undo log)
> [2]. I think this is great, and I'm looking forward for publishing zheap
> in mailing lists. But I'm concerning about its compatibility with
> pluggable table access methods API. Does zheap use table AM API from this
> thread? Or does it just override current heap and needs to be adopted to
> use table AM API? Or does it implements own API?
>

I am also not sure about on which API the zheap is implemented. But it will
be good if we all come up with table AM API's that works for all the
external storage methods.

Latest rebased patch series are attached.

Regards,
Hari Babu
Fujitsu Australia