Re: Custom Scan APIs (Re: Custom Plan node)

2014-01-29 Christian Convey <christian(dot)convey(at)gmail(dot)com>:
>
> On Mon, Jan 27, 2014 at 7:14 PM, Kouhei Kaigai <kaigai(at)ak(dot)jp(dot)nec(dot)com> wrote:
>>
>> FDW's join pushing down is one of the valuable use-cases of this
>> interface,
>> but not all. As you might know, my motivation is to implement GPU
>> acceleration
>> feature on top of this interface, that offers alternative way to scan or
>> join
>> relations or potentially sort or aggregate.
>
>
> I'm curious how this relates to the pluggable storage idea discussed here
> https://wiki.postgresql.org/wiki/PgCon_2013_Developer_Meeting and here
> http://www.databasesoup.com/2013/05/postgresql-new-development-priorities-2.html
>
> I haven't seen a specific proposal about how much functionality should be
> encapsulated by a pluggable storage system. But I wonder if that would be
> the best place for specialized table-scan code to end up?
>
If you are interested in designing your own storage layer (thus it needs to
have own scan/writer implementation), FDW is an option currently available.
It defines a set of interface that allows extensions to generate "things to be
there" on the fly. It does not force to perform as a client of remote database,
even though it was originally designed for dblink purpose.
In other words, FDW is a feature to translate a particular data source into
something visible according to the table definition. As long as driver can
intermediate table definition and data format of your own storage layer,
it shall work.

On the other hands, custom-scan interface, basically, allows extensions to
implement "alternative way to access" the data. If we have wiser way to
scan or join relations than built-in logic (yep, it will be a wiser
logic to scan
a result set of remote-join than local join on a couple of remote scan results),
this interface suggest the backend "I have such a wise strategy", then planner
will choose one of them; including either built-in or additional one, according
to the cost.

Let's back to your question. This interface is, right now, not designed to
implement pluggable storage layer. FDW is an option now, and maybe
a development item in v9.5 cycle if we want regular tables being pluggable.
Because I'm motivated to implement my GPU acceleration feature to
perform on regular relations, I put my higher priority on the interface to
allow extension to suggest "how to scan" it.

Thanks,
--
KaiGai Kohei <kaigai(at)kaigai(dot)gr(dot)jp>