Re: Custom Scans and private data

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Wed, Aug 26, 2015 at 11:23 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
>> While that comment made me laugh, I'm not really sure why my examples
>> are bait. One is the probably most used fdw, the other the only user of
>> the custom scan interface I know of.

> I suspect what Tom is getting at is that he thinks the custom scan
> interface is ...

[ROTFL] No, actually, I'm not interested in fighting that battle.
But I do think that letting custom-scan extensions fool about with
the semantics of plan-copying (and plan-serialization for that matter)
is a risky choice that is not justified by some marginal gains in code
readability, especially when there are other feasible ways to attack
the readability problem. copyObject needs to be a pretty self-contained
operation with not a lot of external dependencies, and I'm afraid there
would be too much temptation for extensions to take shortcuts that would
fail later.

> [ ruminations about how to improve the system's extensibility ]

Yeah, I've entertained similar thoughts in the past. It's not very clear
how we get there from here, though --- for instance, any fundamental
change in the Node system would break so much code that it's unlikely
anyone would thank us for it, even assuming we ever finished the
conversion project.

I'm also less than convinced that "I should be able to install major new
features without touching the core code" is actually a sane or useful goal
to have. It sounds great in the abstract, but long ago (at HP) I worked
on systems that were actually built to be able to do that ... and I can
tell you that the tradeoffs you have to make to do that are not very
pleasant.

As a concrete example, you mentioned the lack of extensibility of the
bison parser, which is undoubtedly a blocking issue if you want a new
feature that would require new SQL syntax. While bison is certainly
a PITA in some ways, it has one saving grace that I think a lot of people
underappreciate: if it compiles your grammar, the grammar is (probably)
unambiguous and has no unreachable elements. The extensible parser we
had at HP was, um, considerably less capable of detecting conflicts or
ambiguity. Maybe we just didn't know how to do it, but I suspect that
that is a fundamental tradeoff when you have incomplete information.

I don't want to sound too negative --- there are probably improvements
we can make in this area. But I'm skeptical of sticking in hooks that
haven't been pretty carefully thought through. We've done a lot of that
in the past, and I think the net result is that we have a lot of useless
hooks, or at least hooks that can't be used in a way that we don't break
every release or two.

regards, tom lane