Re: scanner/parser minimization

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Thu, Feb 28, 2013 at 4:09 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I believe however that it's possible to extract an idea of which
>> tokens the parser believes it can see next at any given parse state.
>> (I've seen code for this somewhere on the net, but am too lazy to go
>> searching for it again right now.) So we could imagine a rule along
>> the lines of "if IDENT is allowed as a next token, and $KEYWORD is
>> not, then return IDENT not the keyword's own token".
>>
>> This might be unworkable from a speed standpoint, depending on how
>> expensive it is to make the determination about allowable next symbols.
>> But it seems worth looking into.

> Interesting idea. But wouldn't that change the semantics of the
> grammar in some places? In particular, keywords would generally
> become less-reserved than they are now, but in a context-dependent
> way.

Yeah, Greg already raised that point. It shouldn't change the semantics
of the grammar *today*, but it would become much harder to tell whether
future changes create any conflicts; we'd lose the mechanical
verification we get from bison with the current method. That might be
sufficient reason not to pursue the idea.

It's possible that static analysis of the grammar could replace bison
verification; that is, assuming we have a way to identify all states in
which both IDENT and some-keyword are allowable, we could generate a
report listing which keywords are potentially going to be considered
non-reserved, and then watch for changes in that report. But this would
require writing even more code that we don't have today.

I suspect also that we'd still need a notion of fully reserved keywords
that don't get flipped to IDENT even if that's a legal next symbol.

regards, tom lane