Re: WIP Incremental JSON Parser

On 2024-03-25 Mo 19:02, Andrew Dunstan wrote:
>
>
> On Mon, Mar 25, 2024 at 6:15 PM Jacob Champion
> <jacob(dot)champion(at)enterprisedb(dot)com> wrote:
>
> On Wed, Mar 20, 2024 at 11:56 PM Andrew Dunstan
> <andrew(at)dunslane(dot)net> wrote:
> > Thanks, included that and attended to the other issues we
> discussed. I think this is pretty close now.
>
> Okay, looking over the thread, there are the following open items:
> - extend the incremental test in order to exercise the semantic
> callbacks [1]
>
>
>
> Yeah, I'm on a super long plane trip later this week, so I might get
> it done then :-)
>
> - add Assert calls in impossible error cases [2]
>
>
> ok, will do
>
> - error out if the non-incremental lex doesn't consume the entire
> token [2]
>
>
> ok, will do
>
> - double-check that out of memory is an appropriate failure mode for
> the frontend [3]
>
>
>
> Well, what's the alternative? The current parser doesn't check stack
> depth in frontend code. Presumably it too will eventually just run out
> of memory, possibly rather sooner as the stack frames could  be more
> expensive than the incremental parser stack extensions.
>
>
> Just as a general style nit:
>
> > +   if (lex->incremental)
> > +   {
> > +       lex->input = lex->token_terminator = lex->line_start = json;
> > +       lex->input_length = len;
> > +       lex->inc_state->is_last_chunk = is_last;
> > +   }
> > +   else
> > +       return JSON_INVALID_LEXER_TYPE;
>
> I think flipping this around would probably make it more readable;
> something like:
>
>     if (!lex->incremental)
>         return JSON_INVALID_LEXER_TYPE;
>
>     lex->input = ...
>
>
>
> Noted. will do, Thanks.
>
>

Here's a new set of patches, with I think everything except the error
case Asserts attended to. There's a change to add semantic processing to
the test suite in patch 4, but I'd fold that into patch 1 when committing.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com