Re: literal limits in 8.3

Sam Mason <sam(at)samason(dot)me(dot)uk> writes:
> On Mon, Nov 24, 2008 at 10:45:42AM -0500, Tom Lane wrote:
>> Well, I can't reproduce that here. Something strange about your
>> configuration maybe?

> Not that I know of. I've just created a test cluster to make sure and I
> get the same behaviour.

Hmm ... the third machine I tried was able to reproduce the problem.

What it boils down to is lack of error checking in psql (not the
backend). Specifically, we fail to enlarge the output buffer for
psqlscan.l, which causes appendBinaryPQExpBuffer to silently not insert
the chunk it's currently being passed. Which you might think would be
some random subset of the input string, leading to a parse error on
the backend side --- but no, this is the output of a lexical scan which
means what is dropped is exactly the contents of the multi-megabyte
string literal, not less or more. And then later insertions work fine
since *they* aren't provoking an out-of-memory problem. So eventually
the backend receives
INSERT INTO test (col) VALUES ('');
which of course it finds nothing wrong with.

This is sort of a PITA to fix :-(. The easiest solution from the point
of view of psql would be to have realloc failure just print "out of
memory" and exit(1), but pqexpbuffer.c is part of libpq and so it's not
too reasonable for it to do that. And we have also got to think about
the prospect of similarly non-random lossage in other uses of
PQexpbuffer, anyhow.

The least API-damaging solution I can think of is to add an error
indicator flag to PQexpbuffer, comparable to ferror() on stdio files.
Callers would have to check this after loading up a PQexpbuffer if
they wanted to be sure there was no memory overrun. But that seems
pretty fragile, and it wouldn't be back-patchable.

A variant on that is to clear the buffer and insert "out of memory"
in this scenario, but that's not too pleasant either.

Better ideas anyone?

regards, tom lane