Re: big text field -> message type 0x44

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Tomas Berndtsson <tomas(at)nocrew(dot)org> writes:
> > Yep, there is only one installation of PostgreSQL on the machine. My
> > application is multithreaded, and I have been very careful to open a
> > new connection for each thread. Could it have anything to do with
> > semaphores and shared memory in Solaris?
>
> I wouldn't think so; the client-side code doesn't have anything to do
> with either shared memory or semaphores. But your comment about
> threading immediately focuses my attention on that.
>
> Let's see (checks ASCII codes...) message 0x44 is 'D' which is a data
> message. The only situations I've seen before in which libpq comes out
> with this complaint are (1) when it's lost sync with the backend as a
> result of running out of memory to store a large query result (its
> recovery from that situation is pretty crummy :-(), or (2) when
> someone's confused libpq by trying concurrent queries with one PGconn.
>
> You say you didn't do (2), so that leaves (1). Is it possible that your
> threading setup limits the amount of memory libpq can malloc?

I don't know what I would do to limit it. The machine has 2GB RAM, and
over 1GB free.

However, after some semi-random looking through the source code of
libpq, I tried to change a value, namely here:

fe-misc.c row 510 in pqReadData():

if (conn->inEnd > 32768 &&
(conn->inBufSize - conn->inEnd) >= 8192)
{
someread = 1;
goto tryAgain;
}

I changed the 32768 value to 131072, and sure enough, my application
was able to get larger fields without any errors. The best thing would
of course be to have no limit to it. That would mean taking the whole
if-statement out, right? I've only tried with the value change,
though. There's a comment above this, saying it's a hack for some
kernels that only give back one packet, even if there is more. But, it
seems to confuse the Solaris kernel in some mysterious way when
running threads.

I haven't seen that it breaks anything else by changing this value,
but if you think it might, please tell me. I wouldn't want to risk
breaking other stuff.

Tomas