Re: [HACKERS] LIBPQ patches ...

* Alfred Perlstein <bright(at)wintelcom(dot)net> [000108 16:08] wrote:
> * Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> [000108 14:56] wrote:
>
> > The
> > insertion of a pqFlush into PQconsumeInput, in particular, looks like
> > an ill-thought-out hack that could break some applications.
>
> I think I agree, the code I was using would attempt an PQconsumeInput()
> before doing a PQendcopy(), there could be data in the send buffer
> that would make PQconsumeInput() never succeed hence the need for a
> flush.
>
> I'm going to try it without the PQconsumeInput() before the PQendcopy()
> my modifications for PQendcopy() should make it non-blocking safe.
> but in the meanwhile here's my (probably wrong) reasoning behind this
> 'hack':
>
> No, IMHO it's needed, the problem is that there may be data
> in the send buffer that hasn't been sent yet, it could be
> part of a request to the backend that you are explicitly
> waiting for a result from.
>
> This can happen when doing a COPY into the database.
>
> What happens is that you send data, then when you send the
> 'end copy' it can get buffered, then you loop forever
> attempting to read a result for a query that was never
> sent.
>
> In regards to it breaking applications, the send buffer
> should be opaque to the libpq user, libpq never has offered
> a truly non-blocking api, and even when using non-blocking
> the flush will fail if it can't be done and PQconsumeInput()
> will error out accordingly.
>
> Old applications can be snagged by the Flush since in theory
> PQconsumeInput shouldn't block, however I'm not sure if
> there's a real workaround for this except
>
> 1.. saving the blocking status of the connection,
> 2.. setting it non-blocking and attempting a flush and then
> 3.. restoring the blocking status.
>
> It seems that old applications can break (looping on an
> unsent line) regardless because of the not-flushed-query
> problem.
>
> If you can figure an occasion where this might actually happen
> (with the exception of my accidentaly abuse of libpq) then it
> may need to be revisited.
>
> I'll get back to you guys on the PQendcopy before PQconsumeInput
> tests.

I just remebered where the problem is (sorry it's been about 2 weeks
since i've read through the code) it's a bit different
and messier than I thought:

I begin my COPY commands with a PQsendQuery() which has this block
of code in it:

/* send the query to the backend; */
/* the frontend-backend protocol uses 'Q' to designate queries */
if (pqPutnchar("Q", 1, conn) ||
pqPuts(query, conn) ||
pqFlush(conn))
{
handleSendFailure(conn);
return 0;
}

It can get really hairy for non-blocking connections if any of the
functions in the 'if' conditional fail, any ideas on a workaround?

One that comes to mind is using the low/high watermarks in sockets,
if we do that then a write-ready true condition would garantee that
we have X number of bytes available in our send buffer and we can
safely queue the data. This doesn't seem portable and would be
pretty complex.

Another is to attempt a flush beforehand aborting early if it fails
however we still need to flush after we pqPutnchar and pqPuts,
if that fails we are back to needing to call pqFlush from PQconsumeInput
beccause the only failure is that the backend's pipe is potentially full
and we may have queued something.

This 'hack' would be to allow the last flush to fail and always call
pqFlush in PQconsumeInput when the connection is non-blocking because
with the new code it shouldn't block, PQconsumeInput will function to
drive data to the backend in that situation. We'll only do the
flush in PQconsumeInput() for non-blocking connections because
the conditional in PQsendQuery shouldn't fail for blocking connections
unless something is seriously wrong.

I hate to say it, but I like the hack approach, it is somewhat wierd
but looks like it would work quite well as any error returned from
the backend would be delayed until PQconsumeInput and a broken connection
would still be returned immediatly from PQsendQuery.

Your opinion?

--
-Alfred Perlstein - [bright(at)rush(dot)net|alfred(at)freebsd(dot)org]
Wintelcom systems administrator and programmer
- http://www.wintelcom.net/ [bright(at)wintelcom(dot)net]