Re: Libpq COPY optimization

"Alon Goldshuv" <agoldshuv(at)greenplum(dot)com> writes:
> Please help me understand this better. It appears to me that when the
> client->backend pipe fills up, pqSendSome() consumes any incoming
> NOTICE/WARNING messages before waiting, which should prevent deadlock.

Hm, I had forgotten that the low-level pqSendSome routine does that.
That makes the PQconsumeInput call in PQputCopyData redundant (or
almost; see below). The parseInput call is still needed, because it's
there to pull NOTICE messages out of the input buffer and get rid of
them, rather than possibly having the input buffer grow to exceed
memory. But when there's nothing for it to do, parseInput is cheap
enough that there's no real need to bypass it.

In short, if you just remove the PQconsumeInput call I think you'll find
that it does what you want.

The only case where it's helpful to have it there is if there's a
incomplete message in the input buffer, as parseInput isn't quite so
fast if it has to determine that the message is incomplete. Without
the PQconsumeInput call, the incomplete-message state could persist
for a long time, and you'd pay the parseInput overhead each time through
PQputCopyData. However, that's certainly not the normal situation,
so I think we could leave that case slightly pessimal. It's certainly
true that that path in parseInput is a lot faster than a kernel call,
so it'd still be better than it is now.

regards, tom lane