Re: More thoughts about FE/BE protocol

tgl(at)sss(dot)pgh(dot)pa(dot)us wrote:
> I've been thinking some more about the FE/BE protocol redesign,
> specifically the desire to ensure that we can recover from error
> conditions without losing synchronization. The fact that the existing
> protocol doesn't do very well at this shows up in several places:
> * having to drop and restart the connection after a COPY error
> * fastpath function calls also lose sync if there's an error
> * libpq gets terribly confused if it runs out of memory for
> a query result
>
> I'm coming around to the idea that the cleanest solution is to require
> *all* protocol messages, in both directions, to have an initial length
> word. That is, the general message format would look like
> <message type> 1 byte
> <payload length> number of following bytes (4 bytes MSB-first)
> ... message data as needed ...
>
> The advantage of doing it this way is that the recipient can absorb the
> whole message before starting to parse the contents; then, errors detected
> while processing the message contents don't cause us to lose protocol
> synchronization. Also, even if the message is so large as to run the
> recipient out of memory, it can still use the <payload length> to count
> off the number of bytes it has to drop before looking for another message.
> This would make it quite a bit easier for libpq to cope with
> out-of-memory, as an example.
>
> These advantages seem enough to me to justify adding three or four bytes
> to the length of each message. Does anyone have a problem with that?
> ...

Well, just to be a bit contrary, I suggest that the loss of synch problem
seems to not be caused by lack of an overall message length word.

In the case of fastpath function calls, the length of each parameter is
there now, so the backend could already read all the data before doing
any error checking. The comment in tcop/fastpath.c:HandleFunctionRequest()
says this, but then it loses me when it goes on to say this is impossible
because the message can't be read until doing type lookups. I don't
understand why, but if this is true, how will an overall length word help?
You either read the whole thing into memory before error checking, or you don't.

In the case of COPY, what would your overall length word apply to, since
the copy data is stream oriented, not message oriented? I don't understand
backend error handling, but if the "copy" function loses control when an
error occurs (for example, bad data type for a column), I don't see how
knowing the overall message or data length helps in this case.