More thoughts about FE/BE protocol

I've been thinking some more about the FE/BE protocol redesign,
specifically the desire to ensure that we can recover from error
conditions without losing synchronization. The fact that the existing
protocol doesn't do very well at this shows up in several places:
* having to drop and restart the connection after a COPY error
* fastpath function calls also lose sync if there's an error
* libpq gets terribly confused if it runs out of memory for
a query result

I'm coming around to the idea that the cleanest solution is to require
*all* protocol messages, in both directions, to have an initial length
word. That is, the general message format would look like
<message type> 1 byte
<payload length> number of following bytes (4 bytes MSB-first)
... message data as needed ...

The advantage of doing it this way is that the recipient can absorb the
whole message before starting to parse the contents; then, errors detected
while processing the message contents don't cause us to lose protocol
synchronization. Also, even if the message is so large as to run the
recipient out of memory, it can still use the <payload length> to count
off the number of bytes it has to drop before looking for another message.
This would make it quite a bit easier for libpq to cope with
out-of-memory, as an example.

These advantages seem enough to me to justify adding three or four bytes
to the length of each message. Does anyone have a problem with that?

A related point is that I had been thinking of the new "extended query"
facility (separate PARSE/BIND/DESCRIBE/EXECUTE steps) in terms of
sending just one big message to the backend per interaction cycle, with
the processing-step commands appearing as fields within that message.
But putting a length word in front would effectively require the
frontend to marshal the whole sequence before sending any of it.
It seems better to send each of the processing-step commands as an
independent message. To do that, we need to introduce an additional
processing step, call it SYNC, that substitutes for the functionality
associated with the overall message boundary in the other way.
Specifically:
* ReadyForQuery (Z) is sent in response to SYNC.
* If no BEGIN has been issued, SYNC is the point at which an implicit
COMMIT is done.
* After an error occurs in an extended-query command, the backend reads
and discards messages until it finds a SYNC, then issues ReadyForQuery
and resumes processing messages. This allows the frontend to be
certain what has been processed and what hasn't.

Comments?

regards, tom lane