Flushing large data immediately in pqcomm

Hi hackers

I've been looking into ways to reduce the overhead we're having in pqcomm
and I'd like to propose a small patch to modify how socket_putmessage works.

Currently socket_putmessage copies any input data into the pqcomm send
buffer (PqSendBuffer) and the size of this buffer is 8K. When the send
buffer gets full, it's flushed and we continue to copy more data into the
send buffer until we have no data left to be sent.
Since the send buffer is flushed whenever it's full, I think we are safe to
say that if the size of input data is larger than the buffer size, which is
8K, then the buffer will be flushed at least once (or more, depends on the
input size) to store and all the input data.

Proposed change modifies socket_putmessage to send any data larger than
8K immediately without copying it into the send buffer. Assuming that the
send buffer would be flushed anyway due to reaching its limit, the patch
just gets rid of the copy part which seems unnecessary and sends data
without waiting.

This change affects places where pq_putmessage is used such as
pg_basebackup, COPY TO, walsender etc.

I did some experiments to see how the patch performs.
Firstly, I loaded ~5GB data into a table [1], then ran "COPY test TO
STDOUT". Here are perf results of both the patch and HEAD

HEAD:
- 94,13% 0,22% postgres postgres [.] DoCopyTo
- 93,90% DoCopyTo
- 91,80% CopyOneRowTo
+ 47,35% CopyAttributeOutText
- 26,49% CopySendEndOfRow
- 25,97% socket_putmessage
- internal_putbytes
- 24,38% internal_flush
+ secure_write
+ 1,47% memcpy (inlined)
+ 14,69% FunctionCall1Coll
+ 1,94% appendBinaryStringInfo
+ 0,75% MemoryContextResetOnly
+ 1,54% table_scan_getnextslot (inlined)

Patch:
- 94,40% 0,30% postgres postgres [.] DoCopyTo
- 94,11% DoCopyTo
- 92,41% CopyOneRowTo
+ 51,20% CopyAttributeOutText
- 20,87% CopySendEndOfRow
- 20,45% socket_putmessage
- internal_putbytes
- 18,50% internal_flush (inlined)
internal_flush_buffer
+ secure_write
+ 1,61% memcpy (inlined)
+ 17,36% FunctionCall1Coll
+ 1,33% appendBinaryStringInfo
+ 0,93% MemoryContextResetOnly
+ 1,36% table_scan_getnextslot (inlined)

The patch brings a ~5% gain in socket_putmessage.

Also timed the pg_basebackup like:
time pg_basebackup -p 5432 -U replica_user -X none -c fast --no_maanifest
-D test

HEAD:
real 0m10,040s
user 0m0,768s
sys 0m7,758s

Patch:
real 0m8,882s
user 0m0,699s
sys 0m6,980s

It seems ~11% faster in this specific case.

I'd appreciate any feedback/thoughts.

[1]
CREATE TABLE test(id int, name text, time TIMESTAMP);
INSERT INTO test (id, name, time) SELECT i AS id, repeat('dummy', 100) AS
name, NOW() AS time FROM generate_series(1, 100000000) AS i;

Thanks,
--
Melih Mutlu
Microsoft