question regarding copyData containers

Hello,

I have been working on a node.js streaming client for different COPY
scenarios.
usually, during CopyOut, clients tend to buffer network chunks until they
have gathered a full copyData message and pass that to the user.

In some cases, this can lead to very large copyData messages. when there
are very long text fields or bytea fields it will require a lot of memory
to be handled (up to 1GB I think in the worst case scenario)

In COPY TO, I managed to relax that requirement, considering that copyData
is simply a transparent container. For each network chunk, the relevent
message content is forwarded which makes for 64KB chunks at most.

If that makes things clearer, here is an example scenarios, with 4 network
chunks received and the way they are forwarded to the client.

in: CopyData Int32Len Byten1
in: Byten2
in: Byten3
in: CopyData Int32Len Byten4

out: Byten1
out: Byten2
out: Byten3
out: Byten4

We loose the semantics of the "row" that copyData has according to the
documentation
https://www.postgresql.org/docs/10/protocol-flow.html#PROTOCOL-COPY
>The backend sends a CopyOutResponse message to the frontend, followed by
zero or more >CopyData messages (**always one per row**), followed by
CopyDone

but it is not a problem because the raw bytes are still parsable (rows +
fields) in text mode (tsv) and in binary mode)

Now I started working on copyBoth and logical decoding scenarios. In this
case, the server send series of copyData. 1 copyData containing 1 message :

at the network chunk level, in the case of large fields, we can observe

in: CopyData Int32 XLogData Int64 Int64 Int64 Byten1
in: Byten2
in: CopyData Int32 XLogData Int64 Int64 Int64 Byten3
in: CopyData Int32 XLogData Int64 Int64 Int64 Byten4

out: XLogData Int64 Int64 Int64 Byten1
out: Byten2
out: XLogData Int64 Int64 Int64 Byten3
out: XLogData Int64 Int64 Int64 Byten4

but at the XLogData level, the protocol is not self-describing its length,
so there is no real way of knowing where the first XLogData ends apart from
- knowing the length of the first copyData (4 + 1 + 3*8 + n1 + n2)
- knowing the internals of the output plugin and benefit from a plugin
that self-describe its span

when a network chunks contains several copyDatas
in: CopyData Int32 XLogData Int64 Int64 Int64 Byten1 CopyData Int32
XLogData Int64 Int64 Int64 Byten2
we have
out: XLogData Int64 Int64 Int64 Byten1 XLogData Int64 Int64 Int64 Byten2

and with test_decoding for example it is impossible to know where the
test_decoding output ends without remembering the original length of the
copyData.

now my question is the following :
is it ok to consider that over the long term copyData is simply a transport
container that exists only to allow the multiplexing of events in the
protocol but that messages inside could be chunked over several copyData
events ?

if we put test_decoding apart, do you consider that output plugins XLogData
should be self-aware of their length ? I suppose (but did not fully verify
yet) that this is the case for pgoutput ? I suppose that wal2json could
also be parsed by balancing the brackets.

I am wondering because when a client sends copyData to the server, the
documentation says
>The message boundaries are not required to have anything to do with row
boundaries, >although that is often a reasonable choice.

I hope that my message will ring a bell on the list.
I tried the best I could to describe my very specific research.
Thank you for your help,
---
Jérôme