Re: Make COPY format extendable: Extract COPY TO format implementations

Hi Kou-san,

On Fri, Jan 26, 2024 at 5:02 PM Junwang Zhao <zhjwpku(at)gmail(dot)com> wrote:
>
> On Fri, Jan 26, 2024 at 4:55 PM Sutou Kouhei <kou(at)clear-code(dot)com> wrote:
> >
> > Hi,
> >
> > In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ(at)mail(dot)gmail(dot)com>
> > "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
> > Junwang Zhao <zhjwpku(at)gmail(dot)com> wrote:
> >
> > > CopyToProcessOption()/CopyFromProcessOption() can only handle
> > > single option, and store the options in the opaque field, but it can not
> > > check the relation of two options, for example, considering json format,
> > > the `header` option can not be handled by these two functions.
> > >
> > > I want to find a way when the user specifies the header option, customer
> > > handler can error out.
> >
> > Ah, you want to use a built-in option (such as "header")
> > value from a custom handler, right? Hmm, it may be better
> > that we call CopyToProcessOption()/CopyFromProcessOption()
> > for all options including built-in options.
> >
> Hmm, still I don't think it can handle all cases, since we don't know
> the sequence of the options, we need all the options been parsed
> before we check the compatibility of the options, or customer
> handlers will need complicated logic to resolve that, which might
> lead to ugly code :(
>

I have been working on a *COPY TO JSON* extension since yesterday,
which is based on your V6 patch set, I'd like to give you more input
so you can make better decisions about the implementation(with only
pg-copy-arrow you might not get everything considered).

V8 is based on V6, so anybody involved in the performance issue
should still review the V7 patch set.

0001-0008 is your original V6 implementations

0009 is some changes made by me, I changed CopyToGetFormat to
CopyToSendCopyBegin because pg_copy_json need to send different bytes
in SendCopyBegin, get the format code along is not enough, I once had
a thought that may be we should merge SendCopyBegin/SendCopyEnd into
CopyToStart/CopyToEnd but I don't do that in this patch. I have also
exported more APIs for extension usage.

00010 is the pg_copy_json extension, I think this should be a good
case which can utilize the *extendable copy format* feature, maybe we
should delete copy_test_format if we have this extension as an
example?

> >
> > Thanks,
> > --
> > kou
>
>
>
> --
> Regards
> Junwang Zhao

--
Regards
Junwang Zhao