Re: Make COPY format extendable: Extract COPY TO format implementations

On Tue, Feb 4, 2025 at 6:19 PM Vladlen Popolitov
<v(dot)popolitov(at)postgrespro(dot)ru> wrote:
>
> Masahiko Sawada писал(а) 2025-02-05 08:32:
> > On Tue, Feb 4, 2025 at 2:46 AM Vladlen Popolitov
>
> >> >> Standard PostgreSQL realisation for new methods to use USING
> >> >> keyword. Every
> >> >> new method could have own options (FORMAT is option of internal 'copy
> >> >> from/to'
> >> >> methods),
> >> >
> >> > Ah, I didn't think about USING.
> >> >
> >> > You suggest "COPY ... USING json" not "COPY ... FORMAT json"
> >> > like "CREATE INDEX ... USING custom_index", right? It will
> >> > work. If we use this interface, we should reject "COPY
> >> > ... FORMAT ... USING" (both of FORMAT/USING are specified).
> >> >
> >> >
> >> I cannot recommend about rejecting, I do not know details
> >> of realisation of this part of code. Just idea - FORMAT value
> >> could be additional option to copy handler or NULL
> >> if it is omitted.
> >> If you add extensibility, than every handler will be the
> >> extension, that can handle one or more formats.
> >
> > Hmm, if we use the USING clause to specify the format type, we end up
> > having two ways to specify the format type (e.g., 'COPY ... USING
> > text' and 'COPY .. WITH (format = text)'), which seems to confuse
> > users.
> WITH clause has list of options defined by copy method define in USING.
> The clause WITH (format=text) has options defined for default copy
> method,
> but other methods will define own options. Probably they do not need
> the word 'format' in options. The same as in index access methods.
> For example, copy method parquete:
> COPY ... USING parquete WITH (row_group_size=1000000)
> copy method parquete need and will define the word 'row_group_size'
> in options, the word 'format' will be wrong for it.

I think it's orthological between the syntax and options passed to the
custom format extension. For example, even if we specify the options
like "COPY ... WITH (format 'parquet', row_group_size '1000000',
on_error 'ignore)", we can pass only non-built-in options (i.e. only
row_group_size) to the extension.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com