Re: Parallel copy

At Tue, 18 Feb 2020 15:59:36 +0530, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote in
> On Tue, Feb 18, 2020 at 7:28 AM Kyotaro Horiguchi
> <horikyota(dot)ntt(at)gmail(dot)com> wrote:
> >
> > In an extreme case, if we didn't see a QUOTE in a chunk, we cannot
> > know the chunk is in a quoted section or not, until all the past
> > chunks are parsed. After all we are forced to parse fully
> > sequentially as far as we allow QUOTE.
> >
>
> Right, I think the benefits of this as compared to single reader idea
> would be (a) we can save accessing shared memory for the most part of
> the chunk (b) for non-csv mode, even the tokenization (finding line
> boundaries) would also be parallel. OTOH, doing processing
> differently for csv and non-csv mode might not be good.

Agreed. So I think it's a good point of compromize.

> > On the other hand, if we allowed "COPY t FROM f WITH (FORMAT CSV,
> > QUOTE '')" in order to signal that there's no quoted section in the
> > file then all chunks would be fully concurrently parsable.
> >
>
> Yeah, if we can provide such an option, we can probably make parallel
> csv processing equivalent to non-csv. However, users might not like
> this as I think in some cases it won't be easier for them to tell
> whether the file has quoted fields or not. I am not very sure of this
> point.

I'm not sure how large portion of the usage contains quoted sections,
so I'm not sure how it is useful..

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center