Re: Parallel copy

On Tue, Feb 18, 2020 at 5:59 PM Ants Aasma <ants(at)cybertec(dot)at> wrote:
>
> On Tue, 18 Feb 2020 at 12:20, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > This is something similar to what I had also in mind for this idea. I
> > had thought of handing over complete chunk (64K or whatever we
> > decide). The one thing that slightly bothers me is that we will add
> > some additional overhead of copying to and from shared memory which
> > was earlier from local process memory. And, the tokenization (finding
> > line boundaries) would be serial. I think that tokenization should be
> > a small part of the overall work we do during the copy operation, but
> > will do some measurements to ascertain the same.
>
> I don't think any extra copying is needed.
>

I am talking about access to shared memory instead of the process
local memory. I understand that an extra copy won't be required.

> The reader can directly
> fread()/pq_copymsgbytes() into shared memory, and the workers can run
> CopyReadLineText() inner loop directly off of the buffer in shared memory.
>

I am slightly confused here. AFAIU, the for(;;) loop in
CopyReadLineText is about finding the line endings which we thought
that the reader process will do.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com