Re: An idea for parallelizing COPY within one backend

Brian Hurt wrote:
> Tom Lane wrote:
>> "Florian G. Pflug" <fgp(at)phlo(dot)org> writes:
>>> ...
>>> Neither the "dealer", nor the "workers" would need access to the either
>>> the shared memory or the disk, thereby not messing with the "one backend
>>> is one transaction is one session" dogma.
>>> ...
>> Unfortunately, this idea has far too narrow a view of what a datatype
>> input function might do. Just for starters, consider "enum" input,
>> which certainly requires catalog access. We have also explicitly
>> acknowledged the idea that datatype I/O functions might try to store
>> typmod-related data in some special catalog somewhere.

> Would it be possible to determine when the copy is starting that this
> case holds, and not use the parallel parsing idea in those cases?

In theory, yes. In pratice, I don't want to be the one who has to answer
to an angry user who just suffered a major drop in COPY performance
after adding an ENUM column to his table.

I was thinking more along the line of letting a datatype specify a
function "void* ioprepare(typmod)" which returns some opaque object
specifying all that the input and output function needs to know.
We could than establish the rule that input/output functions may not
access the catalog, and instead pass them a pointer to that opaque object.

All pretty pie-in-the-sky at the moment, though...

regards, Florian Pflug