An idea for parallelizing COPY within one backend

From: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>
To: Postgresql-Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: An idea for parallelizing COPY within one backend
Date: 2008-02-27 02:43:11
Message-ID: 47C4CE3F.7090801@phlo.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

As far as I can see the main difficulty in making COPY run faster (on
the server) is that pretty involved conversion from plain-text lines
into tuples. Trying to get rid of this conversion by having the client
send something that resembles the data stored in on-disk tuples is not a
good answer, either, because it ties the client too closely to
backend-version specific implementation details.

But those problems only arise if the *client* needs to deal with the
binary format. What I envision is parallelizing that conversion step on
the server, controlled by a backend process, kind of like a filter
between the server and the client.

Upon reception of a COPY INTO command, a backend would
.) Retrieve all catalog information required to convert a plain-text
line into a tuple
.) Fork off a "dealer" and N "worker" processes that take over the
client connection. The "dealer" distributes lines received from the
client to the N workes, while the original backend receives them
as tuples back from the workers.

Neither the "dealer", nor the "workers" would need access to the either
the shared memory or the disk, thereby not messing with the "one backend
is one transaction is one session" dogma.

Now I'm eagerly waiting to hear all the reasons why this idea is broken
as hell ;-)
regards, Florian Pflug

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Joshua D. Drake 2008-02-27 03:29:55 Re: One more option for pg_dump...
Previous Message Robert Lor 2008-02-27 02:35:55 Re: Proposed changes to DTrace probe implementation