multi-threaded pgloader needs your tests

From: Dimitri Fontaine <dfontaine(at)hi-media(dot)com>
To: pgsql-performance(at)postgresql(dot)org
Subject: multi-threaded pgloader needs your tests
Date: 2008-02-26 12:08:53
Message-ID: 200802261308.53459.dfontaine@hi-media.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-performance

Hi,

You may remember some thread about data loading performances and
multi-threading support in pgloader:
http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php

The pgloader code to handle this is now ready to get tested, a more structured
project could talk about a Release Candidate status.
http://pgloader.projects.postgresql.org/dev/TODO.html
http://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_loading
http://packages.debian.org/pgloader --- experimental has the next version

As for the performances benefit of this new version (2.3.0~dev2), all the work
could be reduced to zilch because of the python Global Interpreter Lock,
which I've been aware of late in the development effort.
http://docs.python.org/api/threads.html

This documentation states that (a) using generators you're not that
concerned, and (b) the global lock still allows for IO and processing at the
same time. As pgloader uses generators, I'm still not sure how much a problem
this will be.

I'd like to have some feedback about the new version, in term of bugs
encountered and performance limitations (is pgloader up to what you would
expect a multi-threaded loader to be at?)

Regards,
--
dim

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Stuart Brooks 2008-02-26 12:15:07 Re: autovacuum not freeing up unused space on 8.3.0
Previous Message Gregory Stark 2008-02-26 12:05:15 Re: autovacuum not freeing up unused space on 8.3.0

Browse pgsql-performance by date

  From Date Subject
Next Message valgog 2008-02-26 13:26:52 Re: response time when querying via JDBC and via psql differs
Previous Message Tom Lane 2008-02-26 01:25:10 Re: when is a DELETE FK trigger planned?