From: | Dimitri Fontaine <dfontaine(at)hi-media(dot)com> |
---|---|
To: | pgsql-performance(at)postgresql(dot)org |
Subject: | multi-threaded pgloader needs your tests |
Date: | 2008-02-26 12:08:53 |
Message-ID: | 200802261308.53459.dfontaine@hi-media.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general pgsql-performance |
Hi,
You may remember some thread about data loading performances and
multi-threading support in pgloader:
http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php
The pgloader code to handle this is now ready to get tested, a more structured
project could talk about a Release Candidate status.
http://pgloader.projects.postgresql.org/dev/TODO.html
http://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_loading
http://packages.debian.org/pgloader --- experimental has the next version
As for the performances benefit of this new version (2.3.0~dev2), all the work
could be reduced to zilch because of the python Global Interpreter Lock,
which I've been aware of late in the development effort.
http://docs.python.org/api/threads.html
This documentation states that (a) using generators you're not that
concerned, and (b) the global lock still allows for IO and processing at the
same time. As pgloader uses generators, I'm still not sure how much a problem
this will be.
I'd like to have some feedback about the new version, in term of bugs
encountered and performance limitations (is pgloader up to what you would
expect a multi-threaded loader to be at?)
Regards,
--
dim
From | Date | Subject | |
---|---|---|---|
Next Message | Stuart Brooks | 2008-02-26 12:15:07 | Re: autovacuum not freeing up unused space on 8.3.0 |
Previous Message | Gregory Stark | 2008-02-26 12:05:15 | Re: autovacuum not freeing up unused space on 8.3.0 |
From | Date | Subject | |
---|---|---|---|
Next Message | valgog | 2008-02-26 13:26:52 | Re: response time when querying via JDBC and via psql differs |
Previous Message | Tom Lane | 2008-02-26 01:25:10 | Re: when is a DELETE FK trigger planned? |