From: | "Worky Workerson" <worky(dot)workerson(at)gmail(dot)com> |
---|---|
To: | "Craig A(dot) James" <cjames(at)modgraph-usa(dot)com> |
Cc: | "Jim C(dot) Nasby" <jim(at)nasby(dot)net>, "Merlin Moncure" <mmoncure(at)gmail(dot)com>, pgsql-performance(at)postgresql(dot)org |
Subject: | Re: Best COPY Performance |
Date: | 2006-10-25 12:03:38 |
Message-ID: | ce4072df0610250503g26f32cb7r4e668613c8665774@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-performance |
On 10/25/06, Craig A. James <cjames(at)modgraph-usa(dot)com> wrote:
> Jim C. Nasby wrote:
> > Well, given that perl is using an entire CPU, it sounds like you should
> > start looking either at ways to remove some of the overhead from perl,
> > or to split that perl into multiple processes.
>
> I use Perl for big database copies (usually with some processing/transformation along the
> way) and I've never seen 100% CPU usage except for brief periods, even when copying
> BLOBS and such. My typical copy divides operations into blocks, for example doing
I'm just doing CSV style transformations (and calling a lot of
functions along the way), but the end result is a straight bulk load
of data into a blank database. And we've established that Postgres
can do *way* better than what I am seeing, so its not suprising that
perl is using 100% of a CPU.
However, I am still curious as to the rather slow COPYs from psql to
local disks. Like I mentioned previously, I was only seeing about 5.7
MB/s (1.8 GB / 330 seconds), where it seemed like others were doing
substantially better. What sorts of things should I look into?
Thanks!
From | Date | Subject | |
---|---|---|---|
Next Message | Alex Stapleton | 2006-10-25 12:16:15 | Re: Best COPY Performance |
Previous Message | Craig A. James | 2006-10-25 05:36:04 | Re: Best COPY Performance |