Re: import function's progress report

Dave Page wrote:
> Hi
>
> On Wed, Dec 31, 2008 at 7:14 AM, Quan Zongliang <quanzongliang(at)gmail(dot)com> wrote:
>> Hi, all

>> 3. Columns page:
>> Set data format ( date & number), space trim and column's order
>> I plan to support express, user can input a pgsql express to treat data.
>
> Hmm, pgsql expressions could be slow as they can only be evaluated in
> insert statements, which is really not good for such a tool. I would
> suggest:
>
> - Consider embedding Python (which is on the TODO anyway) and allowing
> transforms to be scripted on the client.
> - Convert the import data to a COPY compatible format so it can be fed
> directly to the server without using INSERT which is *much* slower.

Agreed, and a feature I would very much like to see.

FYI, I recently wrote a small tool that did *just* this step (the
conversion while loading). When running both the client and the server
on the same machine, it still came out more than 3x faster if I did the
simple transforms in Python and maintained the COPY format rather than
switching to INSERT format.

Do you plan to support loading multiple tables in one batch
(transaction)? If not initially, it would be great if you could plan for
the ability to do so on in the future - that is, don't make assumptions
that we will not want it. Because that would be a great feature.

//Magnus