Re: A little COPY speedup

On Fri, 2007-03-02 at 16:25 +0000, Gregory Stark wrote:
> "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>
> > "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> >> I'm slightly worried though since that seems to have changed from 8.2,
> >> which I oprofiled over Christmas.
> >
> > If you were testing a case with wider rows than Heikki tested, you'd see
> > less impact --- the cost of the old way was O(N^2) in the number of
> > tuples that fit on a page, so the behavior gets rapidly worse as you get
> > down to smaller tuple sizes. (Come to think of it, the cmin/cmax
> > collapse would be a factor here too.)
>
> Or larger block sizes of course. A 32kb block would be 16x as bad which starts
> to be pretty serious.

Well, I was only using 8kb blocks.

But I think the message is clear: we need to profile lots of different
combinations. I was using a 2 col table with integer, char(100).

IIRC there are issues with delimiter handling when we have lots of
columns in the input on COPY FROM, and num of cols on COPY TO. I've not
looked at those recently though.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com