Bulk Record upload (was Re: Increasing number of PG connections)

On Monday 02 February 2004 19:39, Qing Zhao wrote:
> I am new here. I have a question related to this in some way.

Hmm - no real connection I can see - might have been better to start a new
thread rather than replying to this one. Also, it is usually considered best
practice not to quote large amounts of the previous message if you're not
replying to it,

> Our web site needs to upload a large volume of data into Postgres at a
> time. The performance deterioates as number of rows becomes larger.
> When it reaches 2500 rows, it never come back to GUI. Since the tests
> were run through GUI, my suspision is
> that it might be caused by the way the application server talking to
> Postgres server, the connections, etc.. What might be the factors
> involved here? Does anyone know?

You don't really give us enough information. What GUI are you talking about?
How are you loading this data - as a series of INSERT statements, text-file
with separators, from Access/MySQL etc?

In general, the fastest way to add a large number of rows is via the COPY sql
command. Next best is to batch your inserts together into larger transactions
of say 100-1000 inserts.

Two other things to be aware of are: use of VACUUM/ANALYZE and configuration
tuning (see http://www.varlena.com/varlena/GeneralBits/Tidbits/index.php)

PG shouldn't have a problem with inserting a few thousand rows, so I suspect
it's something to do with your application/GUI setup.

Hope that helps, if not try turning on statement logging for PG and then we
can see what commands your GUI is sending.

--
Richard Huxton
Archonet Ltd