Re: Postgresql 7.4.2 and OutOfMemoryError

Sharib Anis wrote:
> Hello All,
>
> I'm using Postgresql for our java application. I started with Postgres
> 7.2.1 and everything worked fine. Then however, I decided to upgrade to
> 7.4.2.
>
> Since then I'm facing this frustrating problem where I'm always getting
> OutOfMemoryError. Basically there is an operation which involves
> inserts/updates in batches of 20K records. With the earlier version of
> Postgresql and driver, it worked smoothly, but not now. Any ideas why?
> What should I do to overcome this? I'm using the latest 7.4 driver too.
> I suspect it's an issue with the driver.

20k sounds high for batch updates. The driver has to keep the batched
statements in memory until executeBatch(). Lowering the batch size may
help. I find that, over a 100mb network, there's little-to-no benefit
from batch sizes above 100 (@ 1k data per insert), even when the driver
is really doing batching (which the current driver doesn't do).

If you don't mind trying a very-experimental driver I have a version
with substantial improvements to bytea support & batch updates waiting
in the wings. I've seen a 5-fold speedup (and drastic reduction in
memory use) of a microbenchmark that does batch inserts of bytea data.
Let me know if you'd like to try that driver.

(I'm looking for testers if any other adventurous people want to try it
out..)

-O