ps.setCharacterStream() and memory usage

From: Sebastiaan van Erk <sebster(at)sebster(dot)com>
To: pgsql-jdbc(at)postgresql(dot)org
Subject: ps.setCharacterStream() and memory usage
Date: 2004-10-29 11:56:52
Message-ID: 20041029115651.GE4199@sebster.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-jdbc

Hi,

Coming back to another problem I have with the following insert statement:
(Postgres 7.4, driver build 215, jdbc 3)

ps.setCharacterStream(1, reader, (int) messageFile.length());
ps.executeUpdate();

The reason I do this (using a character stream to load a text file to a
TEXT field) is that I wish to avoid loading the file into memory completely.

I already noticed that the postgres driver does not stream the data to
the backend (I don't know if postgres actually supports this).

But what is worse, is that the file actually gets copied THREE times
into memory, causing my Java file to already get a java.lang.OutOfMemory
error with an insert of a 10M text file (Java allocates 20M for this,
since it has 2 byte chars, and the driver makes (at least) 3 copies
(referenced at the same time, not allowing one to be GC'ed), making 60M).

First of all, in setCharacterStream() it loads the file into a char array.
Then this is cloned into a String and passed to setString. (Here one can
already cause the char[] to go out of scope, at least allowing it to be
cleaned up). Secondly, setString causes another clone in escapeString().
(Ideally, one could read the stream into a StringBuffer big enough to allow
it to be escaped, and thereby only load the file into memory once instead
of three times). Finally, the driver (possibly) keeps large objects in
memory too long: for example, in setString() right after the bind, one can
already do a "sbuf.setLength(0)". This way, one does not have to wait
for setString (or another setter) to be called before the sbuf variable
is cleared.

Greetings,
Sebastiaan van Erk

Responses

Browse pgsql-jdbc by date

  From Date Subject
Next Message Vadim Nasardinov 2004-10-29 14:24:03 Re: \0 and IllegalArgumentException
Previous Message Sebastiaan van Erk 2004-10-29 11:46:46 Re: \0 and IllegalArgumentException