Re: FW: Java Memory Issue while Loading Postgres library

-----Original Message-----
From: Craig Ringer [mailto:craig(at)postnewspapers(dot)com(dot)au]
Sent: Thursday, April 29, 2010 6:02 PM
To: Bhattacharya, A
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: FW: [GENERAL] Java Memory Issue while Loading Postgres
library

On 29/04/2010 7:34 PM, A(dot)Bhattacharya(at)sungard(dot)com wrote:

> We have a java exe making a call to a postgres function. This postgres
> function internally makes a call to a dll (which is written using
> Postgres extended C).

If I understand you correctly, you have a dll that uses ecpg to
communicate with postgresql. It is loaded by a JVM via JNI, and that JVM

then uses JNI calls to use your ecpg-based DLL to talk to the database.

If I've understood you right: Why this strange architecture? Usually
Java apps just talk to the database via JDBC.

As far as I know there's no particular reason you *can't* do calls to
your ecpg-using dll via JNI, though. I'd be surprised if your problems
didn't turn out to be misuse/misunderstanding of the
notoriously-hard-to-get-right JNI interface, or issues with your DLL
and/or its use of ecpg.

If I have NOT understood how you are doing things correctly, then please

be MUCH MORE SPECIFIC. Making people guess what you're doing doesn't
help you get accurate, useful answers.

I strongly recommend that you try to produce a self-contained test case.

Write a minimalist ecpg dll that does only one simple thing. Write a
minimalist, simple JNI-using Java program that loads and uses that DLL.
See if it leaks memory. If this minimalist test case demonstrates the
leak, post the source code to BOTH the dll and your simple Java program
here. Make sure both can be compiled without extra libraries and include

any project files etc required to compile them.

If your new test case doesn't leak, then you know the basic JNI + ECPG
combination isn't the issue. You need to start investigating what in
your code is causing the leak. If you just can't figure it out, then
start chopping things out and disabling things until the leak stops
happening to narrow it down.

Without a lot more detail than you have provided, it is unlikely that
anybody here can help you.

> Now the issue is that, when we make a call to this dll, it consumes a
> lot of memory and this memory is getting consumed from the heap space
of
> the original java process causing an out of memory exception in Java.
>
> Is this a known issue.

I rather doubt that anybody else anywhere in the world is doing what
you're doing ;-)

Not as far as I know it's not, no. Nor have you provided any evidence
it's an issue with PostgreSQL not your own code yet.

> Do we have a way to keep these processes disjoint
> and not eat up heap space of the original process?

http://google.com/search?q=inter+process+communication

... but you should probably fix the problem that's causing the excessive

memory use/leakage rather than just working around it by running the
code out-of-process.

--
Craig Ringer
------------------------------------------------------------------------
----------------

Thanks Craig for your comment.

Your understanding is slightly incorrect. Actually we required to uses a
special library from postgres. For this we write a wrapper around the
library and composed a dll out of it (which uses ECPG to communicate to
postgres). From the application front (Java) we make a JDBC call to
postgres function. This postgres function then makes call to the dll.
But the strange thing we observed that at runtime, somehow the execution
of dll is attached with java.exe. In other words, when we monitor the
performance of the application we realized that the execution of dll is
consuming java heap memory and is shown as a thread in java.exe process.
We expected that it should be linked with postgres.exe. And this
incorrect linking resulted in out of heap memory error in java. But when
we run the same postgres function using pgAdmin tool, the memory and CPU
utilization of the dll is shown attached with the postgres.exe
(corresponding to pgAdmin).

I hope I am able to explain the issue.