Libpq question related to allocated resources

From: Karl Denninger <karl(at)denninger(dot)net>
To: pgsql-general(at)lists(dot)postgresql(dot)org
Subject: Libpq question related to allocated resources
Date: 2022-06-27 16:35:24
Message-ID: 5cb156e4-0c40-d66e-5fcd-94556f58bb2b@denninger.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I've got a fairly sizeable application that runs as a CGI app under
Apache which I am attempting to convert to FastCGI.

"Once through and done" apps tend not to care much if deallocation is
less than perfect, since exit(0) (or otherwise) tends to heal all
wounds.  Not so much when something's running for hours, days or weeks. 
There memory leaks are ruinous.

I've wrapped all of my internal functionality plus all calls to
PQexecParams and the escape/unescape functions, all of which must be
deallocated after being used.  All counters after a pass through the
code are zero (increment on allocate, decrement on PQfremem or PQclear.)

But -- I still have a /lot /of memory out on the heap according to
jemalloc stats that is not being deallocated, and what's worse is that
if I rig the code to call PQfinish and then PQconnect once again I get
/even more /imbalanced allocate/free counts (and the memory use in said
buckets to go with them.)

The application itself is nothing particularly fancy although it
typically makes dozens of Postgres calls; single-threaded, no prepared
statements or async requests.

This is under 14.1; I haven't rolled the code forward, but I see nothing
in the notes implying there is a problem in libpq that has been
corrected, or that there was one in the past in this regard. Its also
possible that the FastCGI wrapper has a problem internally.  The app,
when run under valgrind to do cron processing, comes back clean -- it
does show allocations on exit however, although "still accessibl/e" /and
those which do come up are related from OpenSSL's error string
initialization in /lib/libcrypto.so (I'm on FreeBSD and openssl was
specified as "--with-openssl" when Postgres was built.)

The obvious question, given the warnings in the FastCGI library: Does
libpq /modify /the process environment?  Reading from it (provided you
don't modify anything from the pointers you access; if you want to then
you must copy them somewhere and make the modification outside of the
environment itself) is perfectly fine but writing it, directly or
indirectly, is NOT.  A quick grep implies that indeed it may in
backend/libpq/auth.c at least, but I do not have ENABLE_GSS defined in
my configuration so that code /shouldn't /be there.

--
Karl Denninger
karl(at)denninger(dot)net
/The Market Ticker/
/[S/MIME encrypted email preferred]/

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Bryn Llewellyn 2022-06-27 19:36:54 Re: Outer joins and NULLs (old subject "ERROR: failed to find conversion function from key_vals_nn to record[]")
Previous Message Magnus Hagander 2022-06-27 14:33:35 Re: Table space not returned to the OS ?