Re: vm/swap used until exhausted

Zane <Zane(at)mail4z(dot)com> writes:
> Different memory usage 7.4.3 vs 8.0.0beta1
> client does:
> begin
> bulk inserts into single table via PQexecParams (1.2 million records)
> commit
> under 7.4.3 memory usage is static
> under 8.0.0beta1 server used increasing memory untill depletion of vm/swap

I've looked into this, and the source of the problem is the new
ResourceOwner mechanism we added to manage locks etc. held by
subtransactions. Each of the INSERT commands takes out another
lock on the target table. In prior releases this had no effect
except to increment a lock count in shared memory. In CVS tip,
each lock request is also recorded in a ResourceOwner object,
and it's the accumulation of those that is responsible for the
memory leak.

To deal with this, I am thinking about creating a new hash table
(local in each backend) that records locks already held, the
ResourceOwner(s) they are held on behalf of, and a lock count
for each one. Increasing the lock count for a lock already held
would thus not need any additional memory. Another nice property
is that we could have the shared-memory lock table register only
one lock count per backend; increasing the local lock count for
an already-obtained lock wouldn't require touching shared memory
and thus not require obtaining the LockMgrLock. (This would be
comparable to the existing mechanism for private vs. shared reference
counts for buffers.) That might be enough of a win to buy back
the extra time spent maintaining the additional hash table.

This is a bigger change than I'd really like to be making in beta,
but I don't see any other good solution to the memory-leak problem.
Anyone have a better idea?

regards, tom lane