Memory leak in GIN index build

From: Julien Rouhaud <julien(dot)rouhaud(at)dalibo(dot)com>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Cc: Marc Cousin <marc(dot)cousin(at)dalibo(dot)com>
Subject: Memory leak in GIN index build
Date: 2016-04-16 17:31:09
Message-ID: 571276DD.5050303@dalibo.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello,

Another colleague provided a report of memory leak, during a GIN index
build. Test case to reproduce the attached (need to create a gin index
on the val column after loading). Sorry, it generates a 24GB table, and
memory start leaking with a 1GB maintenance_work_mem after reaching 8 or
9 times the m_w_m setting, leading to ~ 9GB used in Gin build temporary
context.

After some digging, the leak comes from walbufbegin palloc in
registerLeafRecompressWALData().

IIUC, walbufbegin isn't pfree-d and can't be before XLogInsert() is
called, which happens in ginPlaceToPage().

I don't see a simple way to fix that. My first idea would be to change
GinBtreeData's placeToPage (and all other needed) functions signature to
pass a pointer to pfree in ginPlaceToPage() if needed, but it doesn't
really seem appealing. In any case, I'd be happy to work on a patch if
needed.

Also, in dataPlaceToPageLeaf() and ginVacuumPostingTreeLeaf(), shouldn't
the START_CRIT_SECTION() calls be placed before the xlog code?

Regards.

--
Julien Rouhaud
http://dalibo.com - http://dalibo.org

Attachment Content-Type Size
generate_data.pl application/x-perl 411 bytes

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2016-04-16 18:45:07 Re: Memory leak in GIN index build
Previous Message Tom Lane 2016-04-16 17:06:36 Spinlocks and semaphores in 9.2 and 9.3