From: | Heikki Linnakangas <hlinnaka(at)iki(dot)fi> |
---|---|
To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
Cc: | james+postgres(at)carbocation(dot)com, pgsql-bugs(at)postgresql(dot)org |
Subject: | Re: BUG #14654: With high statistics targets on ts_vector, unexpectedly high memory use & OOM are triggered |
Date: | 2017-07-12 19:09:23 |
Message-ID: | 8cd9ffd4-b32b-2eeb-9451-e1a104e93be2@iki.fi |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-bugs |
On 07/12/2017 06:30 PM, Tom Lane wrote:
> Heikki Linnakangas <hlinnaka(at)iki(dot)fi> writes:
>> Yes, I can see that happening here too. The problem seems to be that the
>> analyze-function detoasts every row in the sample. Tsvectors can be very
>> large, so it adds up.
>
>> That's pretty easy to fix, the analyze function needs to free the
>> detoasted copies as it goes. But in order to do that, it needs to make
>> copies of all the lexemes stored in the hash table, instead of pointing
>> directly to the detoasted copies.
>
>> Patch attached. I think this counts as a bug, and we should backport this.
>
> +1. I didn't test the patch, but it looks sane to the eyeball.
Ok, committed.
In some quick testing on my laptop, and the extra palloc+pfree adds
about 10% of overhead, in the worst case scenario that every tsvector in
the sample consists of totally unique lexemes. That's a bit unfortunate,
but it's a lot better than consuming gigabytes of memory.
- Heikki
From | Date | Subject | |
---|---|---|---|
Next Message | Kyotaro HORIGUCHI | 2017-07-13 04:10:43 | Re: PostgreSQL hot standby Hangs due to AccessExclusiveLock on pg_attribute or pg_type tables |
Previous Message | Peter Geoghegan | 2017-07-12 17:44:33 | Re: BUG #14722: Segfault in tuplesort_heap_siftup, 32 bit overflow |