Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Sun, Sep 29, 2019 at 02:30:44PM -0300, Alvaro Herrera wrote:
>On 2019-Sep-29, Amit Kapila wrote:
>
>> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
>
>> > So that's what I did in the attached patch - I've renamed the GUC to
>> > logical_decoding_work_mem, detached it from m_w_m and set the default to
>> > 64MB (i.e. the same default as m_w_m).
>>
>> Fair enough, let's not argue more on this unless someone else wants to
>> share his opinion.
>
>I just read this part of the conversation and I agree that having a
>separate GUC with its own value independent from other GUCs is a good
>solution. Tying it to m_w_m seemed reasonable, but it's true that
>people frequently set m_w_m very high, and it would be undesirable to
>propagate that value to logical decoding memory usage.
>
>
>I wonder what would constitute good advice on how to set this value, I
>mean what is the metric that the user needs to be thinking about. Is
>it the total of memory required to keep all concurrent write transactions
>in memory? (Quick example: if you do 2048 wTPS and each transaction
>lasts 1s, and each transaction does 1kB of logically-decoded changes,
>then ~2MB are sufficient for the average case. Is that correct?

Yes, something like that. Essentially we'd like to keep all concurrent
transactions decoded in memory, to eliminate the need to spill to disk.
One of the subsequent patches adds some subscription-level stats, so
maybe we don't need to worry about this too much - the stats seem like a
better source of information for tuning.

>I *think* that full-page images do not count, correct? With these
>things in mind users could go through pg_waldump output and figure out
>what to set the value to.)
>

Right, FPW do not matter here.

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services