Re: Time based lag tracking for logical replication

On 03/05/17 08:28, Simon Riggs wrote:
> On 23 April 2017 at 01:10, Petr Jelinek <petr(dot)jelinek(at)2ndquadrant(dot)com> wrote:
>> Hi,
>>
>> The time based lag tracking commit [1] added interface for logging
>> progress of replication so that we can report lag as time interval
>> instead of just bytes. But the patch didn't contain patch for the
>> builtin logical replication.
>>
>> So I wrote something that implements this. I didn't like all that much
>> the API layering in terms of exporting the walsender's LagTrackerWrite()
>> for use by plugin directly. Normally output plugin does not have to care
>> if it's running under walsender or not, it uses abstracted write
>> interface for that which can be implemented in various ways (that's how
>> we implement SQL interface to logical decoding after all). So I decided
>> to add another function to the logical decoding write api called
>> update_progress and call that one from the output plugin. The walsender
>> then implements that new API to call the LagTrackerWrite() while the SQL
>> interface just does not implement it at all. This seems like cleaner way
>> of doing it.
>>
>> Thoughts?
>
> Agree cleaner.
>
> I don't see any pacing or comments about it, nor handling of
> intermediate messages while we process a large transaction.

Agreed, pacing is good idea because on busy server storing info for
every commit could get expensive.

Don't understand what you mean by "handling of intermediate messages
while we process a large transaction". Logical replication is
transaction based so far, it does not stream like physical replication
so it seems like there is limited usefulness in doing this outside of
commit no?

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services