Re: Question regarding logical replication

Hi. Thank you very much for such detailed explanation. :)
We are currently testing the overhead of log-based Change Data Capture
method (i.e. logical decoding) over Postgresql.
The test setting consists of one processing running TPC-C on node1,
which issued transactions against a database residing on node2, which is
accessed by three CDC processes running on three different nodes, say
node3-5.
The difference between three CDC processes are only different table sets
they are interested while the commonness is that each of they would
sleep periodically during keeping capturing changes from node2.

We always measured the impact of CDC on original TPC-C workload by
looking into the transaction throughput on node1.
We selected 0ms, 200ms and 400ms as three different sleeping periods for
CDC processes to control their impact on node2.
We expect that the longer a sleeping period is set for CDC, the less
impact is incurred over Postgresql, since less I/Os are triggered to
fetch data from xlog.
However, the plots showed different trend (currently I don't have plots
on my laptop) which shows that the more frequently are the CDC processes
reading from logical slots, the less overhead is incurred over
PostgreSQL, which leads to higher throughput.

That's the reason why I asked the previous question, whether logical
slot is implemented as queue.
Without continuous dequeuing the "queue" get larger and larger, thus
lowering the OLTP workload.

Regards;

Weiping

On 26.10.2017 21:42, Alvaro Aguayo Garcia-Rada wrote:
> Hi. I've had experience with both BDR & pglogical. For each replication slot, postgres saves a LSN which points to the last xlog entry read by the client. When a client does not reads xlog, for example, if it cannot connect to the server, then the distance between such LSN(pg_replication_slots.restart_lsn) and the current xlog location(pg_current_xlog_insert_location()) will enlarge over the time. Not sure about the following, but postgres will not clear old xlog entries which are still pending to be read on any replication slot. Such situation may also happen, in lower degree, if the client cannot read WAL as fast as it's produced. Anyhow, what will happen is xlog will grow more and more. However, that will probably not impact performance, as xlog is written anyway. But if you don't have enough free space, you could get your partition full of xlog.
>
> Regards,
>
> Alvaro Aguayo
> Operations Manager
> Open Comb Systems E.I.R.L.
>
> Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
> Web: www.ocs.pe
>
> ----- Original Message -----
> From: "Weiping Qu" <qu(at)informatik(dot)uni-kl(dot)de>
> To: "PostgreSql-general" <pgsql-general(at)postgresql(dot)org>
> Sent: Thursday, 26 October, 2017 14:07:54
> Subject: [GENERAL] Question regarding logical replication
>
> Dear postgresql community,
>
> I have a question regarding understanding the implementation logic
> behind logical replication.
>
> Assume a replication slot created on the master node, will more and more
> data get piled up in the slot and the size of replication slot
> continuously increase if there is no slave reading/dequeuing data out of
> this slot or very slowly, thus incurring high I/Os and slow down the
> transaction throughput?
>
> Looking forward to your explanation.
>
>
> Kindly review and please share your comments on this matter.
>
>
>
>