Re: BUG #13484: Performance problem with logical decoding

Hi

>> To be more concise :
>> - ONE transaction with 4095 operations -> consumed in 80mS
>> - TEN transactions with 4095 operations each (so 40950 operations) -> 380mS (which as you said is very good -less than linear growth-)
>> - ONE transaction with 4096 operations -> consumed in 4204mS (ouch ...). I confirm there are only 4096 changes in one transaction
>> - TEN transactions with 4095 operations each (so 40950 operations) -> 34998mS, ouch again>
>
>FWIW, I can't reproduce those results at all. I just tried, and for me
>all results are 34ms, 121ms, 149ms, 374ms. When using the streaming
>interface there's no discernible delays at all.

Very interesting

>> I also tried pg_recvlogical feeding a file, but got the exact same
>> results so i'm stuck.
>
>Are you stopping pg_recvlogical between the runs, or are you letting it
>run? The point of using it is that it's a streaming, i.e. that you do
>not need to pay to "startup" costs of logical decoding, which can be
>noticeable.

No, I just let it run. In my case the bottleneck seems to be on server side when spilling to disk. I can see the server build the files but very slowly.

>> What is observe is that the spilling occurs, and
>> when the .snap file is created then pg_recvlogical will consume data
>> but it does take a long time exactly the same time as
>> pg_logical_slot_get_changes in fact.
>
>Another possibility is that there's some windows specific problem here.

I do agree. It must be a Windows issue then. I reproduced it on two differents dev machines both with SSD and Windows 7. I can make a try on a real Windows 2008 or 2012 server with SAN access just to check. Will do it tomorrow

>> BTW there is also a bug in pg_recvlogical with option -f - (output to
>> stdout), pg_recvlogical tries to flush with fsync which does not work
>> on windows and display an error message.
>
>Uh, that should obviously not be the case. Thanks for the report, will
>fix.

Thanks

Best regards,
Olivier