Multiple message delivery on logical replication

From: Christophe Pettus <xof(at)thebuild(dot)com>
To: psycopg(at)postgresql(dot)org
Subject: Multiple message delivery on logical replication
Date: 2018-11-03 01:28:51
Message-ID: A7E70A6B-04AB-4D26-BD7E-5E44DC0DA96D@thebuild.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: psycopg

I'm working with the logical replication support in psycopg2, and have found something surprising... this may be my error, of course!

My sample program is below. It works wonderfully, but in the case when it starts, it re-receives the last message that it handled, even with flushing it.

Example:

postgres(at)localhost:~/wal2pubsub$ python waltest.py
{"change":[{"kind":"insert","schema":"public","table":"x","columnnames":["i"],"columntypes":["integer"],"columnvalues":[6]},{"kind":"insert","schema":"public","table":"x","columnnames":["i"],"columntypes":["integer"],"columnvalues":[7]}]}
^C
postgres(at)localhost:~/wal2pubsub$ python waltest.py
{"change":[{"kind":"insert","schema":"public","table":"x","columnnames":["i"],"columntypes":["integer"],"columnvalues":[6]},{"kind":"insert","schema":"public","table":"x","columnnames":["i"],"columntypes":["integer"],"columnvalues":[7]}]}

There was no database activity in that period; it just replayed the same message. Shouldn't it have flushed to the end of the WAL stream and not reprocessed the last message?

--

import psycopg2
from psycopg2.extras import LogicalReplicationConnection, REPLICATION_LOGICAL

conn = psycopg2.connect('dbname=postgres', connection_factory=LogicalReplicationConnection)
cur = conn.cursor()

cur.start_replication(slot_name='test_slot', slot_type=REPLICATION_LOGICAL)

from select import select
from datetime import datetime

def consume(msg):
print(msg.payload)
msg.cursor.send_feedback(flush_lsn=msg.data_start)

try:
cur.consume_stream(consume)
except:
pass

--
-- Christophe Pettus
xof(at)thebuild(dot)com

Responses

Browse psycopg by date

  From Date Subject
Next Message Jonathan S. Katz 2018-11-03 01:31:45 Re: Multiple message delivery on logical replication
Previous Message Tony Shelver 2018-10-31 06:35:55 Max message size / constraints?