Re: Streaming replication and non-blocking I/O

I'm interested in abstracting out features of replication from libpq too. It
would be nice if we could implement different communication bus modules.

For example if you have dozens of replicas you may want to use something
like spread to distribute the records using multicast.

Sorry for top posting -- I haven't yet figured out how not to in this
client.

On 16 Dec 2009 09:54, "Heikki Linnakangas" <
heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:

Fujii Masao wrote: > On Tue, Dec 15, 2009 at 3:47 AM, Heikki Linnakangas >
<heikki(dot)linnakangas(at)enter(dot)(dot)(dot)
Great! The logical next step is move the handling of TimelineID and
system identifier out of libpq as well.

I'm thinking of refactoring the protocol along these lines:

0. Begin by connecting to the master just like a normal backend does. We
don't necessarily need the new ProtocolVersion code either, though it's
probably still a good idea to reject connections to older server versions.

1. Get the system identifier of the master.

Slave -> Master: Query message, with a query string like
"GET_SYSTEM_IDENTIFIER"

Master -> Slave: RowDescription, DataRow CommandComplete, and
ReadyForQuery messages. The system identifier is returned in the DataRow
message.

This is identical to what happens when a query is executed against a
normal backend using the simple query protocol, so walsender can use
PQexec() for this.

2. Another query exchange like above, for timeline ID. (or these two
steps can be joined into one query, to eliminate one round-trip).

3. Request a backup history file, if needed:

Slave -> Master: Query message, with a query string like
"GET_BACKUP_HISTORY_FILE XXX" where XXX is XLogRecPtr or file name.

Master -> Slave: RowDescription, DataRow CommandComplete and
ReadyForQuery messages as usual. The file contents are returned in the
DataRow message.

4. Start replication

Slave -> Master: Query message, with query string "START REPLICATION:
XXXX", where XXXX is the RecPtr of the starting point.

Master -> Slave: CopyOutResponse followed by a continuous stream of
CopyData messages with WAL contents.

This minimizes the changes to the protocol and libpq, with a clear way
of extending by adding new commands. Similar to what you did a long time
ago, connecting as an actual backend at first and then switching to
walsender mode after running a few queries, but this would all be
handled in a separate loop in walsender instead of running as a
full-blown backend. We'll still need small changes to libpq to allow
sending messages back to the server in COPY_IN mode (maybe add a new
COPY_IN_OUT mode for that).

Thoughts?

-- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com --

Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org) To make
changes to your subscript...