Re: Streaming replication and non-blocking I/O

On Wed, Dec 16, 2009 at 6:53 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Great! The logical next step is move the handling of TimelineID and
> system identifier out of libpq as well.

All right.

> 0. Begin by connecting to the master just like a normal backend does. We
> don't necessarily need the new ProtocolVersion code either, though it's
> probably still a good idea to reject connections to older server versions.

And, I think that such backend should switch to walsender mode when the startup
packet arrives. Otherwise, we would have to authenticate such backend twice
on different context, i.e., a normal backend and walsender. So the settings for
each context would be required in pg_hba.conf. This is odd, I think. Thought?

> 1. Get the system identifier of the master.
>
> Slave -> Master: Query message, with a query string like
> "GET_SYSTEM_IDENTIFIER"
>
> Master -> Slave: RowDescription, DataRow CommandComplete, and
> ReadyForQuery messages. The system identifier is returned in the DataRow
> message.
>
> This is identical to what happens when a query is executed against a
> normal backend using the simple query protocol, so walsender can use
> PQexec() for this.

s/walsender/walreceiver ?

A signal cannot cancel PQexec() during waiting for the message from the
server. We might need to change SIGTERM handler of walreceiver so as to
call proc_exit() immediately if it's during PQexec().

> 2. Another query exchange like above, for timeline ID. (or these two
> steps can be joined into one query, to eliminate one round-trip).
>
> 3. Request a backup history file, if needed:
>
> Slave -> Master: Query message, with a query string like
> "GET_BACKUP_HISTORY_FILE XXX" where XXX is XLogRecPtr or file name.
>
> Master -> Slave: RowDescription, DataRow CommandComplete and
> ReadyForQuery messages as usual. The file contents are returned in the
> DataRow message.
>
> 4. Start replication
>
> Slave -> Master: Query message, with query string "START REPLICATION:
> XXXX", where XXXX is the RecPtr of the starting point.
>
> Master -> Slave: CopyOutResponse followed by a continuous stream of
> CopyData messages with WAL contents.

Seems OK.

> This minimizes the changes to the protocol and libpq, with a clear way
> of extending by adding new commands. Similar to what you did a long time
> ago, connecting as an actual backend at first and then switching to
> walsender mode after running a few queries, but this would all be
> handled in a separate loop in walsender instead of running as a
> full-blown backend.

Agreed. Only walsender should be allowed to handle the query strings that
you proposed, in order that we avoid touching a parser.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center