Re: PostgreSQL + Replicator developer meeting 10/28

Hannu Krosing wrote:
> On Tue, 2008-10-28 at 22:37 -0300, Alvaro Herrera wrote:

> > If you mean that we're duplicating the effort that's already going
> > elsewhere, my opinion is yes, we are.
>
> duplicating the effort is not always a bad thing. I was mostly
> suggesting to watch discussions and dig around in materials and/or
> asking people who have been working on these same issues.
>
> And of course to _think_ deeply about design before writing lots of
> duplicate code which ends up being an often inferior implementation of
> something that already exists, ( see:
> http://thedailywtf.com/Articles/The_Complicator_0x27_s_Gloves.aspx )
> ;-)

Are you asking whether the Replicator implementation contains gloves'
replacement? Yes, we are. In fact, a lot of the time I've spent on
Replicator code has been removing those (which were written by the
original Replicator developers) and just using already existing gloves.
For example we removed dealing TCP sockets directly; we now use libpq,
to which we only had to add a single support function.

> I was a heavy user (at Skype) at some point and have helped in fixing
> some. But in the end we could not figure out how to make it robust and
> extracted the good stuff for pgQ and wrote our own replication based on
> that, which we could make perform and be robust when changing conf.

A lot of the implementation effort has gone into the queue
implementation. Personally I am not happy with it yet. Perhaps we
should take a look at pgQ and remove our own implementation ...

> Do you transmit changes to and apply changes on slave as binary or as
> SQL statements ?

We have a binary protocol. We haven't worked out the details yet but it
certainly won't be SQL statements.

> Do slaves also have to be modified just to receive changes ?

Modify in what form? Obviously they must be capable of decoding the
binary protocol ...

> I think the hairy part will be getting the order of commands _exactly_
> right (like Hot Standby again), but if you are similar to
> Slony/pgQ/Londiste in that you just transfer logical changes, not
> physical page-level changes, then the DDL locking on master may be
> enough to guarantee the right order. That is assuming that you already
> can guarantee right (commit "time") order on slaves. this is not the
> same as transaction start order, which may give wrong/inconsistent data
> states.

Yeah, we transfer logical changes, not physical. We enqueue transaction
data at commit time (in commit order). I think being able to modify the
Postgres commit source code was the only reason we could do it.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support