Re: GDQ iimplementation

On 5/11/10, Jan Wieck <JanWieck(at)yahoo(dot)com> wrote:
> On 5/11/2010 9:36 AM, Marko Kreen wrote:
> > And although PgQ can operate with any N >= 2 segments, it queries
> > on 2 at a time, same as Slony. Rest are just there to give admins
> > some safety room for "OH F*CK" moments. With short rotation times,
> > it starts to seem useful..
> >
>
> Agreed. The rotation time should actually reflect the longest running
> transactions experienced on a frequent base from the application. And there
> needs to be a safeguard against rotating over even longer running
> transactions.

Nightly pg_dump.. ;)

> The problem with a long running transaction is that it could have written
> into log segment 1 before we switched to segment 2. We can only TRUNCATE
> segment 1 after that transaction committed AND the log has been consumed by
> everyone interested in it.
>
> I am not familiar with how PgQ/Londiste do this. Slony specifically
> remembers the highest XID in progress at the time of switching, waits until
> the lowest XID in progress is higher than that (so all log that ever went
> into that segment is now visible or aborted), then waits for all log in that
> segment to be confirmed and finally truncates the log. All this time, it
> needs to do the UNION query over both log segments.

The "highest XID" means actually "own transaction" here?
And it's not committed yet? That's seems to leave transactions
that happen before it's own commit into dubious state?

Although you may be fine, if you don't try to minimize
reading both tables.

PgQ does this:

Rotate:
1) If some consumer reads older table, don't rotate.
2) Set table_nr++, switch_step1 = txid_current(), switch_step2 = NULL
3) Commit
4) Set switch_step2 = txid_current() where switch_step2 IS NULL
5) Commit

Reader:
1) xmin1 = xmin of lower snapshot of batch
2) xmax2 = xmax of higher snapshot of batch
3) if xmax2 < switch_step1, read older table
4) if xmin1 > switch_step2, read newer table
5) otherwise read both

--
marko