Re: Problem with mirrorring

> -----Original Message-----
> From: Devrim GUNDUZ [mailto:devrim(at)gunduz(dot)org]
> Sent: 02 October 2004 21:00
> To: Dave Page
> Cc: PostgreSQL WWW Mailing List
> Subject: Re: [pgsql-www] Problem with mirrorring
>
> >> There was not an 'error' exactly...
> >
> > Ahh, yes. Of course, that's actually rsync working correctly! No
> > changes on the server, so no transfers.
>
> Umm, every cron job outputs an email of 259K, but when there
> is a lockfile, then it decreases to 800 bytes. Rsync gets all
> the files, even if there is no change!

It shouldn't do. Andrew Tridgell (sp?) of the Samba team wrote it for
his Phd thesis, designing it specifically to minimise the data
transferred. Iirc, it only even transfers the differences between files
rather than the whole thing. It's pretty clever stuff.

Anyway (I'll stop babbling now :-) ), if it's transferring everything,
even if it hasn't even been touched at out end, then something,
somewhere is broken.

> > This does mean that our mirroring is quite innefficient. We really
> > should diff the results of each build and only update the
> html files
> > if there is a change. At the moment each mirror is probably pulling
> > the whole site each time :-(
>
> Agreed but... diffing all the files that we have might be
> quite hard for us -- But I now don't have a better idea...
>
> Umm... If we can keep a track of modified files in a db (a
> new or updated FAQ, doc, news, etc...) maybe we could publish
> only them...

It's more tricky than that, because files change even if a new user join
Gborg for example (because of the count in the left column of the
'portal' pages). Besides, thinking about it some more I don't think it
really is that much of a problem. If rsync does it's job as I remember
it should (bear in mind it's a while since I read the thesis, and it's
Saturday night!), then it should minimise the transfer even if the mod
time has been touched.

Regards, Dave.