Re: Thoughts on the mirroring system etc

>> And if I'm stepping on someones toes here, let me apologize
>> in advance.
>
>Only really mine - and I knew you were writing this :-)

:-)
Well, feel free to tell me to stop ;-)

>As an example, I run www.uk.postgresql.org. On the 10th Jan, a date
>picked pretty much at random, I logged 2448 http requests. Each hit on
>the homepage results in about 30(!) httpd requests, so
>represents as few
>as 82 hits!
>
>Yesterday, release day, I only logged 2387 hits!!

Which proves my point (and yours). Thanks :-)

>> My suggestion for this is to limit the number of mirrors to around 5,
>> give or take a few. But instead, put higher demands on these mirrors
>> than we do now. Demand they sync every 30 minutes (or 60, but you get
>> the point). Demand that they have a fast machine and a fast network
>> connection. There have been enough offers of servers and
>networks that
>> this should not be a major problem. Demand that they respond to
>> www.postgresql.org - if it can have a dedicated IP, even better.
>> Distributed across the world of course.
>
>Yes - we are already planning to do this, and indeed some of the work
>has been done. The mirror tracker checks whether or not a mirror will
>respond to www.postgresql.org requests, and the backend database has a
>flag to mark the 'primary' mirrors.

Ok. All good. Though I beleive some of this is unnecessary - if you
operate from the standpoint that *all* approved mirrors need to answer
requests for www.postgresql.org.

>> Then do some "DNS magic" to do the load balancing:
>
><snip DNS Magic>
>
>Yes, the current mirror tracker could easily be adapted to do this.

Right. And IMHO, it should be moved off webmaster and have a separate
system - it has different needs, and separation of critical services is
good.

>> A similar solution for wwwmaster, of course.
>>
>
>The major problem with wwwmaster is that we need multimaster
>replication
>to handle it properly, without having a single point of
>failure. Slony 1
>will not resolve that basic issue.

No, I beleive you can solve this. Let's assume we don't care if we can't
add/remove news and events. AFAIK, then the database is almost only
INSERTs right - answers to surveys, redirect logging etc?
For this, create two tables, say "log1" and "log2". Where one of the
servers each own one table, and only writes to that one. You set up two
sets of slony replications, one in each direction. Then you create a
view that is a UNION ALL of these, that's the one used when you read
from the table.
Simplified, but most of the time you can spot fairly easy ways to do
this in the application.

Getting rid of as many single points of failure as possible should be an
overall goal.

>> wwwmaster
>> ---------
>> If you hit the ftp browser (or a download link), and then
>> click anything
>> in the menu, you get the whole site served from wwwmaster. If
>> the above
>> is fixed, so mirrors are all referred to as www.postgresql.org, it
>> should be as simple as sticking a <base href> in there or
>> something. BUt
>> until then, perhaps some creative coding in the framework can
>> fix it so
>> links that are hit on wwwmaster point back to www whereas the static
>> site uses relative links only?
>
>Yes, I need to think about this. At the moment, the flags on the mirror
>pages have been hardcoded back to www, but a better solution is needed.

IF the decision is made to require that all mirrors "we care about"
answer to www.postgresql.org, then the problem goes away.

>> Wow. That was a lot longer than initially intended. Hope
>> someone has the
>> patience to read it all ;-)
>
>I did :-)

Yay!

//Magnus