Re: Interesting post-mortem on a near disaster with git

On Mon, Mar 25, 2013 at 7:07 PM, Stefan Kaltenbrunner
<stefan(at)kaltenbrunner(dot)cc> wrote:
> On 03/24/2013 11:22 PM, Andrew Dunstan wrote:
>>
>> On 03/24/2013 06:06 PM, Michael Paquier wrote:
>>> On Mon, Mar 25, 2013 at 12:52 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us
>>> <mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us>> wrote:
>>>
>>> Over the weekend, KDE came within a gnat's eyelash of losing *all*
>>> their authoritative git repos, despite having seemingly-extensive
>>> redundancy. Read about it here:
>>> http://jefferai.org/2013/03/24/too-perfect-a-mirror/
>>>
>>> It is really great that KDE people are actually sharing this
>>> experience. This is really profitable for other projects as well as
>>> individuals.
>>> And thanks for sharing it here.
>>>
>>>
>>> We should think about protecting our own repo a bit better,
>>> especially
>>> after the recent unpleasantness with a bogus forced update. The idea
>>> of having clones that are deliberately a day or two behind seems
>>> attractive ...
>>>
>>> Just an idea here: why not adding a new subdomain in postgresql.org
>>> <http://postgresql.org> for mirrors of the official GIT repository
>>> similar to the buildfarm?
>>> People registered in this service could publish themselves mirrors and
>>> decide by themselves the delay their
>>> clone keeps with the parent repo. The scripts used by each mirror
>>> maintainer (for backup, sync repo with
>>> a given delay) could be centralized in a way similar to buildfarm code
>>> so as everybody in the community could
>>> use it and publish it if they want.
>>>
>>> Also, the mirrors published should be maintained by people that are
>>> well-known inside the community,
>>> and who would not add extra commits which would make the mirror
>>> out-of-sync with the parent repo.
>>>
>>> Such an idea is perhaps too much if the point is to maintain 2-3
>>> mirrors of the parent repo, but gives
>>> enough transparency to actually know where the mirrors are and what is
>>> the sync delay maintained.
>>
>>
>>
>> This strikes me as being overkill. The sysadmins seem to have it covered.
>>
>> Back when we used CVS for quite a few years I kept 7 day rolling
>> snapshots of the CVS repo, against just such a difficulty as this. But
>> we seem to be much better organized with infrastructure these days so I
>> haven't done that for a long time.
>
> well there is always room for improvement(and for learning from others)
> - but I agree that this proposal seems way overkill. If people think we
> should keep online "delayed" mirrors we certainly have the resources to
> do that on our own if we want though...

Yeah, definitely.

It's also interesting to note that one of the things they do is to
"stop using mirrored clones". The fact that we *don't* use mirrored
clones for our anon repository is exectly how we caught the issue
caused by the invalid push that Kevin did a short while ago...

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/