Re: gitweb is no longer a real-time view

On Mon, Mar 4, 2013 at 1:50 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> On 2013-03-04 08:32:35 +0100, Magnus Hagander wrote:
>> On Mon, Mar 4, 2013 at 8:16 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> > On Mon, Mar 4, 2013 at 3:33 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>> >> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>> >>> cron job? I was under the impression there was some sort of push
>> >>> operation driven by a commit trigger. The web site has certainly
>> >>> updated nearly immediately for as long as we've been using git.
>> >>> Until this week, that is.
>> >>
>> >> Curiously, there's two cron jobs, apparently. There's a 'push' one and
>> >> then another, independent, 'pull' one. I'll assume they're actually
>> >> doing different things, but I wonder if the pull isn't just a
>> >> hold-over.. In any case, the push-to-anon, which I hadn't seen
>> >
>> > No, the pull one doesn't do the main repository - that one pulls
>> > "third party" repositories in as mirrors, such as e.g. Bucardo. That
>> > one was, btw, broken for several months and nobody noticed - so that's
>> > clearly a less important one :)
>> >
>> >> initially (looking at the pull side instead of the push side), does run
>> >> once a minute, though it looks like there's a hook mechanism which
>> >> would allow us to trigger the webserver to do a pull when a commit
>> >> happens and would still be better than a once-a-minute cronjob.
>> >
>> > The one that deals with the main repo mirroring runs once per minute.
>> > The commit trigger only drops a file in the filesystem so the cronjob
>> > knows there is something to do. So that we don't risk holding up the
>> > actual commit in case there is a problem with the anonymous mirror
>> > server.
>> >
>> >
>> >> Even with that, however, the concern raised was that the gitweb perl
>> >> script is quite expensive to run for every request, hence the reason for
>> >> doing the cacheing. I've lowered the varnish cacheing to a 5m ttl and
>> >> 15m grace and I'll keep an eye on it.
>> >
>> > That's fixing the wrong problem. The cache is supposed to be
>> > automatically purged whenever the push happens, to make sure that the
>> > data never gets more than maybe a second stale. So clearly something
>> > in that is not working - I didn't specifically test it with the
>> > postgres.git repository, but I tested it with a couple of other ones
>> > where it worked fine. But maybe we did something special with the main
>> > one...
>>
>> Actually, looking closer, I'm seeing a failure when it actually tries the push:
>>
>> To ssh://git(at)git(dot)postgresql(dot)org/postgresql.git
>> ! [rejected] master -> master (non-fast-forward)
>>
>>
>> So the problem might have nothing at all to do with the cacheing.
>>
>> AFAICT, the three missing commits are materialized views,
>> accidentally committed .orig file and \l support.
>>
>> But. The *anonymous* repository also has:
>> bc61878682051678ade5f59da7bfd90ab72ce13b Fix
>> map_sql_value_to_xml_value() to treat domains like their base types.
>>
>> This patch is *not* in the master repository, it's only in anonymous.
>> (The object is in the repository, but it's not part of any branch)
>>
>> How the hell did *that* happen?
>>
>> The master repo has:
>> commit 5db5974c692b0fc68e7608dd85a6b4e6173a0f28
>> Author: Peter Eisentraut <peter_e(at)gmx(dot)net>
>>
>> psql: Let \l accept a pattern
>>
>> commit d63977eea3ab18fdec05e370b633d10b9fd20179
>> Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>
>>
>> Remove accidentally-committed .orig file.
>>
>> commit 3bf3ab8c563699138be02f9dc305b7b77a724307
>> Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>
>>
>> Add a materialized view relations.
>>
>> commit b15a6da29217b14f02895af1d9271e84415a91ae
>> Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>>
>> Get rid of any toast table when converting a table to a view.
>>
>>
>> And the anonymous one has:
>> commit bc61878682051678ade5f59da7bfd90ab72ce13b
>> Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>>
>> Fix map_sql_value_to_xml_value() to treat domains like their base types.
>>
>> commit 3bf3ab8c563699138be02f9dc305b7b77a724307
>> Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>
>>
>> Add a materialized view relations.
>>
>> commit b15a6da29217b14f02895af1d9271e84415a91ae
>> Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>>
>> Get rid of any toast table when converting a table to a view.
>>
>>
>>
>> Does anybody have an explanation for that? Did someone do a force-push
>> on the master repository, overwriting some old history?
>
> This really looks like Kevin forced a push.
>
> Could you show
> git show --pretty=raw d63977eea3ab18fdec05e370b633d10b9fd20179
> git show --pretty=raw bc61878682051678ade5f59da7bfd90ab72ce13b
> git show --pretty=raw 5db5974c692b0fc68e7608dd85a6b4e6173a0f28

In which of the repositories - master, anonymous, or both?

> ?
> I don't have all the commits here...
>
> The raw format will explicitly tell which parents are referenced in the
> commits...
>
>> We don't explicitly forbid this on the master repo, since we expect
>> committers to know how things work.. Maybe we need to do that, and
>> manually turn it off in case someone actually *needs* to do a non fast
>> forward push? But either way, it would be good to actually know how
>> tihs happened...
>
> If you allow it the mirroring script should always force a push,
> otherwise it will stop pushing in those cases.

Yeah. But we shouldn't do it, so disallowing it seems like a good thing.

> I think the most realistic way to resolve this is to push the current
> anongit state to master and cherry-pick
> bc61878682051678ade5f59da7bfd90ab72ce13b ontop it. That way only
> committers need to deal with rebasing their branches and not everyone
> else.

Yeah, I agree that's probably the best way to do it - the number of
committers is a lot more limited...

Anyone else have an opinion on that part?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/