Re: gitweb is no longer a real-time view

On Mon, Mar 4, 2013 at 8:16 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Mon, Mar 4, 2013 at 3:33 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>>> cron job? I was under the impression there was some sort of push
>>> operation driven by a commit trigger. The web site has certainly
>>> updated nearly immediately for as long as we've been using git.
>>> Until this week, that is.
>>
>> Curiously, there's two cron jobs, apparently. There's a 'push' one and
>> then another, independent, 'pull' one. I'll assume they're actually
>> doing different things, but I wonder if the pull isn't just a
>> hold-over.. In any case, the push-to-anon, which I hadn't seen
>
> No, the pull one doesn't do the main repository - that one pulls
> "third party" repositories in as mirrors, such as e.g. Bucardo. That
> one was, btw, broken for several months and nobody noticed - so that's
> clearly a less important one :)
>
>> initially (looking at the pull side instead of the push side), does run
>> once a minute, though it looks like there's a hook mechanism which
>> would allow us to trigger the webserver to do a pull when a commit
>> happens and would still be better than a once-a-minute cronjob.
>
> The one that deals with the main repo mirroring runs once per minute.
> The commit trigger only drops a file in the filesystem so the cronjob
> knows there is something to do. So that we don't risk holding up the
> actual commit in case there is a problem with the anonymous mirror
> server.
>
>
>> Even with that, however, the concern raised was that the gitweb perl
>> script is quite expensive to run for every request, hence the reason for
>> doing the cacheing. I've lowered the varnish cacheing to a 5m ttl and
>> 15m grace and I'll keep an eye on it.
>
> That's fixing the wrong problem. The cache is supposed to be
> automatically purged whenever the push happens, to make sure that the
> data never gets more than maybe a second stale. So clearly something
> in that is not working - I didn't specifically test it with the
> postgres.git repository, but I tested it with a couple of other ones
> where it worked fine. But maybe we did something special with the main
> one...

Actually, looking closer, I'm seeing a failure when it actually tries the push:

To ssh://git(at)git(dot)postgresql(dot)org/postgresql.git
! [rejected] master -> master (non-fast-forward)

So the problem might have nothing at all to do with the cacheing.

AFAICT, the three missing commits are materialized views,
accidentally committed .orig file and \l support.

But. The *anonymous* repository also has:
bc61878682051678ade5f59da7bfd90ab72ce13b Fix
map_sql_value_to_xml_value() to treat domains like their base types.

This patch is *not* in the master repository, it's only in anonymous.
(The object is in the repository, but it's not part of any branch)

How the hell did *that* happen?

The master repo has:
commit 5db5974c692b0fc68e7608dd85a6b4e6173a0f28
Author: Peter Eisentraut <peter_e(at)gmx(dot)net>

psql: Let \l accept a pattern

commit d63977eea3ab18fdec05e370b633d10b9fd20179
Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>

Remove accidentally-committed .orig file.

commit 3bf3ab8c563699138be02f9dc305b7b77a724307
Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>

Add a materialized view relations.

commit b15a6da29217b14f02895af1d9271e84415a91ae
Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>

Get rid of any toast table when converting a table to a view.

And the anonymous one has:
commit bc61878682051678ade5f59da7bfd90ab72ce13b
Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>

Fix map_sql_value_to_xml_value() to treat domains like their base types.

commit 3bf3ab8c563699138be02f9dc305b7b77a724307
Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>

Add a materialized view relations.

commit b15a6da29217b14f02895af1d9271e84415a91ae
Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>

Get rid of any toast table when converting a table to a view.

Does anybody have an explanation for that? Did someone do a force-push
on the master repository, overwriting some old history?

We don't explicitly forbid this on the master repo, since we expect
committers to know how things work.. Maybe we need to do that, and
manually turn it off in case someone actually *needs* to do a non fast
forward push? But either way, it would be good to actually know how
tihs happened...

Copying this one to Kevin and Peter too, since they seem to be the
other people who are involved in any of these pushes.

Seems like we have a problem much worse than just the web...

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/