Re: gitweb is no longer a real-time view

On 2013-03-04 08:32:35 +0100, Magnus Hagander wrote:
> On Mon, Mar 4, 2013 at 8:16 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> > On Mon, Mar 4, 2013 at 3:33 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> >> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> >>> cron job? I was under the impression there was some sort of push
> >>> operation driven by a commit trigger. The web site has certainly
> >>> updated nearly immediately for as long as we've been using git.
> >>> Until this week, that is.
> >>
> >> Curiously, there's two cron jobs, apparently. There's a 'push' one and
> >> then another, independent, 'pull' one. I'll assume they're actually
> >> doing different things, but I wonder if the pull isn't just a
> >> hold-over.. In any case, the push-to-anon, which I hadn't seen
> >
> > No, the pull one doesn't do the main repository - that one pulls
> > "third party" repositories in as mirrors, such as e.g. Bucardo. That
> > one was, btw, broken for several months and nobody noticed - so that's
> > clearly a less important one :)
> >
> >> initially (looking at the pull side instead of the push side), does run
> >> once a minute, though it looks like there's a hook mechanism which
> >> would allow us to trigger the webserver to do a pull when a commit
> >> happens and would still be better than a once-a-minute cronjob.
> >
> > The one that deals with the main repo mirroring runs once per minute.
> > The commit trigger only drops a file in the filesystem so the cronjob
> > knows there is something to do. So that we don't risk holding up the
> > actual commit in case there is a problem with the anonymous mirror
> > server.
> >
> >
> >> Even with that, however, the concern raised was that the gitweb perl
> >> script is quite expensive to run for every request, hence the reason for
> >> doing the cacheing. I've lowered the varnish cacheing to a 5m ttl and
> >> 15m grace and I'll keep an eye on it.
> >
> > That's fixing the wrong problem. The cache is supposed to be
> > automatically purged whenever the push happens, to make sure that the
> > data never gets more than maybe a second stale. So clearly something
> > in that is not working - I didn't specifically test it with the
> > postgres.git repository, but I tested it with a couple of other ones
> > where it worked fine. But maybe we did something special with the main
> > one...
>
> Actually, looking closer, I'm seeing a failure when it actually tries the push:
>
> To ssh://git(at)git(dot)postgresql(dot)org/postgresql.git
> ! [rejected] master -> master (non-fast-forward)
>
>
> So the problem might have nothing at all to do with the cacheing.
>
> AFAICT, the three missing commits are materialized views,
> accidentally committed .orig file and \l support.
>
> But. The *anonymous* repository also has:
> bc61878682051678ade5f59da7bfd90ab72ce13b Fix
> map_sql_value_to_xml_value() to treat domains like their base types.
>
> This patch is *not* in the master repository, it's only in anonymous.
> (The object is in the repository, but it's not part of any branch)
>
> How the hell did *that* happen?
>
> The master repo has:
> commit 5db5974c692b0fc68e7608dd85a6b4e6173a0f28
> Author: Peter Eisentraut <peter_e(at)gmx(dot)net>
>
> psql: Let \l accept a pattern
>
> commit d63977eea3ab18fdec05e370b633d10b9fd20179
> Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>
>
> Remove accidentally-committed .orig file.
>
> commit 3bf3ab8c563699138be02f9dc305b7b77a724307
> Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>
>
> Add a materialized view relations.
>
> commit b15a6da29217b14f02895af1d9271e84415a91ae
> Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>
> Get rid of any toast table when converting a table to a view.
>
>
> And the anonymous one has:
> commit bc61878682051678ade5f59da7bfd90ab72ce13b
> Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>
> Fix map_sql_value_to_xml_value() to treat domains like their base types.
>
> commit 3bf3ab8c563699138be02f9dc305b7b77a724307
> Author: Kevin Grittner <kgrittn(at)postgresql(dot)org>
>
> Add a materialized view relations.
>
> commit b15a6da29217b14f02895af1d9271e84415a91ae
> Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>
> Get rid of any toast table when converting a table to a view.
>
>
>
> Does anybody have an explanation for that? Did someone do a force-push
> on the master repository, overwriting some old history?

This really looks like Kevin forced a push.

Could you show
git show --pretty=raw d63977eea3ab18fdec05e370b633d10b9fd20179
git show --pretty=raw bc61878682051678ade5f59da7bfd90ab72ce13b
git show --pretty=raw 5db5974c692b0fc68e7608dd85a6b4e6173a0f28
?
I don't have all the commits here...

The raw format will explicitly tell which parents are referenced in the
commits...

> We don't explicitly forbid this on the master repo, since we expect
> committers to know how things work.. Maybe we need to do that, and
> manually turn it off in case someone actually *needs* to do a non fast
> forward push? But either way, it would be good to actually know how
> tihs happened...

If you allow it the mirroring script should always force a push,
otherwise it will stop pushing in those cases.

I think the most realistic way to resolve this is to push the current
anongit state to master and cherry-pick
bc61878682051678ade5f59da7bfd90ab72ce13b ontop it. That way only
committers need to deal with rebasing their branches and not everyone
else.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services