Re: More efficient build farm animal wakeup?

On Sat, Nov 19, 2022 at 4:13 AM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:

> Hi,
>
> Is there a way to find out about new git commits that is more
> efficient and timely than running N git fetches or whatever every
> minute in a cron job? Maybe some kind of long polling where you send
> an HTTP request that says "I think the tips of branches x, y, z are at
> 111, 222, 333" and the server responds when that ceases to be true?
>

I'm not aware of any such thing standardized for git, but it wouldn't be
hard to build one for that (I'm talking primarily about the server side
here, not how to integrate that into the buildfarm side of things).

We could also set something up whereby we could fire off webhooks when
branches change (easy enough for registered servers in the buildfarm as we
can easily avoid abuse there -- it would take more work to make something
like that a public service, due to the risk of abuse). But that may
actually be worse off, since I bet a lot of buildfarm animals (most even?)
are probably sitting behind a NAT gateway of some kind, meaning consuming
webhooks is hard.

I did something similar for how we did things on borka (using some internal
pginfra webhooks that are not available to the public at this time), but I
had to revert that because of issues with concurrent buildfarm runs in the
environment that we had set up. But we are using it for the snapshots docs
builder, to make sure the website for that gets updated immediately after a
commit on master. But the principle definitely work.

Another thing to consider would be that something like this would cause all
buildfarm clients to start git pull:ing down changes at more or less
exactly the same time. Though in total that would probably still mean a lot
less load than those that "git pul" very frequently today, it could
potentially lead to some nets with lots of bf clients experiencing some
level of bandwidth filling or something. Could probably be solved pretty
easily with a random delay (which doesn't have to be long, as for most git
pulls it will be a very quick operation), just something that's worth
considering.

tl,tr; it's not there now, but yes if we can find a smart way for th ebf
clients to consume it, it is something we could build and deploy fairly
easily.

--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/>
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>