Re: More efficient build farm animal wakeup?

On 2022-11-21 Mo 16:20, Magnus Hagander wrote:
> n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > The buildfarm server now creates a companion to
> branches_of_interest.txt
> > called branches_of_interest.json which looks like this:
>
> ... okay ...
>
>
> Yeah, it's not as efficient as something like long polling or web
> sockets, but it is most definitely a lot simpler!
>
> If we're going to have a lot of animals do pulls of this file every
> minute or more, it's certainly a lot better to pull this small file
> than to make multiple git calls.
>
> It could trivially be made even more efficient by making the request
> with either a If-None-Match or If-Modified-Since. While it's still
> small, that cuts the size approximately in half, and would allow you
> to skip even more processing if nothing has changed.

I'll look at that.

>
>
> > It updates this every time it does a git fetch, currently every
> 5 minutes.
>
> That up-to-five-minute delay, on top of whatever cronjob delay one has
> on one's animals, seems kind of sad.  I've gotten kind of spoiled
> maybe
> by seeing first buildfarm results typically within 15 minutes of a
> push.
> But if we're trying to improve matters in this area, this doesn't seem
> like quite the way to go.
>
> But it does seem like this eliminates one expense.  Now that you have
> that bit, maybe we could arrange a webhook or something that allows
> branches_of_interest.json to get updated immediately after a push?
>
>
> Webhooks are definitely a lot easier to implement in between our
> servers yeah, so that shouldn't be too hard. We could use the same
> hooks that we use for borka to build the docs, but have it just run
> whatever script it is the buildfarm needs. I assume it's just
> something trivial to run there, Andrew?

Yes, I think much better between servers. Currently the cron job looks
something like this:

*/5 * * * * cd $HOME/postgresql.git && git fetch -q &&
$HOME/website/bin/branches_of_interest.pl

That script is what sets up the json files.

I know nothing about git webhooks though, someone will have to point me
in the right direction.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com