Re: There's something rotten in the PG infrastructure

On 05/17/2015 01:57 AM, Tom Lane wrote:
> Last night, and again tonight, I noticed weird delays in the buildfarm's
> response to commits. Last night I supposed that the buildfarm server was
> down, and chided Andrew about it --- but now it seems the blame is
> elsewhere, at least in part. Facts (all times GMT-4 unless noted):
>
> * According to git, Peter committed fab6ca23eaf114d1ae12377c7f5c8c952b5e0159
> at Sun, 17 May 2015 03:35:29 +0000 (23:35 -0400). This is not too far
> from reality, because according to my mail logs, the commit message came
> through from pgsql-committers at Sat May 16 23:36:19 2015.
>
> * However, neither of my buildfarm critters noticed anything had happened
> for about an hour and a half. prairiedog lit off with a run around 0:55
> Sunday, dromedary around 1:10. (Both of them check every 20 minutes, not
> on the same schedule.) dromedary's run finished around 1:25.
>
> * As of right now, 1:56 AM, the buildfarm status page is not showing an
> update from dromedary, or indeed any other machine for nearly four hours.
> There should have been a lot of updates by now.
>
> It looks to me like not only is the buildfarm server wedged, but there's
> something wrong with pushing from gitmaster to the mirror used by
> buildfarm members. It's not continuous, because stuff pushed during the
> day Saturday seemed to get acted on promptly, but what's happening now?
>
>

This is my fault, I'm sorry. JD warned me a little while ago that we
were running out of database disk space, and I said I'd do something
about it, and let it slip.

I'm taking some emergency measures to relieve the situation.

cheers

andrew