Re: There's something rotten in the PG infrastructure

On May 17, 2015 08:30, "Stephen Frost" <sfrost(at)snowman(dot)net> wrote:
>
> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> > I wrote:
> > > * However, neither of my buildfarm critters noticed anything had
happened
> > > for about an hour and a half. prairiedog lit off with a run around
0:55
> > > Sunday, dromedary around 1:10. (Both of them check every 20 minutes,
not
> > > on the same schedule.) dromedary's run finished around 1:25.
> >
> > > * As of right now, 1:56 AM, the buildfarm status page is not showing
an
> > > update from dromedary, or indeed any other machine for nearly four
hours.
> > > There should have been a lot of updates by now.
> >
> > Hmm ... actually, it looks like both prairiedog and dromedary are
> > rebuilding repeatedly, which is odd because certainly nothing is
happening
> > on gitmaster. But the run start times quoted above probably only
> > represent the runs that were active when I got annoyed enough to go look
> > at what was happening.
>
> I've been poking around and I don't see any obvious issues on either
> gitmaster or git.p.o. The cronjobs appear to be running. Not saying
> there isn't an issue but it's at least less than obvious if there is.
> Further, these systems are still on wheezy (many other boxes have been
> upgraded to jessie at this point), so nothing much has changed on them
> in quite some time. It's possible that there is an issue due to jessie
> being on the host server, but hopefully not.
>

We've seen some emails from our buildfarm client as well, where the
buildfarm complains about database errors. I'm guessing some intermittent
errors there, since the site itself works. That's also the reason some
snapshots on the ftp site are currently slightly out of date - they are
uploaded at the end of successful buildfarm runs..

So Afaict, the problem seems to be in the buildfarm, not in the pg
infrastructure.

The quick check for the replication between the git servers is to just look
at the gitweb interface. That one reads directly from the same got
repository as the buildfarn clients would.

/Magnus