Re: postgres=# VACUUM FULL pg_statistic => ERROR: missing chunk number 0 for toast value .. in pg_toast_2619

On Sat, May 19, 2018 at 11:24:57AM -0500, Justin Pryzby wrote:
> On Sat, May 19, 2018 at 11:08:23AM -0400, Tom Lane wrote:
> > Justin Pryzby <pryzby(at)telsasoft(dot)com> writes:
> > > I'll defer fixing this for awhile in case someone wants me to save a copy of
> > > the relation/toast/index. From last time, I recall this just needs the right
> > > combination of REINDEX/VACUUM/ANALYZE, and the only complication was me
> > > needing to realize the right combination of affected DB(s).
> >
> > If you could come up with such a sequence that causes the problem
> > reproducibly, that would be of huge interest, and probably lead to
> > a fix promptly. But I don't think that we can do much by looking
> > at the post-mortem state --- once the toast rows are gone, they're
> > gone, especially if the table's been vacuumed since.
>
> This is unlikely to allow reproducing it, but for sake of completeness here's a
> fuller log. I'll try to trigger on another DB.

Did not take long...

[pryzbyj(at)database ~]$ while :; do for db in `psql postgres -Atc "SELECT datname FROM pg_database WHERE datallowconn"`; do for t in pg_statistic pg_attrdef pg_constraint; do echo "$db.$t..."; PGOPTIONS=-cstatement_timeout='9s' psql $db -qc "VACUUM FULL $t"; done; done; done

...
postgres.pg_statistic...
postgres.pg_attrdef...
postgres.pg_constraint...
template1.pg_statistic...
template1.pg_attrdef...
template1.pg_constraint...
ts.pg_statistic...
ERROR: canceling statement due to statement timeout
ts.pg_attrdef...
ts.pg_constraint...
postgres.pg_statistic...
ERROR: missing chunk number 0 for toast value 3372855171 in pg_toast_2619

I'm running this again on another DB, but I wonder if that's enough for anyone
else to reproduce it with some consistency ? I think that took something like
10min before failing.

Justin