Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Adam Sjøgren <adsj(at)novozymes(dot)com> wrote:

> Meanwhile, I can report that I have upgraded from 9.3.14 to 9.3.17 and
> the errors keep appearing the log.

Just a quick update with more observations:

All the errors in the postgres.log from one of the tables are triggered
by a stored procedure that gathers data to put in a field used for full
text search - this stored procedure is called by a before update trigger
on the table. We have only seen it in the log, but not been able to
reproduce it.

We have, however, now got a row in the other big table where we can get
the error just by running a SELECT * on the row, in psql:

user(at)server db=# select * from ourschema.table_a where id = 6121931;
ERROR: unexpected chunk number 0 (expected 1) for toast value 339846807 in pg_toast_10919630
user(at)server db=#

Which is both nice - we can show the error on demand - but also more
worrying, I guess, because that means the problem is "on disk".

Running this in a stored procedure over the record in question:

> SELECT *
> INTO rec
> FROM table_a where id = badid;
> detoast := substr(rec.fts::text,1,2000);
> exception
> when others then
> raise notice 'data for table_a id: % is corrupt', badid;
> continue;

also shows the error:

user(at)server db=# SELECT ourschema.check_sequence(6121931, 6121931);
NOTICE: data for table_a id: 6121931 is corrupt
check_sequence
----------------

(1 row)

We are running this over the entire (160M+ row) table now, to see if any
other rows are affected.

So, we can reproduce the error message, but we can't reproduce the
problem from scratch.

Any ideas on what to look at, given a non-transient problem-row?

Our next step will be to try to switch to 9.3.17 with
6c243f90ab6904f27fa990f1f3261e1d09a11853 reverted as suggested by Alvaro
Herrera last week.

Best regards,

Adam

--
"Lägg ditt liv i min hand Adam Sjøgren
Sälj din själ till ett band" adsj(at)novozymes(dot)com