Re: catalog corruption bug

Here is some additional information that I have managed to gather today
regarding this. It is not really what causes it, so much as what does
not.

I removed all plperl from the loading processes. I did a VACUUM FULL
ANALYZE, and then I reindexed everything in the database (Including
starting the backend in standalone mode and running REINDEX SYSTEM
dbname). They still failed.

So it is apparently not that plperl issue which was being discussed
earlier.

Also, what I said about the corruption not having persisted into other
backends was not quite correct. It was leaving behind types in pg_type
which were in some of the pg_temp* schemas which corresponded to some of
the temp tables. But I took those out and still had issues (slightly
different).

Here is some interesting stuff too. I just stopped my processes to start
up a batch again to copy the error message I got now, but before doing so
I was doing a VACUUM FULL ANALYZE VERBOSE so I could hopefully start from
a relatively clean state. I got a few warnings I don't remember seeing
before.

INFO: vacuuming "pg_catalog.pg_shdepend"
INFO: "pg_shdepend": found 108 removable, 440 nonremovable row versions
in 15 p
ages
DETAIL: 0 dead row versions cannot be removed yet.
Nonremovable row versions range from 53 to 53 bytes long.
There were 1492 unused item pointers.
Total free space (including removable row versions) is 89780 bytes.
7 pages are or will become empty, including 0 at the end of the table.
12 pages containing 89744 free bytes are potential move destinations.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "pg_shdepend_depender_index" now contains 448 row versions in
33 pages
DETAIL: 108 index row versions were removed.
23 index pages have been deleted, 23 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.10 sec.
WARNING: index "pg_shdepend_depender_index" contains 448 row versions,
but table contains 440 row versions
HINT: Rebuild the index with REINDEX.
INFO: index "pg_shdepend_reference_index" now contains 448 row versions
in 12 pages
DETAIL: 108 index row versions were removed.
3 index pages have been deleted, 3 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
WARNING: index "pg_shdepend_reference_index" contains 448 row versions,
but table contains 440 row versions
HINT: Rebuild the index with REINDEX.
INFO: "pg_shdepend": moved 4 row versions, truncated 15 to 4 pages
DETAIL: CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "pg_shdepend_depender_index" now contains 448 row versions in
33 pages
DETAIL: 4 index row versions were removed.
23 index pages have been deleted, 23 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
WARNING: index "pg_shdepend_depender_index" contains 448 row versions,
but table contains 440 row versions
HINT: Rebuild the index with REINDEX.
INFO: index "pg_shdepend_reference_index" now contains 448 row versions
in 12 pages
DETAIL: 4 index row versions were removed.
4 index pages have been deleted, 4 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
WARNING: index "pg_shdepend_reference_index" contains 448 row versions,
but table contains 440 row versions
HINT: Rebuild the index with REINDEX.
INFO: analyzing "pg_catalog.pg_shdepend"
INFO: "pg_shdepend": scanned 4 of 4 pages, containing 440 live rows and 0
dead rows; 440 rows in sample, 440 estimated total rows

Similar for pg_type, there being 248 index row versions vs 244 row
versions in the table.

1631 vs 1619 in pg_attribute
95 vs 94 in pg_index

Looks like it may be time to start a standalone backend and REINDEX
again...

--
Don't smoke the next cigarette. Repeat.