WAL partition ran out of space, pg_subtrans & pg_clog partially written...

This is an FYI re: a bug I ran across.

Background: RHEL 6 & ext4. PGDATA, a table space, and WAL logs are all on their own partitions.

The WAL partition filled up (wal_keep_segments was changed, but Pg hadn't been restarted), and a write happened, and it appears to have resulted in an index page being corrupt. REINDEX'ing the table didn't work with the following error:

> WARNING: concurrent delete in progress within table "tblA"

> WARNING: concurrent delete in progress within table "tblA"
> ERROR: could not access status of transaction 86081816
> DETAIL: Could not read from file "pg_subtrans/0521" at offset 131072: Success.

pg_subtrans/0521 was a 90112 byte file and pg_clog/0052 was 24576 bytes in size.

I haven't dug in to the code, but I did see the commit message for pgsql/src/backend/access/transam/slru.c 1.23.4.2, which I'm guessing is related. My WAG is that there's an assumption someplace that pg_subtrans and pg_clog are on the same partition as pg_xlog, or that creation of files in pg_subtrans and pg_clog will either absolutely succeed or absolutely fail. It could also be that Linux reported back a successful write(2), but it didn't actually have the space available (ext4).

Anyway, after extending pg_subtrans/0521 w/ zeros to its proper 256KB size, I was able to REINDEX the table, but there was a stream of WARNINGs about "concurrent inserts and deletes" that I didn't dig in to. Upon learning the WAL files were removed as a temporary solution to the space problem, I opted to dump, re-initdb, and load the data, which worked without any errors or warnings being reported.

I've saved the data if there are pointed questions about their contents.

-sc

--
Sean Chittenden
sean(at)chittenden(dot)org