Re: Confusing error message with too-large file in pg_basebackup

I wrote:
> Actually ... why don't we get rid of the limit? wikipedia's entry on
> tar format says
> ... only 11 octal digits can be stored. This gives a maximum file size
> of 8 gigabytes on archived files. To overcome this limitation, star in
> 2001 introduced a base-256 coding that is indicated by setting the
> high-order bit of the leftmost byte of a numeric field. GNU-tar and
> BSD-tar followed this idea.
> If that extension is as widespread as this suggests, then following it
> when we have a file > 8GB seems like a better answer than failing
> entirely. If you try to read the dump with an old tar program, old
> pg_restore, etc, it might fail ... but are you really worse off than
> if you couldn't make the dump at all?

I looked into the GNU tar sources and confirmed that gtar supports this
concept. (It looks from the GNU sources like they've supported it for
a *really long time*, like since the 90's, in which case wikipedia's
credit to "star" for the idea is full of it.)

Hence, I propose something like the attached (WIP, has no doc
corrections). I've done simple testing on the pg_dump/pg_restore code
path, but not on basebackup --- anyone want to test that?

I'm not sure whether we should treat this as a back-patchable bug fix
or a new feature for HEAD only. If we don't back-patch it, there are
in any case several bugs here that we must fix. In particular, the
existing coding in ReceiveTarFile:

size_t filesz = 0;
...
sscanf(&tarhdr[124], "%11o", (unsigned int *) &filesz);

is utterly, absolutely, completely broken; it'll fail grossly on
any 64-bit big-endian hardware. There are other places with misplaced
faith that "unsigned long" is at least as wide as size_t.

Comments?

regards, tom lane