Re: Confusing error message with too-large file in pg_basebackup

Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
> Guillaume Lelarge wrote:
>> Looking at the file's size is probably a better idea. As far as I know,
>> PostgreSQL doesn't create files bigger than 1GB, except for log files. I'm
>> not sure about this but I guess pg_basebackup doesn't ship log files. So,
>> looking at the size would work.

> Hmm, so we let configure --with-segsize to change the file size. The
> configure help says that the limit should be "less than your OS' limit
> on file size". We don't warn them that this could cause backup
> problems later on. Should we add a blurb about that somewhere?

Actually ... why don't we get rid of the limit? wikipedia's entry on
tar format says

... only 11 octal digits can be stored. This gives a maximum file size
of 8 gigabytes on archived files. To overcome this limitation, star in
2001 introduced a base-256 coding that is indicated by setting the
high-order bit of the leftmost byte of a numeric field. GNU-tar and
BSD-tar followed this idea.

If that extension is as widespread as this suggests, then following it
when we have a file > 8GB seems like a better answer than failing
entirely. If you try to read the dump with an old tar program, old
pg_restore, etc, it might fail ... but are you really worse off than
if you couldn't make the dump at all?

regards, tom lane