Re: Windows pg_basebackup unable to create >2GB pg_wal.tar tarballs ("could not close file: Invalid argument" when creating pg_wal.tar of size ~ 2^31 bytes)

Hi Thomas!

On Thu, Nov 21, 2024 at 2:38 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:

> On Thu, Nov 21, 2024 at 11:44 PM Jakub Wartak
> <jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
> > This literally looks like something like off_t/size_t would be limited
> to 2^31 somewhere.
>
> off_t is 32 bits on Windows. I'd be quite suspicious of the
> arithmetic involving 'currpos'. What happens if you change all the
> off_t in walmethods.c/.h to pgoff_t? The lseek() is suspicious too,
> and might need to be redirected to _lseeki64().
>

Thank you for responding and suggesting this. First one was not enough, on
REL_15_STABLE with just pgoff_t i still got the same error:

C:\compiledpg\bin>pg_basebackup.exe -U postgres -D "c:\backup6" -F t -P -X
stream -c fast --compress=none --create-slot --slot=slot10
Password:
pg_basebackup: error: could not close file "0000000100000008000000AE":
Invalid argument
pg_basebackup: error: background process terminated unexpectedly

but with attached _lseeki64 dirty patch I've got success and resulting
tarball greater than 2^31 too:

C:\compiledpg\bin>pg_basebackup.exe -U postgres -D "c:\backup7" -F t -P -X
stream -c fast --compress=none --create-slot --slot=slot11
Password:
18134035/18134035 kB (100%), 1/1 tablespace
C:\compiledpg\bin>
C:\compiledpg\bin>dir c:\backup7\pg_wal.tar
[..]
11/22/2024 10:37 AM 4,026,778,112 pg_wal.tar

PoC patch is attached and I'll register CFentry to see how the tests will
go (all of this MSVC/MinGW stuff is frightening to me, of course this will
fail on non-Win32...).

-J.