Re: pg_dump and large files - is this a problem?

Philip Warner writes:

> Yes, but there is no guarantee that off_t is implemented as such, nor would
> we be wise to assume so (most docs say explicitly not to do so).

I suspect you're reading old documents, which is why I asked what you
were referring to. In the '80s what you are saying would have been
best practice, no question: 64 bit type support was not common.

When talking of near-current systems with 64 bit off_t you are not
going to find one without support for 64 bit integral types.

> Again yes, but the problem is the same: we need a way of making the *value*
> of an off_t portable (not just assuming it's a int64). In general that
> involves knowing how to turn it into a more universal data type (eg. int64,
> or even a string).

So you need to know the size of off_t, which will be 32 bit or 64 bit,
and then you need routines to convert that to a portable representation.
The canonical solution is XDR, but I'm not sure that you want to bother
with it or if it has been extended universally to support 64 bit types.

If you limit the file sizes to 1GB (your less preferred option, I
know;-) then like the rest of the PostgreSQL code you can safely
assume that off_t fits into 32 bits and have a choice of functions
(XDR or ntohl() etc) to deal with them and ignore 64 bit off_t
issues altogether.

If you intend pg_dump files to be portable avoiding the use of large
files will be best. It also avoids issues on platforms such as HP-UX
where large file support is available, but it has to be enabled on a
per-filesystem basis. :-(

> Does the large file API have functions for representing
> the off_t values that is portable across architectures? And is the API also
> portable?

The large files API is a way to access large files from 32 bit
processes. It is reasonably portable, but is a red herring for
what you are wanting to do. (I'm not convinced I am understanding
what you're trying to do, but I have 'flu which is not helping. :-)

Regards,

Giles