Re: [PATCH] COPY .. COMPRESSED

On 1/13/13 9:16 PM, Stephen Frost wrote:
> On top of this I plan to submit a trivial patch to add support for
> this to file_fdw, allowing creation of FDW tables which operate
> directly on compressed files (including CSVs, which is what I need
> this patch for).
>
> I've also begun working on a patch to allow this capability to be used
> through pg_dump/pg_restore which would reduce the bandwidth used
> between the client and the server for backups and restores. Ideally,
> one would also be able to use custom format dumps, with compression,
> even if the client-side pg_dump/pg_restore wasn't compiled with zlib
> support.

I think a problem is that this code is now serving such different uses.

Operating on compressed files transparently in file_fdw is obviously
useful, but why only gzip? The gold standard is GNU tar, which can
operate on any compressed file in a variety of compression formats
without even having to specify an option.

Writing compressed COPY output files on the backend has limited uses, at
least none have been clearly explained, and the popen patch might
address those better.

Writing compressed COPY output on the frontend can already be done
differently.

Compression on the wire is a different debate and it probably shouldn't
be snuck in through this backdoor.

Putting compressed COPY output from the backend straight into a
compressed pg_dump file sounds interested, but this patch doesn't do
that yet, and I think there will be more issues to solve there.