Re: [PATCH] COPY .. COMPRESSED

On Tue, Jan 15, 2013 at 3:37 PM, Christopher Browne <cbbrowne(at)gmail(dot)com> wrote:
> That points towards a fix that involves having a set of non-arbitrary commands
> that we allow plain users to use.
>
> Hmm. There's an interesting thought...
>
> How about having a "pg_filters" table in pg_catalog which allows capturing
> labels and names of known-to-be-safe binary filters:
>
> insert into pg_filters (label, location)
> values
> ('zcat', '/usr/bin/zcat'),
> ('bzip2', '/usr/bin/bzip2'),
> ('bunzip2', '/usr/bin/bunzip2');
>
> And then having some capability to grant permissions to roles to use
> these filters.

I suspect that's going to be less efficient than using a compression
library that's linked into the backend, because you have to copy all
the data through the kernel to another process and back. And it's
certainly a lot more complex.

If it greatly broadened the applicability of this feature I might
think it was worthwhile, but I can't see that it does. I suspect that
supporting zlib, which we already linked against, would cater to
something well upwards of 90% of the use cases here. Sure, there are
other things, but zlib is very widely used and bzip2 IME is far too
slow to be taken seriously for this kind of application. The
additional space savings that you get for the additional CPU
investment is typically small, and if you really need it, having to
un-gzip and re-bzip2 on the client is always an option. If you're
using bzip2 you obviously have CPU time to burn.

At any rate, I think it would be good to avoid letting our desire for
infinite flexibility get in the way of doing something useful.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company