Re: pg_basebackup: Allow use of arbitrary compression program

From: Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
To: Michael Harris <harmic(at)gmail(dot)com>
Cc: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: pg_basebackup: Allow use of arbitrary compression program
Date: 2017-04-07 16:12:29
Message-ID: CAMkU=1wp3ftSiwkq8Q7y-c03DWc82ZrjPQsVgQ_8TaMg7BeF6g@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Thu, Apr 6, 2017 at 7:04 PM, Michael Harris <harmic(at)gmail(dot)com> wrote:

> Hello,
>
> Back in pg 9.2, we hacked a copy of pg_basebackup to add a command
> line option which would allow the user to specify an arbitrary
> external program (potentially including arguments) to be used to
> compress the tar backup.
>
> Our motivation was to be able to use pigz (parallel gzip
> implementation) to speed up the compression. It also allows using
> tools like bzip2, xz, etc instead of the inbuilt zlib.
>
> I never ended up submitting that upstream, but now it looks like I
> will have to repeat the exercise for 9.6, so I was wondering if such a
> feature would be welcomed.
>

I would welcome it. I would really like to be able to use parallel pigz
and pxz.

You can stream the data into a compression tool of your choice as long as
you use tar mode and specify '-D -', but that is incompatible with table
spaces, and with xlog streaming, and so is not a very good solution.

Cheers,

Jeff

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Heikki Linnakangas 2017-04-07 16:20:55 Re: Duplicate assignment in Unicode/convutils.pm
Previous Message Yorick Peterse 2017-04-07 15:57:01 Re: [PATCH] Document the order of changing certain settings when using hot-standby servers