Re: WIP/PoC for parallel backup

Greetings,

* Ahsan Hadi (ahsan(dot)hadi(at)gmail(dot)com) wrote:
> On Fri, 23 Aug 2019 at 10:26 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> > I would expect you to quickly want to support compression on the server
> > side, before the data is sent across the network, and possibly
> > encryption, and so it'd likely make sense to just have independent
> > processes and connections through which to do that.
>
> It would be interesting to see the benefits of compression (before the data
> is transferred over the network) on top of parallelism. Since there is also
> some overhead associated with performing the compression. I agree with your
> suggestion of trying to add parallelism first and then try compression
> before the data is sent across the network.

You're welcome to take a look at pgbackrest for insight and to play with
regarding compression-before-transfer, how best to split up the files
and order them, encryption, et al. We've put quite a bit of effort into
figuring all of that out.

Thanks!

Stephen