Re: skipping pg_log in basebackup (was Re: pg_basebackup and pg_stat_tmp directory)

On Thu, Jun 11, 2015 at 2:20 PM, Abhijit Menon-Sen <ams(at)2ndquadrant(dot)com> wrote:
> At 2015-06-10 13:22:27 -0400, robertmhaas(at)gmail(dot)com wrote:
>>
>> I'm not clear on which of these options you are voting for:
>>
>> (1) include pg_log in pg_basebackup as we do currently
>> (2) exclude it
>> (3) add a switch controlling whether or not it gets excluded
>>
>> I can live with (3), but I bet most people want (2).
>
> Thanks for spelling out the options.
>
> I strongly prefer (2), but I could live with (3) if it were done as a
> GUC setting. (And if that's what we decide to do, I'm willing to write
> up the patch.)
>
> Whether or not it's a good idea to let one's logfiles grow to >8GB, the
> fact that doing so breaks base backups means that being able to exclude
> pg_log *somehow* is more of a necessity than personal preference.
>
> On the other hand, I don't like the idea of doing (3) by adding command
> line arguments to pg_basebackup and adding a new option to the command.
> I don't think that level of "flexibility" is justified; it would also
> make it easier to end up with a broken base backup (by inadvertently
> excluding more than you meant to).

After spending the night thinking about that, honestly, I think that
we should go with (2) and keep the base backup as light-weight as
possible and not bother about a GUC. (3) would need some extra
intelligence to decide if some files can be skipped or not. Imagine
for example --skip-files=global/pg_control or --skip-files=pg_clog
(because it *is* a log file with much data), that would just corrupt
silently your backup, but I guess that it is what you had in mind. In
any case (3) is not worth the maintenance burden because we would need
to update the things to filter each time a new important folder is
added in PGDATA by a patch.
--
Michael