Re: patch to allow disable of WAL recycling

There have been quite a few comments since last week, so at this point I am
uncertain how to proceed with this change. I don't think I saw anything
concrete in the recent emails that I can act upon.

I would like to respond to the comment about trying to "self-tune" the
behavior based on inferences made about caching during setup. I can't speak
for many other filesystems, but for ZFS, the ARC size is not fixed and will
vary based on the memory demands against the machine. Also, what files are
cached will vary based upon the workloads running on the machine. Thus, I
do not think there is a valid way to make inferences about future caching
behavior based upon a point-in-time observation.

I am still happy to update the man pages to explain the new tunable better
if that is acceptable.

Thanks,
Jerry

On Sun, Jul 15, 2018 at 6:32 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Thu, Jul 5, 2018 at 4:39 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> > This is formulated *WAY* too positive. It'll have dramatic *NEGATIVE*
> > performance impact of non COW filesystems, and very likely even negative
> > impacts in a number of COWed scenarios (when there's enough memory to
> > keep all WAL files in memory).
> >
> > I still think that fixing this another way would be preferrable. This'll
> > be too much of a magic knob that depends on the fs, hardware and
> > workload.
>
> I tend to agree with you, but unless we have a pretty good idea what
> that other way would be, I think we should probably accept the patch.
>
> Could we somehow make this self-tuning? On any given
> filesystem/hardware/workload, either creating a new 16MB file is
> faster, or recycling an old file is faster. If the old file is still
> cached, recycling it figures to win on almost any hardware. If not,
> it seems like something of a toss-up. I suppose we could try to keep
> a running average of how long it is taking us to recycle WAL files and
> how long it is taking us to create new ones; if we do each one of
> those things at least sometimes, then we'll eventually get an idea of
> which one is quicker. But it's not clear to me that such data would
> be very reliable unless we tried to make sure that we tried both
> things fairly regularly under circumstances where we could have chosen
> to do the other one.
>
> I think part of the problem here is that whether a WAL segment is
> likely to be cached depends on a host of factors which we don't track
> very carefully, like whether it's been streamed or decoded recently.
> If we knew when that a particular WAL segment hadn't been accessed for
> any purpose in 10+ minutes, it would probably be fairly safe to guess
> that it's no longer in cache; if we knew that it had been accessed <15
> seconds ago, that it is probably still in cache. But we have no idea.
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>