Re: finding changed blocks using WAL scanning

On Thu, Apr 18, 2019 at 03:51:14PM -0400, Robert Haas wrote:
> I was thinking that a dedicated background worker would be a good
> option, but Stephen Frost seems concerned (over on the other thread)
> about how much load that would generate. That never really occurred
> to me as a serious issue and I suspect for many people it wouldn't be,
> but there might be some.

WAL segment size can go up to 1GB, and this does not depend on the
compilation anymore. So scanning a very large segment is not going to
be free. I think that the performance concerns of Stephen are legit
as now we have on the WAL partitions sequential read and write
patterns.

> It's cool that you have a command-line tool that does this as well.
> Over there, it was also discussed that we might want to have both a
> command-line tool and a background worker. I think, though, that we
> would want to get the output in some kind of compressed binary format,
> rather than text. e.g.

If you want to tweak it the way you want, please feel free to reuse
it for any patch submitted to upstream. Reshaping or redirecting the
data is not a big issue once the basics with the WAL reader are in
place.
--
Michael