Re: finding changed blocks using WAL scanning

On Mon, Apr 22, 2019 at 11:48 AM Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> My point is that most WAL archive tools will order and remove files
> based on their lexical ordering, so if you put the start first, the file
> will normally be removed when it should be kept, e.g., if you have WAL
> files like:
>
> 000000010000000000000001
> 000000010000000000000002
> 000000010000000000000003
> 000000010000000000000004
> 000000010000000000000005
>
> putting the start first and archiving some wal would lead to:
>
> 000000010000000000000001-000000010000000000000004.modblock
> 000000010000000000000003
> 000000010000000000000004
> 000000010000000000000005
>
> We removed 1 and 2, but kept the modblock file, which looks out of
> order. Having the end at the start would have:
>
> 000000010000000000000003
> 000000010000000000000004
> 000000010000000000000004-000000010000000000000001.modblock
> 000000010000000000000005
>
> My point is that you would normally only remove the modblock file when 4
> is removed because this modblock files is useful for incremental backups
> from base backups that happened between 1 and 4.

That's an interesting point. On the other hand, I think it would be
typical to want the master to retain .modblock files for much longer
than it retains WAL segments, and in my design, the WAL archive
wouldn't see those files at all; they'd be stored on the master. I
was actually thinking that they should possibly be stored in a
separate directory to avoid confusion.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company