On markers of changed data

From: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Cc: root(at)simply(dot)name
Subject: On markers of changed data
Date: 2017-10-06 12:25:16
Message-ID: F0627DEB-7D0D-429B-97A9-D321450365B4@yandex-team.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi, hackers!

Currently I'm working on page-level incremental backups using WAL-G codebase[0]. And I have two questions that I cannot resolve myself.

Incremental backup is a set of changes, that should be applied over preexisting backup. I use page LSN to understand should page be backup`ed or not.

Question 1. FSM and VM.
As you can see here [1] FSM and VM files are exempt from incremental tracking and are backuped as whole files. I've done it this way, because sanity checks [2] of page headers have indicated a lot of "invalid" pages in FSM and VM files. But seems like in some pages headers are valid with sane LSNs.
Can I use LSNs as history marker on FSM and VM pages? On 1Tb backup I get like 150Mb of FSM+VM, and it's kind of a lot.

Question 2. File dates.
Is it safe to use file modification time to track that file were changes since previous backup? If the file has date before start of previous backup I just add it to "skip list" [3].
I have assumption: every time file is changes in filesystem, it's modification date is updated to higher value.
Is this assumption valid for most of used platforms and filesystems? Or can I check this "capacity" of FS?

Thank you for your attention. I'll be glad to receive any information\pointers on this matter.

Best regards, Andrey Borodin, Yandex.

[0] https://github.com/wal-g/wal-g/pull/29
[1] https://github.com/wal-g/wal-g/pull/29/files#diff-d77406e827f5f947d4d4a1e6d76c1f4eR114
[2] https://github.com/wal-g/wal-g/pull/29/files#diff-d77406e827f5f947d4d4a1e6d76c1f4eR50
[3] https://github.com/wal-g/wal-g/pull/29/files#diff-f5c8f0067297f98eb5acc6e2c6b1b234R87

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Petr Jelinek 2017-10-06 12:29:17 Re: Issue with logical replication: MyPgXact->xmin already is valid
Previous Message Michael Paquier 2017-10-06 12:21:03 Re: [PATCH] Assert that the correct locks are held when calling PageGetLSN()