Re: Complete data erasure

Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com> writes:
> I think it depends how exactly it's implemented. As Tom pointed out in
> his message [1], we can't do the erasure itself in the post-commit is
> not being able to handle errors. But if the files are renamed durably,
> and the erasure happens in a separate process, that could be OK. The
> COMMIT may wayt for it or not, that's mostly irrelevant I think.

How is requiring a file rename to be completed post-commit any less
problematic than the other way? You still have a non-negligible
chance of failure.

>> 1. Writes a commit WAL record, finalizing the system catalog change.
>> 2. Puts the data files in the trash bin or renames them.
>> 3. Erase the file content and delete the file. This could take a long time.
>> 4. COMMIT replies success to the client.

> I don't think the COMMIT has to wait for (3) - it might, of course, but
> for some use cases it may be better to just commit and leave the
> bgworker do the work. And then allow checking if it completed.

This doesn't seem like a path that will lead to success. The fundamental
point here is that COMMIT has to be an atomic action --- or if it isn't,
failure partway through has to lead to a database crash & restart, which
isn't very pleasant, especially if WAL replay of the commit after the
restart re-encounters the same error.

Up to now, we've sort of looked the other way with respect to failures
of file unlinks post-commit, reasoning that the worst that will happen
is disk space leakage from no-longer-referenced files that we failed to
unlink. (Which is bad, certainly, but not catastrophic; it's immaterial
to database semantics.) This patch basically needs to raise the level of
guarantee that exists in this area, or it won't do what it says on the
tin. But I've not seen any indication that we know how to do that in a
workable way.

regards, tom lane