Re: [PATCH] Lazy xid assingment V2

August Zajonc <augustz(at)augustz(dot)com> writes:
> I do have a question about jamming though. Will the system work if the
> file ended up stuck in this folder? Let's say the move destination has a
> duplicate file that conflicts, or permissions change under you, or disks
> fill.

Yeah, the move-the-file approach seems to introduce its own set of
failure modes, which hardly seems like something we want.

I had an idea this morning that might be useful: back off the strength
of what we try to guarantee. Specifically, does it matter if we leak a
file on crash, as long as it isn't occupying a lot of disk space?
(I suppose if you had enough crashes to accumulate many thousands of
leaked files, the directory entries would start to be a performance drag,
but if your DB crashes that much you have other problems.) This leads
to the idea that we don't really need to protect the open(O_CREAT) per
se. Rather, we can emit a WAL entry *after* successful creation of a
file, while it's still empty. This eliminates all the issues about
logging an action that might fail. The WAL entry would need to include
the relfilenode and the creating XID. Crash recovery would track these
until it saw the commit or abort or prepare record for the XID, and if
it didn't find any, would remove the file.

With this approach I think we'd not even need to force-fsync the WAL
entry; instead treat it like an async COMMIT record (pass its LSN to
the walwriter). Even in the absence of any subsequent WAL activity,
it would reach disk via the walwriter before the new file could be
filled to a size that would bother anyone. (If the new file is being
filled via WAL-logged insertions, then you can probably make even
stronger statements than that, but we do have operations like COPY
and CREATE INDEX that can fill a file with unlogged insertions.)
So the performance impact would be about nil.

I still don't think that this area is in urgent need of a fix, but
I wanted to get this idea into the archives.

regards, tom lane