RE: BUG #15460: Error while creating index or constraint

Well, I can test.
If you'll provide me with the call (incl flags) that is done on that moment to remove the directory I can see what is needed to make that fail and possibly how to circumvent that

-----Original Message-----
From: Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com>
Sent: vrijdag 2 november 2018 1:26
To: Peter Geoghegan <pg(at)bowt(dot)ie>
Cc: Paul van der Linden <paul(dot)vanderlinden(at)mapcreator(dot)eu>; PostgreSQL mailing lists <pgsql-bugs(at)lists(dot)postgresql(dot)org>; Heikki Linnakangas <hlinnaka(at)iki(dot)fi>
Subject: Re: BUG #15460: Error while creating index or constraint

On Wed, Oct 31, 2018 at 2:37 AM Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com> wrote:
> On Tue, Oct 30, 2018 at 4:15 AM Peter Geoghegan <pg(at)bowt(dot)ie> wrote:
> > On Mon, Oct 29, 2018 at 2:11 PM Peter Geoghegan <pg(at)bowt(dot)ie> wrote:
> > > This first line looks like it might be interesting:
> > >
> > > LOG: could not rmdir directory
> > > "base/pgsql_tmp/pgsql_tmp5088.0.sharedfileset": Directory not
> > > empty
> > > ERROR: could not determine size of temporary file "0"
> >
> > (Thomas Munro is CC'd here.)
> >
> > > I suppose that this could actually just be a result of the ERROR;
> > > the exact order isn't a reliable indicator of the sequence of
> > > events across processes (A useful log_line_prefix setting might
> > > clear this up if you collect the trace_sort output again).
> >
> > Hmm. So apparently Windows has a habit of setting an ENOTEMPTY
> > errcode when rmdir'ing a directory that somebody merely has a handle
> > to. It could just be that somebody has a Windows Explorer window
> > open -- you still get ENOTEMPTY [1]! Not sure if this is truly
> > relevant to the problem at hand, but it seems worth being aware of.
>
> We only try to remove the directory after removing everything in it,
> so that does seem to require a pretty strange explanation like that.
> I also wondered if a worker having a handle to an unlinked file (as
> permitted by the FILE_SHARE_DELETE flag we use) inside that directory
> could produce that effect. Perhaps RemovedDirectoryA[1] would be
> better than rmdir, since it "... marks a directory for deletion on
> close. Therefore, the directory is not removed until the last handle
> to the directory is closed." The documentation for rmdir[2] just says
> deprecated, and _rmdir[3] mentions ENOTEMPTY, but it says you get
> EACCES if someone has an open handle to the directory.

Is there a Windows developer who would like to experiment with this stuff -- perhaps with a small standalone program that tries to unlink a directory while it has another handle to the directory open -- and make a recommendation? Otherwise I'll try to do this with AppVeyor.
While having the directory open in Explorer.exe might seem a bit unusual for a production database, I bet there are other things that periodically go around opening and reading our directories: backup tools and SearchIndexer.exe presumably traverse the whole filesystem from time to time. By my reading, RemovedDirectoryA() is probably the right thing to defend against this, here and any other place we remove directories.

--
Thomas Munro
http://www.enterprisedb.com