Re: Unkillable processes creating millions of tiny temp files

Jerry Sievers <gsievers19(at)comcast(dot)net> writes:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>> The "unkillable" aspect is odd, but I wonder if that's just a red
>> herring. A query that's generated lots of temp files will try to
>> clean them up at termination, so maybe the backend is just sitting
>> there removing temp files before it'll give control back.

> I believe this is confirmed. I see that the backend after being
> sig-term'd are now cycling through unlinks as seen by strace -p $pid.

> None too quickly I might add and as mentioned earlier, the number of
> these files is in the millions so it's hard to predict when cleanup will
> finish.

Hm. I don't recall exactly what sort of data structure we use to track
open temp files, but it's certainly plausible that it's not too efficient
with millions of temp files :-(

> As we did one night prior, a hard shutdown got the rogue jobs closed a
> lot more quickly and we noticed that on the recovery restart, Pg took
> about 15 minutes to clear >23M files from the temp area. I assume we'd
> experience the same again if we need to do it.

Not sure how fast that is either. If you need to do it again, you could
try manually rm'ing everything under the pgsql_tmp directory before
letting the postmaster start.

regards, tom lane