Re: PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device

FOUTE K. Jaurès wrote:

> Le mer. 1 juil. 2020 à 00:11, raf <raf(at)raf(dot)org> a écrit :
>
> > FOUTE K. Jaurčs wrote:
> >
> > > Hi everyone,
> > >
> > > I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
> > and
> > > all is working fine until i faced this error below.
> > >
> > > ERROR: could not rename file
> > > "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
> > > "pg_logical/snapshots/1A-7C00D890.snap": No space left on device
> > >
> > > Is there someone who faced this problem?
> > > Any idea how I can solve it ?
> > >
> > > BTW: I don't have any problem with space.
> > >
> > > Best Regard
> > >
> > > --
> > > Jaurčs FOUTE
> >
> > If you really haven't run out of space,
> > you might have run out of inodes.
> > The -i option of df should show you.
> > 100,000 empty files could cause that.
> > I wouldn't think that renaming would
> > require a new inode, but I also wouldn't
> > think that renaming would require any
> > more space on a file system.
>
>
> The result of df -i
>
> Sys. de fichiers Inœuds IUtil. ILibre IUti% Monté sur
> udev 3065149 433 3064716 1% /dev
> tmpfs 3072780 665 3072115 1% /run
> /dev/sdb2 59973632 356029 59617603 1% /
> tmpfs 3072780 7 3072773 1% /dev/shm
> tmpfs 3072780 10 3072770 1% /run/lock
> tmpfs 3072780 17 3072763 1% /sys/fs/cgroup
> /dev/sdb1 0 0 0 - /boot/efi
> tmpfs 3072780 19 3072761 1% /run/user/108
> tmpfs 3072780 5 3072775 1% /run/user/1001
> tmpfs 3072780 5 3072775 1% /run/user/1000

So that's not it. It would take ~60 million
files to fill up your / inode table. I can't
think of another explanation for that error
message if df without -i also shows free space.

I tried googling for "No space left on device"
and everything suggested checking inodes with "df -i".
One suggested looking for files that were deleted
but that were still open by a process (can be found
with "lsof / | grep deleted") but if that were the
problem, df would show a full disk (but du wouldn't
be able to account for it).

Could it be some other "device"? like shared memory
segment space or IPC resources? That seems unlikely if
the operation was just a file system rename. Perhaps
strace/stap could help check if it really was a file
system rename that went wrong (if there's any reason to
doubt it). It does seem to clearly be a file rename
though.

If there any chance that the drive is failing? But
you'd expect "I/O error" messages if that were the case.

Is it on a journalling file system and could the journal
be the device that is full? I don't know how to check
that (or if it even makes sense).

cheers,
raf