Re: pg_basebackup bug: base backup is double the size of the database

One clarification:

On Wed, Jan 21, 2015 at 9:32 AM, Craig James <cjames(at)emolecules(dot)com> wrote:

> We've encountered a serious bug with pg_basebackup. It seems to be
> following hard links and duplicating all files in the tablespaces rather
> than preserving links.
>

It could be barman, not pg_basebackup, that has the bug. I assumed that
barman was using pg_basebackup, but one of my colleagues pointed out that
we're using barman, which may or may not use pg_basebackup to do its work.

Craig

>
> Drilling down into one specific tablespace, we find this:
>
> # ls -l /data/postgres-9.3/main/pg_tblspc/16747
> lrwxrwxrwx 1 postgres postgres 27 2014-08-18 11:28
> /data/postgres-9.3/main/pg_tblspc/16747 -> /postgres/tablespaces/uorsy/
>
> # du -sh /data/postgres-9.3/tablespaces/uorsy
> *35G* /data/postgres-9.3/tablespaces/uorsy
>
> # du -sh /data/postgres-9.3/tablespaces/uorsy/*
> *35G* /data/postgres-9.3/tablespaces/uorsy/8208624
> *8.1M* /data/postgres-9.3/tablespaces/uorsy/PG_9.3_201306121
> 4.0K /data/postgres-9.3/tablespaces/uorsy/pgsql_tmp
> 4.0K /data/postgres-9.3/tablespaces/uorsy/PG_VERSION
>
> # find /data/postgres-9.3/tablespaces/uorsy \! -links 1 -type f | wc -l
> *740*
>
>
> In other words, this tablespace has 35G of real data, plus 740 hard links
> that effectively duplicate each data file.
>
> When we look at the same data in the archive that pg_basebackup creates
> (invoked via barman), we find this:
>
> # du -sh /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747
> *70G* /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747
>
> # du -sh /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747/*
> *35G*
> /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747/8208624
> *35G*
> /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747/PG_9.3_201306121
> 4.0K
> /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747/pgsql_tmp
> 4.0K
> /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747/PG_VERSION
>
> # find /pg_archive/staging/base/20150114T170002/pgdata/pg_tblspc/16747 \!
> -links 1 -type f | wc -l
> *0*
>
>
> That is, no hard links, and all of the data files are duplicated. And of
> course, when we try to actually use this archive to recover, it's twice the
> size as the original database and doesn't fit on our disks.
>
> My guess is that pg_basebackup is using (or doing the equivalent of)
> rsync(1) without the --hard-links option, and that these hard links were
> created by pg_upgrade when we went from 8.4.17 to 9.3.5.
>
> What can we do to fix this? The whole cluster is about 350 databases and
> 800GB.
>
> Thanks,
> Craig
>
>

--
---------------------------------
Craig A. James
Chief Technology Officer
eMolecules, Inc.
---------------------------------