pgBackRest for a 50 TB database

From: Abhishek Bhola <abhishek(dot)bhola(at)japannext(dot)co(dot)jp>
To: pgsql-general <pgsql-general(at)postgresql(dot)org>
Cc: Stephen Frost <sfrost(at)snowman(dot)net>
Subject: pgBackRest for a 50 TB database
Date: 2023-08-27 14:56:38
Message-ID: CAEDsCzi3DbpavRjDf0y4e+aYS9aGveD4F3ZGC+JVq7BO-FgCwA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi

I am trying to use pgBackRest for all my Postgres servers. I have tested it
on a sample database and it works fine. But my concern is for some of the
bigger DB clusters, the largest one being 50TB and growing by about
200-300GB a day.

I plan to mount NAS storage on my DB server to store my backup. The server
with 50 TB data is using DELL Storage underneath to store this data and has
36 18-core CPUs.

As I understand, pgBackRest recommends having 2 full backups and then
having incremental or differential backups as per requirement. Does anyone
have any reference numbers on how much time a backup for such a DB would
usually take, just for reference. If I take a full backup every Sunday and
then incremental backups for the rest of the week, I believe the
incremental backups should not be a problem, but the full backup every
Sunday might not finish in time.

I think converting a diff/incr backup to a full backup has been discussed
here <https://github.com/pgbackrest/pgbackrest/issues/644>, but not yet
implemented. If there is a workaround, please let me know. Or if someone is
simply using pgBackRest for a bigger DB (comparable to 50TB), please share
your experience with the exact numbers and config/schedule of backups. I
know the easiest way would be to use it myself and find out, but since it
is a PROD DB, I wanted to get some ideas before starting.

Thanks
Abhishek

--
_This correspondence (including any attachments) is for the intended
recipient(s) only. It may contain confidential or privileged information or
both. No confidentiality or privilege is waived or lost by any
mis-transmission. If you receive this correspondence by mistake, please
contact the sender immediately, delete this correspondence (and all
attachments) and destroy any hard copies. You must not use, disclose, copy,
distribute or rely on any part of this correspondence (including any
attachments) if you are not the intended
recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。_

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Stephen Frost 2023-08-27 15:22:30 Re: pgBackRest for a 50 TB database
Previous Message pan snowave 2023-08-27 01:54:25 回复: ident auth does not works as usual