Re: PG_DUMP backup

Hi Husam,

I know the problem is not pg_dump. It is a combination of things.
The hardware is old and the configuration is terrible:
CPU AMD Opteron(tm) Processor 246
RAM 8GB
DISKS single volume 300GB with 70GB FREE
1GB SWAP

It might be there are some problems also with the database design and application, how it works.

What I am trying to do is to gather some ideas on how I can improve the backup time so we can at least backup daily.
Once we have improved certain things internally then I am sure we will upgrade this Server.

How you guys are backing up your servers?
Using pg_dump using pitr? Combination?

I have an idea of what hardware I should have to run a 176GB database, but what sort of configuration are you guys running?

Thank you very much

Best regards

Renato

Renato Oliveira
Systems Administrator
e-mail: renato(dot)oliveira(at)grant(dot)co(dot)uk

Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
http://www.grant.co.uk/

Grant Instruments (Cambridge) Ltd

Company registered in England, registration number 658133

Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK

-----Original Message-----

From: Tomeh, Husam [mailto:HTomeh(at)facorelogic(dot)com]
Sent: 12 February 2010 19:31
To: Renato Oliveira; pgsql-admin(at)postgresql(dot)org
Subject: RE: [ADMIN] PG_DUMP backup

Backing up a 170GB in 28 hours definitely doesn't sound right and I
almost certain has nothing to do with pg_dump, but rather your hardware,
ie, server, disk, etc. with a 170GB, backup should be done in a couple
of hours in my opinion. Seems to be more like a system resource issue.

Regards,
Husam

-----Original Message-----
From: pgsql-admin-owner(at)postgresql(dot)org
[mailto:pgsql-admin-owner(at)postgresql(dot)org] On Behalf Of Renato Oliveira
Sent: Friday, February 12, 2010 1:59 AM
To: pgsql-admin(at)postgresql(dot)org
Subject: [ADMIN] PG_DUMP backup
Importance: High

Dear all,

I have a server running 8.2.4 and has a database 170GB in size.
Currently I am backing it up using pg_dump and it takes around 28 hours,
sadly.
I was asked to check and compare the newly created DUMP file to the live
database and compare records.

I personally cannot see an easy or quick way of doing this, and even the
point in doing so.
I am already restoring the full database to a separate server and no
errors were reported.

How far can I trust pg_dump, can I trust a restore of the full DUMP to a
separate server will be good enough?
The time it takes for me to:
1 - Backup it up
2 - Transfer the dump (12.7GB compressed) to the office across the
internet
3 - Decompress the full dump locally which will be 105GB raw file
4 - Restore the full dump to a test server

It is easily a week to do all of that, when I have finished doing all of
that if a problem has developed with my live server, what good that test
will be? How relevant that will be? How helpful?

My question is:
1 - Is there a more efficient way of backing up such large database,
using pg_dump or any other tool?
2 - Is there an easy way to compare the live database with the DUMP file
just created?
3 - If I restore the database to a separate server, is there a point to
do such a check, especially if it is going to take suck a long time to
even start doing such a check?

Idea:
Pg_dump to split the file into smaller usable chuncks, which could be
restored one at time, is that possible?

PS I know about PITR but I can't implement it yet as I am still figuring
out certain things with the restore process.

I would really appreciate helps, please?

Thank you very much in advance

Renato

Renato Oliveira
Systems Administrator
e-mail: renato(dot)oliveira(at)grant(dot)co(dot)uk

Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
http://www.grant.co.uk/

Grant Instruments (Cambridge) Ltd

Company registered in England, registration number 658133

Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK

P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is
confidential. It is intended only for the named recipients(s). If you
are not the named recipient please notify the sender immediately and do
not disclose the contents to another person or take copies.

VIRUSES: The contents of this e-mail or attachment(s) may contain
viruses which could damage your own computer system. Whilst Grant
Instruments (Cambridge) Ltd has taken every reasonable precaution to
minimise this risk, we cannot accept liability for any damage which you
sustain as a result of software viruses. You should therefore carry out
your own virus checks before opening the attachment(s).

OpenXML: For information about the OpenXML file format in use within
Grant Instruments please visit our
http://www.grant.co.uk/Support/openxml.html

--
Sent via pgsql-admin mailing list (pgsql-admin(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
******************************************************************************************
This message may contain confidential or proprietary information intended only for the use of the
addressee(s) named above or may contain information that is legally privileged. If you are
not the intended addressee, or the person responsible for delivering it to the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying this message is strictly
prohibited. If you have received this message by mistake, please immediately notify us by
replying to the message and delete the original message and any copies immediately thereafter.

Thank you.
******************************************************************************************
FACLD

-----Original Message-----

P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies.

VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).

OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our http://www.grant.co.uk/Support/openxml.html