Re: White paper on very big databases

On Wed, 2009-02-04 at 21:23 -0500, Jonah H. Harris wrote:
> On Wed, Feb 4, 2009 at 12:09 PM, Joshua D. Drake
> <jd(at)commandprompt(dot)com> wrote:
>
> Well therein lies the problem. CMD has a customer with a
> multi-terrabyte
> table (not including the rest of the database) but we can't
> really talk
> about it :(
>
> IIRC, EnterpriseDB had one customer with over 1TB of data, but they
> too would have been hush-hush about it. When I was consulting, I saw
> very few Postgres databases at or over 1TB. While Postgres can handle
> fairly large data sets, it lacks some fairly important VLDB features
> which is probably why there are so few people with multi-terabyte PG
> databases. Perhaps JD/Fetter know of more, but I can count the ones I
> know of at < 10.

We have one internal system that is over the 2TB mark.

One of the biggest issues around this is that we have to keep two
completely separate copies of this systems up and running in parallel
for DR purposes. Restoring a dump takes several days (on high end
hardware and disk subsystems), and the system can not be down for that
long - so we need to keep two completely separate instances running in
case the primary fails.

Unfortunately this system still runs on 7.4 largely due to the major
PITA it is to upgrade a database of this size (we finally have the
traction to upgrade this to 8.3). Dump and restore is the only option
for us with this DB for an upgrade. We can't drop our backup node for
the several days and if we want to do it safely we need 3x the allocated
disk space in order to accomplish this - space for production, the
backup, and the newly minted 8.3 system That's a lot of high end disk
array real estate to justify.

--
Brad Nicholson 416-673-4106
Database Administrator, Afilias Canada Corp.