Re: large database

On Tue, 2012-12-11 at 10:00 -0800, Jeff Janes wrote:
> On Mon, Dec 10, 2012 at 12:26 PM, Mihai Popa <mihai(at)lattica(dot)com> wrote:
> > Hi,
> >
> > I've recently inherited a project that involves importing a large set of
> > Access mdb files into a Postgres or MySQL database.
> > The process is to export the mdb's to comma separated files than import
> > those into the final database.
> > We are now at the point where the csv files are all created and amount
> > to some 300 GB of data.
>
> Compressed or uncompressed?

uncompressed, but that's not much relief...
and it's 800GB not 300 anymore. I still can't believe the size of this
thing.

> Why did you originally choose MySQL? What has changed that causes you
> to rethink that decision? Does your team have experience with MySQL
> but not with PostgreSQL?

I did not choose it; somebody before me did. I personally have more
experience with Postgres, but not with databases as large as this one
promises to be.

>
> I like PostgreSQL, of course, but if I already had an
> already-functioning app on MySQL I'd be reluctant to change it.

...and I'm not rushing to do it; I was just asking around, maybe there
are known issues with MySQL, or with Postgres for that matter.

> My understanding is that RDS does not support Postgres, so if you go
> that route the decision is already made for you. Or am I wrong here?

That's right, but I could still get an EC2 instance and run my own
Postgres
Or use this: http://www.enterprisedb.com/cloud-database/pricing-amazon

> 1TB of storage sounds desperately small for loading 300GB of csv files.

really? that's good to know; I wouldn't have guessed

> IOPS would mostly depend on how you are using the system, not how large it is.

mostly true

--
Mihai Popa <mihai(at)lattica(dot)com>
Lattica, Inc.