Re: Pros/cons of big databases vs smaller databases and RDS

On 10/5/21 5:06 PM, Wells Oliver wrote:
> Hi-- to keep it short, I feel like I've generally heard the larger your DB
> is, the less efficient it might run, likely due to disk I/O. Maybe I'm
> terribly mistaken in this perception.

Less *efficient*???

Of course /ceteris paribus/ it takes longer to do "whole database"
activities on a larger database because... there's more data, but that
should be obvious to all.

So you scale up the IO (SAN, RAID 50, faster disks, etc)  or partition the
data *or both*.

>
> We have two DBs, one primarily accessed by humans and systems, which is
> ~1TB in size, and aggregates most of what we store in raw, longer format
> on a second DB that is about ~6TB in size.
>
> As we consider plans to migrate to RDS, we've talked a lot about
> combining the two as more and more the case is querying the larger DB and
> wanting data only available in the smaller DB.
>
> Of course, we can solve this by copying things back and forth, but we're
> also thinking: why not just one big DB?
>
> Anyone have any experiences with a similar project, and especially any
> technical configurations that might be beneficial in using RDS?

Nothing like your scenario, but we recently migrated an 8TB Oracle database
hosted on a VM + (big fancy) SAN to a 6TB RDS Postgresql database.  Reads
are faster, and writes seem a bit slower, but enough software changed that
comparisons are dubious.

One thing I /can/ say is, "watch your costs!!"  Amazon does an excellent job
of obfuscating that, hiding it from everyone but the accountants.  It's
dreadfully easy for technical people to unknowingly spend tens or hundreds
of thousands of dollars in a month when testing things.

--
Angular momentum makes the world go 'round.