Re: performance optimzations

On Wed, 2003-11-12 at 22:35, Rod Taylor wrote:
> You may want to consider keeping the compressed email in a separate
> table than the information describing it. It would mean descriptive
> information is more likely to be in RAM, where the body probably doesn't
> matter as much (you view them 1 at a time, subjects tend to be listed
> all at once).

Thanks for the suggestions. Splitting the load between several machines
was the original intent of moving the storage from the file system to a
database. I believe the schema I'm already using splits out the body
due to the size of some attachments. Luckily the code already gzips the
email body and abbreviates common email headers so storing compressed
emails isn't a problem.

> Most clients will be interested in say the last 7 days worth of data?
> Great.. Start out with 4GB ram on a good Dual CPU -- Opterons seem to
> work quite well -- and make sure the motherboard can hold double that in
> memory for an upgrade sometime next year when you've become popular.

Unfortunately, the hardware available is pretty much fixed in regards to
the system. I can play around with the raid configurations and have
some limited choice in regards to the raid controller and number of
drivers but that's about all in terms of hardware.

> I firmly believe lots of RAM is the answer to most IO issues until you
> start getting into large sets of active data (>50GB). 64GB ram is fairly
> cheap compared to ongoing maintenance of the 30+ drive system required
> to get decent throughput.

The current file system holding the user and email information indicates
the current data has about 64GB (70K accounts, I'm not sure how many are
active but 50% might be good guess). This seems to be somewhat of a
steady state however.

--
Suchandra Thapa <s-thapa-11(at)alumni(dot)uchicago(dot)edu>