Advice on machine specs for growth

[I sent this to the performance list a couple of days ago and received
no replies. Apologies for the cross-post]

We are looking to upgrade our current database server infrastructure so
that it is suitable for the next 3 years or so.

Presently we have two physical servers with the same specs:

- 220GB database partition on RAID10 SSD on HW RAID
- 128GB RAM
- 8 * Xeon E5-2609

(The HW RAID card is a MegaRAID SAS 9361-8i with BBU)

The second server is a hot standby to the first, and we presently have
about 350 databases in the cluster.

We envisage needing about 800GB of primary database storage in the next
three years, with 1000 databases in the cluster.

We are imagining either splitting the cluster into two and (to have four
main servers) or increasing the disk capacity and RAM in each server.
The second seems preferable from a day-to-day management basis, but it
wouldn't be too difficult to deploy our software upgrades across two
machines rather than one.

Resources on the main machines seem to be perfectly adequate at present
but it is difficult to know at what stage queries might start spilling
to disk. We presently occasionally hit 45% CPU utilisation, load average
peaking at 4.0 and we occasionally go into swap in a minor way (although
we can't determine the reason for going into swap). There is close to no
iowait in normal operation.

It also seems a bit incongruous writing about physical machines these
days, but I can't find pricing on a UK data protection compatible cloud
provider that beats physical price amortised over three years (including
rack costs). The ability to more easily "make" machines to help with
upgrades is attractive, though.

Some comments and advice on how to approach this would be very
gratefully received.

Thanks
Rory