Re: Hardware recommendations to scale to silly load

On Thu, 2003-08-28 at 03:17, matt wrote:
> > Are you *sure* about that???? 3K updates/inserts per second xlates
> > to 10,800,000 per hour. That, my friend, is a WHOLE HECK OF A LOT!
>
> Yup, I know!
>
> > During the 1 hour surge, will SELECTs at 10 minutes after the
> > hour depend on INSERTs at 5 minutes after the hour?
>
> Yes, they do. It's a payments system, so things like account balances
> and purchase histories have to be updated in real time.
>
> > Only one hour out of 168????? May I ask what kind of app it is?
>
> Online voting for an unnamed TV show...
>
> > > If the best price/performance option is a second hand 32-cpu Alpha
> > > running VMS I'd be happy to go that way ;-)
> >
> > I'd love to work on a GS320! You may even pick one up for a million
> > or 2. The license costs for VMS & Rdb would eat you, though.
>
> You'd be amazed how little they do go for actually :-)

Then that's what I'd do. VMS, Rdb, (your favorite 3GL language).
Presumably the SELECT statements will be direct lookup instead
of range retrieval? If so, then I'd create a *large* amount of
GLOBAL BUFFERS, many MIXED AREAs, tables PLACED VIA HASHED INDEXES
so that the index nodes are on the same page as the corresponding
tuples. Thus, 1 disk I/O gets both the relevant index key, plus
the tuple. (Each I/O reads 3 pages into GBs [Global Buffers], so
that if a later statement needs a records nearby, it's already in
RAM.)

With fast storage controllers (dual-redundant, with 512MB each)
you could even use RAID5, and your app may not even know the diffie.
Of course, since the requirements are *so* extreme, better still
stick to RAID10.

I know that a certain pharmaceutical company had a similar situation,
where test results would flood in every morning. A certain North-
eastern US wireless phone company needed to record every time every
phone call was started and stopped.

The technique I described is how both of these high-volume apps
solved The Need For Speed.

With VMS 7.3 and Rdb 7.1.04 and, oh, 16GB RAM, a carefully crafted
stored procedure run an hour or 2 before the show could pull the
necessary 5GB slice of the DB into GBs, and you'd reduce the I/O
load during the show itself.

Sorry it's not PostgreSQL, but I *know* that Rdb+VMS could handle
the task...

--
-----------------------------------------------------------------
Ron Johnson, Jr. ron(dot)l(dot)johnson(at)cox(dot)net
Jefferson, LA USA

"You ask us the same question every day, and we give you the
same answer every day. Someday, we hope that you will believe us..."
U.S. Secretary of Defense Donald Rumsfeld, to a reporter