Re: [HACKERS] A Better External Sort?

Let's pretend we get a 24HD HW RAID solution like that J Baker
says he has access to and set it up as a RAID 10. Assuming
it uses two 64b 133MHz PCI-X busses and has the fastest HDs
available on it, Jeff says he can hit ~1GBps of XFS FS IO rate
with that set up (12*83.3MBps= 1GBps).

Josh says that pg can't do more than 25MBps of DB level IO
regardless of how fast the physical IO subsystem is because at
25MBps, pg is CPU bound.

Just how bad is this CPU bound condition? How powerful a CPU is
needed to attain a DB IO rate of 25MBps?

If we replace said CPU with one 2x, 10x, etc faster than that, do we
see any performance increase?

If a modest CPU can drive a DB IO rate of 25MBps, but that rate
does not go up regardless of how much extra CPU we throw at
it...

Ron

-----Original Message-----
From: Josh Berkus <josh(at)agliodbs(dot)com>
Sent: Oct 3, 2005 6:03 PM
To: "Jeffrey W. Baker" <jwbaker(at)acm(dot)org>
Cc:
Subject: Re: [HACKERS] [PERFORM] A Better External Sort?

Jeffrey,

> I guess database reads are different, but I remain unconvinced that they
> are *fundamentally* different. After all, a tab-delimited file (my sort
> workload) is a kind of database.

Unfortunately, they are ... because of CPU overheads. I'm basing what's
"reasonable" for data writes on the rates which other high-end DBs can
make. From that, 25mb/s or even 40mb/s for sorts should be achievable
but doing 120mb/s would require some kind of breakthrough.

> On a single disk you wouldn't notice, but XFS scales much better when
> you throw disks at it. I get a 50MB/sec boost from the 24th disk,
> whereas ext3 stops scaling after 16 disks. For writes both XFS and ext3
> top out around 8 disks, but in this case XFS tops out at 500MB/sec while
> ext3 can't break 350MB/sec.

That would explain it. I seldom get more than 6 disks (and 2 channels) to
test with.

--
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org