RE: [HACKERS] sort on huge table

Now that's a close to linear as you are going to get. Pretty good I think:
a sort of one billion rows in half an hour.

Mikea

>> -----Original Message-----
>> From: Tatsuo Ishii [mailto:t-ishii(at)sra(dot)co(dot)jp]
>> Sent: Thursday, November 04, 1999 10:30 AM
>> To: Tom Lane
>> Cc: t-ishii(at)sra(dot)co(dot)jp; pgsql-hackers(at)postgreSQL(dot)org
>> Subject: Re: [HACKERS] sort on huge table
>>
>>
>> >
>> >Tatsuo Ishii <t-ishii(at)sra(dot)co(dot)jp> writes:
>> >> I have compared current with 6.5 using 1000000
>> tuple-table (243MB) (I
>> >> wanted to try 2GB+ table but 6.5 does not work in this case). The
>> >> result was strange in that current is *faster* than 6.5!
>> >
>> >> RAID5
>> >> current 2:29
>> >> 6.5.2 3:15
>> >
>> >> non-RAID
>> >> current 1:50
>> >> 6.5.2 2:13
>> >
>> >> Seems my previous testing was done in wrong way or the behavior of
>> >> sorting might be different if the table size is changed?
>> >
>> >Well, I feel better now, anyway ;-). I thought that my first cut
>> >ought to have been about the same speed as 6.5, and after I added
>> >the code to slurp up multiple tuples in sequence, it should've been
>> >faster than 6.5. The above numbers seem to be in line with that
>> >theory. Next question: is there some additional effect that comes
>> >into play once the table size gets really huge? I am thinking maybe
>> >there's some glitch affecting performance once the temp file size
>> >goes past one segment (1Gb). Tatsuo, can you try sorts of say
>> >0.9 and 1.1 Gb to see if something bad happens at 1Gb? I could
>> >try rebuilding here with a small RELSEG_SIZE, but right at the
>> >moment I'm not certain I'd see the same behavior you do...
>>
>> Ok. I have run some testings with various amount of data.
>>
>> RedHat Linux 6.0
>> Kernel 2.2.5-smp
>> 512MB RAM
>> Sort mem: 80MB
>> RAID5
>>
>> 100 million tuples 1:31
>> 200 4:24
>> 300 7:27
>> 400 11:11 <-- 970MB
>> 500 14:01 <-- 1.1GB (segmented files)
>> 600 18:31
>> 700 22:24
>> 800 24:36
>> 900 28:12
>> 1000 32:14
>>
>> I didn't see any bad thing at 1.1GB (500 million).
>> --
>> Tatsuo Ishii
>>
>> ************
>>