From: | Oleg Broytmann <phd(at)sun(dot)med(dot)ru> |
---|---|
To: | PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org> |
Subject: | GROUP BY / ORDER BY string is very slow |
Date: | 1999-01-21 11:19:41 |
Message-ID: | Pine.SOL2.3.96.SK.990121140402.12992A-100000@sun.med.ru |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Hello!
I use a modified version of contrib/apache_logginig. The table I am
using:
CREATE TABLE combinedlog (
host text,
accdate abstime,
request text,
authuser text,
cookie text,
referer text,
useragent text,
stime int2,
status int2,
bytes int4
);
Once a month I run a very simple script to put WWW logs into the table.
The very loading is not fast (I am running postmaster with -F, I use
BEGIN/END and I drop indicies before loading. I remember when I started
using BEGIN/END loading speed up a bit, but not significantly), but is not
my biggest concern. What is worse is spped of my queries.
After inserting into the table, I run this shell script:
sel_f() {
field=$1
psql -d ce_wwwlog -c "SELECT COUNT($field), $field FROM raw_combinedlog GROUP BY 2 ORDER BY 1 DESC;"
}
for i in host request referer useragent; do
sel_f $i > by-$i
done
This works very, very slow. I tried to use indicies:
CREATE INDEX host ON combinedlog (host);
CREATE INDEX request ON combinedlog (request);
CREATE INDEX referer ON combinedlog (referer);
CREATE INDEX useragent ON combinedlog (useragent);
but indicies do not help much, and indexing time is so big, that sum of
CREATE INDEX + SELECT is even bigger :(
Why is it so slow? How can I speed it up?
I am running postgres compiled with --enable-locale. that is, for every
string comparision there are 2 (two) malloc calls and one strcoll. Can I
increase speed turning strcoll off? If so, postgres need a SET command to
turn localization temporary off. I can hack in, as I already rewrote
localization stuff a year ago. The only thing I want to hear from postgres
community (hackers, actually :) is how it should be named:
SET STRCOLL=off
or such?
I remember when I submitted my locale patch there was a discussion on
how to do it the Right Way, but I didn't remember the conlusion. What
finally we decided? I want to add command (if I should to) that is
compliant with other stuff here.
Oleg.
----
Oleg Broytmann http://members.tripod.com/~phd2/ phd2(at)earthling(dot)net
Programmers don't die, they just GOSUB without RETURN.
From | Date | Subject | |
---|---|---|---|
Next Message | Oleg Broytmann | 1999-01-21 13:46:59 | Big files on 32-bit Linux |
Previous Message | Roberto Joao Lopes Garcia | 1999-01-21 11:14:38 | Performance test with -F (Was Postgres Speed or lack thereof) |