GROUP BY / ORDER BY string is very slow

From: Oleg Broytmann <phd(at)sun(dot)med(dot)ru>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: GROUP BY / ORDER BY string is very slow
Date: 1999-01-21 11:19:41
Message-ID: Pine.SOL2.3.96.SK.990121140402.12992A-100000@sun.med.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello!

I use a modified version of contrib/apache_logginig. The table I am
using:

CREATE TABLE combinedlog (
host text,
accdate abstime,
request text,
authuser text,
cookie text,
referer text,
useragent text,
stime int2,
status int2,
bytes int4
);

Once a month I run a very simple script to put WWW logs into the table.
The very loading is not fast (I am running postmaster with -F, I use
BEGIN/END and I drop indicies before loading. I remember when I started
using BEGIN/END loading speed up a bit, but not significantly), but is not
my biggest concern. What is worse is spped of my queries.
After inserting into the table, I run this shell script:

sel_f() {
field=$1
psql -d ce_wwwlog -c "SELECT COUNT($field), $field FROM raw_combinedlog GROUP BY 2 ORDER BY 1 DESC;"
}

for i in host request referer useragent; do
sel_f $i > by-$i
done

This works very, very slow. I tried to use indicies:

CREATE INDEX host ON combinedlog (host);
CREATE INDEX request ON combinedlog (request);
CREATE INDEX referer ON combinedlog (referer);
CREATE INDEX useragent ON combinedlog (useragent);

but indicies do not help much, and indexing time is so big, that sum of
CREATE INDEX + SELECT is even bigger :(

Why is it so slow? How can I speed it up?

I am running postgres compiled with --enable-locale. that is, for every
string comparision there are 2 (two) malloc calls and one strcoll. Can I
increase speed turning strcoll off? If so, postgres need a SET command to
turn localization temporary off. I can hack in, as I already rewrote
localization stuff a year ago. The only thing I want to hear from postgres
community (hackers, actually :) is how it should be named:
SET STRCOLL=off
or such?
I remember when I submitted my locale patch there was a discussion on
how to do it the Right Way, but I didn't remember the conlusion. What
finally we decided? I want to add command (if I should to) that is
compliant with other stuff here.

Oleg.
----
Oleg Broytmann http://members.tripod.com/~phd2/ phd2(at)earthling(dot)net
Programmers don't die, they just GOSUB without RETURN.

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Oleg Broytmann 1999-01-21 13:46:59 Big files on 32-bit Linux
Previous Message Roberto Joao Lopes Garcia 1999-01-21 11:14:38 Performance test with -F (Was Postgres Speed or lack thereof)