Merging large volumes of data

From: "Ambrus Wagner (IJ/ETH)" <ambrus(dot)wagner(at)ericsson(dot)com>
To: <pgsql-performance(at)postgresql(dot)org>
Subject: Merging large volumes of data
Date: 2007-05-07 13:05:58
Message-ID: 53CCFDD6E346CB43994852666C210E91C7AEE8@esealmw116.eemea.ericsson.se
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Dear All,

I have several tables containing data sorted by 2 keys (neither are keys in db terms (not unique), however). I would like to retrieve all rows from all tables sorted by the same keys, essentially merging the contents of the tables together. While I am completely aware of sort order not being a (fundamental) property of an RDBMS table, I am also aware of indices and clustering (in fact, data is inserted into the tables into the correct order, and not consequently modified in any way). I have a union query like this one:

select a,b,c,d,e from table1 union all
select a,b,c,d,e from table2 union all
etc...
select a,b,c,d,e from tablen order by a,b;

Is there a way to prevent PostgreSQL from doing a full sort on the result set after the unions have been completed? Even if I write

(select a,b,c,d,e from table1 order by a,b) union all
(select a,b,c,d,e from table2 order by a,b) union all
etc...
(select a,b,c,d,e from tablen order by a,b) order by a,b;

PostgreSQL does not seem to realise (maybe it should not be able to do this trick anyway) that the last "order by" clause is merely a final merge step on the ordered data sets.

Is there a workaround for this within PostgreSQL (another type of query, parameter tuning, stored procedure, anything) or should I use my back-up plan of making separate queries and merging the results in the target language?

Thanks a lot,
Ambrus

--
Wagner, Ambrus (IJ/ETH/GBD)
Tool Designer
GSDC Hungary

Location: Science Park, A2 40 008
Phone: +36 1 439 5282

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Andreas Kostyrka 2007-05-07 13:44:34 Re: Merging large volumes of data
Previous Message Tom Lane 2007-05-07 00:35:56 Re: Index not being used in sorting of simple table