From: | Craig Gibson <craiggib(at)gmail(dot)com> |
---|---|
To: | pgsql-general(at)postgresql(dot)org |
Subject: | Performance issue with cross table updates |
Date: | 2012-09-10 16:34:29 |
Message-ID: | CADpEpVhUGt+8sMno4xmS4p1PFEdE1xGA+3toMmbYJeA8q5ZDxQ@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general |
Hi all
I am no database wizard so I am hoping someone may be able to assist me :)
I get a daily CSV file of 6.5 million records. I create a temporary
table and COPY them in. On completion I create an index on the mdnid
column. This column is also indexed in table 2. This part is very
fast. I had some 'checkpoint too often' issues, but that I have
resolved.
I then use the following procedure to update all the records, and if a
record does not exist, insert it instead:
declare
v_update e_lookup_load%ROWTYPE;
begin
for v_update in select * from e_lookup_load loop
update e_lookup set
cur_balance = v_update.cur_balance,
last_usage = v_update.last_usage,
b_code = v_update.b_code,
u_date = v_update.u_date,
date_of_insert = v_update.date_of_insert
where mdnid = v_update.mdnid;
if NOT FOUND then
begin
insert into e_lookup(mdnid,cur_balance,last_usage,b_code,u_date,date_of_insert)
values
(v_update.mdnid,
v_update.cur_balance,
v_update.last_usage,
v_update.b_code,
v_update.u_date,
v_date_of_insert);
end;
end if;
end loop;
end;
From my understanding, a for loop is encapsulated in a cursor anyway
so no need to do that. Am I fundamentally doing something wrong as the
operation is slow as molasses? Maybe there is a better way altogether
that I have not thought of? The bottom line is that at no point can
the e_lookup table be unavailable to clients, else I would have just
done a simple drop and rename post the COPY.
Kind Regards
Craig
From | Date | Subject | |
---|---|---|---|
Next Message | Gustav Potgieter | 2012-09-10 16:54:06 | Postgresql replication assistance |
Previous Message | Michael Sacket | 2012-09-10 15:03:53 | Re: [GENERAL] INSERT. RETURNING for copying records |