From: | ashah <ashah(at)opuspro(dot)com> |
---|---|
To: | pgsql <pgsql-performance(at)postgresql(dot)org> |
Subject: | Massive Inserts Strategies |
Date: | 2006-03-22 15:32:10 |
Message-ID: | 200603221032.10207.ashah@opuspro.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-performance |
I have a database with foreign keys enabled on the schema. I receive different
files, some of them are huge. And I need to load these files in the database
every night. There are several scenerios that I want to design an optimal
solution for -
1. One of the file has around 80K records and I have to delete everything from
the table and load this file. The provider never provides a "delta file" so I
dont have a way to identify which records are already present and which are
new. If I dont delete everything and insert fresh, I have to make around 80K
selects to decide if the records exist or not. Now there are lot of tables
that have foreign keys linked with this table so unless I disable the foreign
keys, I cannot really delete anything from this table. What would be a good
practise here?
2. Another file that I receive has around 150K records that I need to load in
the database. Now one of the fields is logically a "foreign key" to another
table, and it is linked to the parent table via a database generated unique
ID instead of the actual value. But the file comes with the actual value. So
once again, I have to either drop the foreign key, or make 150K selects to
determine the serial ID so that the foreign key is satisfied. What would be a
good strategy in this scenerio ?
Please pardon my inexperience with database !
Thanks,
Amit
From | Date | Subject | |
---|---|---|---|
Next Message | Tom Lane | 2006-03-22 15:35:26 | Re: WAL logging of SELECT ... INTO command |
Previous Message | Jim C. Nasby | 2006-03-22 15:14:34 | Re: WAL logging of SELECT ... INTO command |