From: | "Jim C(dot) Nasby" <jnasby(at)pervasive(dot)com> |
---|---|
To: | ashah <ashah(at)opuspro(dot)com> |
Cc: | pgsql <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: Massive Inserts Strategies |
Date: | 2006-03-22 16:09:25 |
Message-ID: | 20060322160924.GJ15742@pervasive.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-performance |
Load the files into a temp table and go from there...
COPY ... FROM file;
UPDATE existing_table SET ... WHERE ...;
INSERT INTO existing_table SELECT * FROM temp_table WHERE NOT EXISTS(
SELECT * FROM existing_table WHERE ...)
On Wed, Mar 22, 2006 at 10:32:10AM -0500, ashah wrote:
> I have a database with foreign keys enabled on the schema. I receive different
> files, some of them are huge. And I need to load these files in the database
> every night. There are several scenerios that I want to design an optimal
> solution for -
>
> 1. One of the file has around 80K records and I have to delete everything from
> the table and load this file. The provider never provides a "delta file" so I
> dont have a way to identify which records are already present and which are
> new. If I dont delete everything and insert fresh, I have to make around 80K
> selects to decide if the records exist or not. Now there are lot of tables
> that have foreign keys linked with this table so unless I disable the foreign
> keys, I cannot really delete anything from this table. What would be a good
> practise here?
>
> 2. Another file that I receive has around 150K records that I need to load in
> the database. Now one of the fields is logically a "foreign key" to another
> table, and it is linked to the parent table via a database generated unique
> ID instead of the actual value. But the file comes with the actual value. So
> once again, I have to either drop the foreign key, or make 150K selects to
> determine the serial ID so that the foreign key is satisfied. What would be a
> good strategy in this scenerio ?
>
> Please pardon my inexperience with database !
>
> Thanks,
> Amit
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
From | Date | Subject | |
---|---|---|---|
Next Message | Scott Marlowe | 2006-03-22 16:16:14 | Re: planner with index scan cost way off actual cost, |
Previous Message | PFC | 2006-03-22 16:07:52 | Re: Massive Inserts Strategies |