Multiple indexes, huge table

From: Aram Fingal <fingal(at)multifactorial(dot)com>
To: Postgres-General General <pgsql-general(at)postgresql(dot)org>
Subject: Multiple indexes, huge table
Date: 2012-09-06 21:22:33
Message-ID: 5E885432-4ADD-451A-9682-2033DA27A4F8@multifactorial.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I have a table which currently has about 500 million rows. For the most part, the situation is going to be that I will import a few hundred million more rows from text files once every few months but otherwise there won't be any insert, update or delete queries. I have created five indexes, some of them multi-column, which make a tremendous difference in performance for the statistical queries which I need to run frequently (seconds versus hours.) When adding data to the table, however, I have found that it is much faster to drop all the indexes, copy the data to the table and then create the indexes again (hours versus days.) So, my question is whether this is really the best way. Should I write a script which drops all the indexes, copies the data and then recreates the indexes or is there a better way to do this?

There are also rare cases where I might want to make a correction. For example, one of the columns is sample name which is a foreign key to a samples table defined with " ON UPDATE CASCADE." I decided to change a sample name in the samples table which should affect about 20 million rows out of the previously mentioned 500 million. That query has now been running for five days and isn't finished yet.

-Aram

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Misa Simic 2012-09-06 21:24:05 Re: pivot functions with variable number of columns
Previous Message John R Pierce 2012-09-06 20:38:35 Re: pg_dump slow on windows