best ways to handle large matrix data

From: "Wenjian Yang" <wenjiany(at)gmail(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: best ways to handle large matrix data
Date: 2006-08-02 15:29:12
Message-ID: bf9d6a9e0608020829j20e2c745p6d500d989710eb0b@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi, this maybe a more general question and not limited to postgresql.

We are generating some matrix like data, e.g. one object will have a fixed
number 100000 values, and we have currently 300 objects. We can visualized
the data as a 100000 by 300 matrix, and each object/column has an
identifier, each row has an identifier as well. The number of rows are
fixed, but columns will increase by time. The data is type double.

The queries that we encounter most often are that given a set of rowids, e.g.
(1000, 2000, 2001, 3241), retrieve the slice of the data. (the set of rowids
are usually generated from SQL)

I know that NetCDF and HDF5 can be used to strore such information, but is
there any way we can save the matrix in postgres database and have easy
retrieval and addition without sacrificing too much space? If not, I guess
I'll try to write some functions in postgres to do some retrieval from
netcdf.

Thanks.

Wenjian

Browse pgsql-general by date

  From Date Subject
Next Message Martijn van Oosterhout 2006-08-02 15:36:28 Re: Autovacuum help..
Previous Message Richard Huxton 2006-08-02 15:03:17 Re: Performance/Issues with CMP and JBoss