Re: multi billion row tables: possible or insane?

What do your "values" consist of?

Would it be possible to group several hundred or thousand of them into a
single row somehow that still makes it possible for your queries to get at
them efficiently?

What kind of queries will you want to run against the data?

For example if you have a measurement of some process value each
millisecond, it might be a good performance tradeoff to pack a whole
second of measurements into a single row if your data processing only
needs to access the values sequentially. With this single step you
immediately reduced your row and transaction number to the 1/1000th.

Please tell us more.

On Tue, 1 Mar 2005, Ramon Bastiaans wrote:

> Hi all,
>
> I am doing research for a project of mine where I need to store several
> billion values for a monitoring and historical tracking system for a big
> computer system. My currect estimate is that I have to store (somehow)
> around 1 billion values each month (possibly more).
>
> I was wondering if anyone has had any experience with these kind of big
> numbers of data in a postgres sql database and how this affects database
> design and optimization.
>
> What would be important issues when setting up a database this big, and
> is it at all doable? Or would it be a insane to think about storing up
> to 5-10 billion rows in a postgres database.
>
> The database's performance is important. There would be no use in
> storing the data if a query will take ages. Query's should be quite fast
> if possible.
>
> I would really like to hear people's thoughts/suggestions or "go see a
> shrink, you must be mad" statements ;)
>
> Kind regards,
>
> Ramon Bastiaans
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)
>