Re: Streaming large data into postgres [WORM like applications]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/12/07 19:49, Dhaval Shah wrote:
> Consolidating my responses in one email.
>
> 1. The total data that is expected is some 1 - 1.5 Tb a day. 75% of
> the data comes in a period of 10 hours. Rest 25% comes in the 14
> hours. Of course there are ways to smooth the load patterns, however
> the current scenario is as explained.
>
> 2 I do expect that the customer rolls in something like a NAS/SAN with
> Tb of disk space. The idea is to retain the data for a duration and
> offload it to tape.

45TB per month???? Wow.

The archival process *must* be considered when designing the system.

PostgreSQL's ability to use tablespaces and partitioned tables will
make that much easier. Otherwise, you'd have to be deleting from
one "side" of the table while inserting into the other "side".

Partitioning will also let you divide the table into multiple
"active" segments, so that multiple inserters can run simultaneously
without stepping on each other while spreading the load across
multiple controllers and RAID-sets.

If it's a SAN/NAS that is organized into RAID-5 groups, make *sure*
that it has *lots* of batter-backed write-back cache.

Regarding compression: if the columns are integers or short VARCHAR
fields, I do not see how compression can help you, unless you use
block-layer compression.

Which Linux doesn't do.

Does FreeBSD have block-level compression?

- --
Ron Johnson, Jr.
Jefferson LA USA

Give a man a fish, and he eats for a day.
Hit him with a fish, and he goes away for good!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGR8mtS9HxQb37XmcRAjXQAJ9TN2FqU1Wo4PZmS6MAhxaJgCm6/wCfXIl8
wZOYG7vWxwODNaRwDGSJxYQ=
=Qh2r
-----END PGP SIGNATURE-----