Re: Confronting the maximum column limitation

On Aug 12, 2008, at 2:11 PM, Jeff Gentry wrote:

> On Tue, 12 Aug 2008, Steve Atkins wrote:
>> What operations do you perform on the data? If it's just store and
>> retrieve, can you serialize them into a bytea (or xml) field?
>
> Store & retrieve although we take advantage of the fact that it's in
> a DB
> to allow for subsetting (done at the postgres level), which cuts
> down on
> client side overhead as well as network traffic.
>
> The DB is accessed by a variety of clients (including a webapp) which
> could all perform that sort of work if necessary, although it's been
> nice
> to subset at the DB level. I'm not very familiar w/ the serialization
> methods you're talking about

I wasn't thinking of anything specific, more just some convenient way
of mapping the data structure into one or more larger chunks of data,
rather than one column per cell.

It may well be possible to do some of the serialization in stored
functions
in the database, to move that away from having to implement it in
each client.

> - would that have me needing to do full
> retrieval and subsetting on the client side? (definitely not a deal
> breaker, I'm just trying to get as many ideas w/ related info as
> possible
> before bringing this whole issue up with the powers that be).
>

Maybe, maybe not. It would depend on how you serialized it, what
your typical subsets were and so on.

Serialization isn't the only solution to storing this sort of data (EAV
of some sort would be another), but it's something worth looking at.

I think that what's sensible to do is going to depend on the details
of the data and (more so) the ways you access it.

Cheers,
Steve