Re: tuptoaster.c must *not* use SnapshotAny

Jan Wieck wrote:

>
> Any datatype that has the potential to be loaded into memory
> entirely by the backend is dangerous. It leads to problems
> like "out of swapspace", which aren't really funny in
> production.
>
> We need something that is restricted to streaming access
> only. Yet, it needs to have copy semantics.
>
> That means to me, that this will not only be a new datatype,
> but some more infrastructure too. We need INSERT ...
> RETURNING ... where the values for BLOBs are given as
> NEW_BLOB() and the returned identifier is subsequently used
> to open the created BLOBs for writing and pump the data in.
>
> We need heap access level tuple support to duplicate those
> values on INSERT ... SELECT, to remove them on DELETE and to
> manipulate them on UPDATE.
>
> We need new functionality in procedural languages to support
> streaming in and out of these guy's, where the actual data is
> buffered in temp files and only blob identifiers passed
> around.
>
> I see the final storage of blob data to happen in the toast
> table, sliced, diced and handy (most probably not
> compressed). But I don't see an easy way to abuse the bytea
> datatype for real blob support.
>

All this may be needed for BLOBs that are *huge*, but there are at least
two classes of BLOB data that are not typically huge in practice. One is
encrypted or hashed data, the other is image data for web pages. Either
of these work very well with the current bytea support.

Joe