Re: BLOB support

=?utf-8?q?Rados=C5=82aw_Smogura?= <rsmogura(at)softperience(dot)eu> writes:
> Introducing streaming for TOAST is little useless, sorry just for cite from
> my, mentoined document:

> (This is generally about on demand stream of TOASTed value, in
> context of LOBs is acceptable, as long not transactional aware LOBs are
> acceptable). If we will add streaming of TOASTed values, so caller will
> get some reference to this value, we need to ensure that pointed data
> will not be changed, nor deleted - I think this will require caller to
> add FOR UPDATE (or silently to add this by server) for each statement
> returning pointers to TOASTed, as client may do transactional query,

It's already been explained to you that that's not the case.

> If this is acceptable I will do following changes.

> Add
> - server_max_in_memory_lob_size - GUC server start-only config to describe
> maximum value of client session parameter max_in_memory_lob.

> - max_in_memory_lob - session GUC describing how huge LOBs may be keept in
> memory before backing up to file

> - rescursivly toasting, detoasting during insert/update/remove for searching
> for LOBs (we need this for arrays and complex types) - this is for last stage
> (error disallowing LOBs in composites/arrays may be quite enaugh, for
> begining) - I want LOBs to be starting point for LOBing other types (e.g. some
> big arrays may be LOBbed).

> - during toasting, lob will be toasted and in place of LOB, the reference to
> it will be putted, and encoded in LOB datum.

> - api for LOB manipulation (few changes to current implementation) in way that
> BLOB is field type and uninon of in_memory, in_file, in_toast. This few bytes
> for LOB will not affect size of datum looking at size of LOB.

> - api for maintaing temoraly lob files, we need this as per session list of id
> -> file desc, to prevent prevent stealing of lobs by different connections
> (security)

> - streaming api for TOASTED values (based on COPY protocol, or changed COPY
> protocol) or at least function calls - I havent looked at this in context of
> TOASTed LOBs.

> Is it good?

This all looks like you decided on a solution first and then started to
look for a problem to apply it to. I don't want to see us inventing a
pile of GUCs for this, and I don't think there is a need to make any
fundamental changes in the TOAST mechanism either. What we do need is a
streaming access protocol to read or write wide field values *without*
forcing people to change the basic structure of their tables. You keep
on wanting to invent weird, IMO unnecessary language features instead.
Try to make the feature as transparent as possible, not as visible as
possible.

regards, tom lane