Re: Analyzing foreign tables & memory problems

On Wed, May 02, 2012 at 12:20:39PM +0200, Albe Laurenz wrote:
> >>> 1) Expose WIDTH_THRESHOLD in commands/vacuum.h and add documentation
> >>> so that the authors of foreign data wrappers are aware of the
> >>> problem and can avoid it on their side.
> >>> This would be quite simple.
>
> >> Seems reasonable. How would the FDW return an indication that a
> value was
> >> non-NULL but removed due to excess width?
> >
> > The FDW would return a value of length WIDTH_THRESHOLD+1 that is
> > long enough to be recognized as too long, but not long enough to
> > cause a problem.
>
> Here is a simple patch for that.

It feels to me like a undue hack to ask FDW authors to synthesize such values.
It's easy enough for data types such as text/bytea. In general, though,
simple truncation may not produce a valid value of the type. That shouldn't
matter, since the next action taken on the value should be to discard it, but
it's fragile. Can we do better?

Just thinking out loud, we could provide an "extern Datum AnalyzeWideValue;"
and direct FDW authors to use that particular datum. It could look like a
toasted datum of external size WIDTH_THRESHOLD+1 but bear va_toastrelid ==
InvalidOid. Then, if future code movement leads us to actually examine one of
these values, we'll get an early, hard error.

Thanks,
nm