Analyzing foreign tables & memory problems

During ANALYZE, in analyze.c, functions compute_minimal_stats
and compute_scalar_stats, values whose length exceed
WIDTH_THRESHOLD (= 1024) are not used for calculating statistics
other than that they are counted as "too wide rows" and assumed
to be all different.

This works fine with regular tables; values exceeding that threshold
don't get detoasted and won't consume excessive memory.

With foreign tables the situation is different. Even though
values exceeding WIDTH_THRESHOLD won't get used, the complete
rows will be fetched from the foreign table. This can easily
exhaust maintenance_work_mem.

A foreign data wrapper has no good way to counter the problem.
It can return truncated values in ist AcquireSampleRowsFunc,
but WIDTH_THRESHOLD is private to analyze.c and it's a bad idea
to hard code a cutoff limit of 1025.

I can think of two remedies:
1) Expose WIDTH_THRESHOLD in commands/vacuum.h and add documentation
so that the authors of foreign data wrappers are aware of the
problem and can avoid it on their side.
This would be quite simple.

2) Instead of one callback that returns all sample rows, have
a callback that just returns the next table row (or the next
table row from a subset of the pages of the table in the
internal case). This function could be called in a short-lived
memory context. Vitter's algorithm for selecting a sample
and the truncation of excessively long values would then be
handled in analyze.c.
This would avoid the problem completely and make it easier
to write a foreign data wrapper.
I haven't thought this out completely, and it would require
bigger changes to analyze.c and the API than are probably
welcome this close to beta.

What is your opinion?

Yours,
Laurenz Albe