Re: Create TOAST table only if AM needs

Hi,

On 2019-05-17 15:13:50 -0400, Robert Haas wrote:
> On Fri, May 17, 2019 at 2:34 PM Andres Freund <andres(at)anarazel(dot)de> wrote:
> > Yea, Robert is also working on this. In fact, we were literally chatting
> > about it a few minutes ago. He'll probably chime in too.
>
> Yeah, I'm aiming to post a patch set very soon that does a bunch of
> refactoring of the TOAST stuff to make life easier for new AMs - maybe
> today, or else Monday. I think your use case of wanting to suppress
> TOAST table creation altogether is a valid one, but I want to do go a
> little further and make it easier for non-heap AMs to implement
> heap-like toasting based on their own page format.
>
> Generally, I would say that the state of play with respect to table
> AMs and toasting is annoyingly bad right now:
>
> - If your AM uses some system other than TOAST to store large values,
> you are out of luck. You will get TOAST tables whether you want them
> or not.

Which is aesthetically and indode usage wise annoying, but not
*terrible*. You get a a bunch of useless pg_class/pg_index entries and a
few close-to-empty relfilenodes.

> - If your AM uses some page or tuple format that results in a
> different maximum tuple size, you are out of luck. You will get TOAST
> tables based on whether a heap table with the same set of columns
> would need one.

> - If your AM would like to use the heap for TOAST data, you are out of
> luck. The AM used for TOAST data must be the same as the AM used for
> the main table.
>
> - If your AM would like to use itself to store TOAST data, you are
> also out of luck, because all of the existing TOAST code works with
> heap tuples.
>
> - Even if you copy all of tuptoaster.c/h - which is a lot of code -
> and change everything that is different for your AM than for the
> regular heap, you are still out of luck, because code that knows
> nothing about tableam is going to call heap_tuple_untoast_attr() to
> detoast stuff, and that code is only going to be happy if you've used
> the same chunk size that we use for the regular heap, and that chunk
> size has a good chance of being mildly to severely suboptimal if your
> heap has made any sort of page format changes.

Well, I don't *quite* buy the suboptimal. As far as I can tell, the
current chunking size doesn't have much going for it for heap either -
and while a few people (me including) have complained about it, it's not
that many people either. My impression is that the current chunking is
essentially a randomly chosen choice without much to go for it, and so
it's not going to be much different for other AMs.

> So I think this basically just doesn't work right now.

I mean, the zheap on tableam code copies more toast code than I'm happy
about, and it's chunking is somewhat suboptimal, but that's not
*terrible*. There's no if(zheap) branches outside of zheap related to
toasting.

Greetings,

Andres Freund