Re: Optimize external TOAST storage

On Wed, Mar 16, 2022 at 2:36 PM Nathan Bossart <nathandbossart(at)gmail(dot)com> wrote:
> Thinking further, is simply reducing the number of TOAST chunks the right
> thing to look at? If I want to add a TOAST attribute that requires 100,000
> chunks, and you told me that I could save 10% in the read path for an extra
> 250 chunks of disk space, I would probably choose read performance. If I
> wanted to add 100,000 attributes that were each 3 chunks, and you told me
> that I could save 10% in the read path for an extra 75,000 chunks of disk
> space, I might choose the extra disk space. These are admittedly extreme
> (and maybe even impossible) examples, but my point is that the amount of
> disk space you are willing to give up may be related to the size of the
> attribute. And maybe one way to extract additional read performance with
> this optimization is to use a variable threshold so that we are more likely
> to use it for large attributes.

Right, so perhaps the ultimate thing here would be a more fine-grained
knob than SET STORAGE EXTERNAL -- something that allows you to specify
that you want to compress only when it really helps. While some people
might find that useful, I think the current patch is less ambitious,
and I think that's OK. It just wants to save something in the cases
where it's basically free. Unfortunately we've learned that it's never
*entirely* free because making the last TOAST chunk longer can always
cost you something, even if it gets longer by only 1 byte. But for
larger values it's hard for that to be significant.

--
Robert Haas
EDB: http://www.enterprisedb.com