Re: RFI: Extending the TOAST Pointer

On Mon, 22 May 2023 at 18:13, Nikita Malakhov <hukutoc(at)gmail(dot)com> wrote:
>
> Hi,

Could you please not top-post.

> Aleksander, I'm interested in extending TOAST pointer in various ways.
> 64-bit TOAST value ID allows to resolve very complex issue for production
> systems with large tables and heavy update rate.

Cool. I agree that this would be nice, though it's quite unlikely
we'll ever be able to use much more than 32 bits concurrently with the
current 32-bit block IDs. But indeed, it is a good way of reducing
time spent searching for unused toast IDs.

> I agree with Matthias that there should not be processing of TOAST pointer
> internals outside TOAST macros. Currently, TOASTed value is distinguished
> as VARATT_IS_EXTERNAL_ONDISK, and it should stay this way. Adding
> compression requires another implementation (extension) of
> VARATT_EXTERNAL because current supports only 2 compression methods -
> it has only 1 bit responsible for compression method, and there is a safe
> way to do so, without affecting default TOAST mechanics - we must keep
> it this way for compatibility issues and not to break DB upgrade.
>
> Also, I must remind that we should not forget about field alignment inside
> the TOAST pointer.

What field alignment inside the TOAST pointers?
Current TOAST pointers are not aligned: the varatt_external struct is
copied into and from the va_data section of varattrib_1b_e when we
need to access the data; so as far as I know this struct has no
alignment to speak of.

> As it was already mentioned, it seems not very reasonable trying to save
> a byte or two while we are storing out-of-line values of at least 2 kb in size.

If we were talking about the data we're storing externally, sure. But
this is data we store in the original tuple, and moving that around is
relatively expensive. Reducing the aligned size of the toast pointer
can help reduce the size of the heap tuple, thus increasing the
efficiency of the primary data table.

Kind regards,

Matthias van de Meent
Neon, Inc.