Re: Pluggable toaster

Hi hackers!

I want to bump this thread and remind the community about Pluggable TOAST.
Overall Pluggable TOAST was revised to work without altering PG_ATTRIBUTE
table
- no atttoaster column, allow to drop unused Toasters and some other
improvements.
Sorry for the big delay, but it was a big piece of work to do, and the work
is still going on.

Here are the main highlights:

1) No need to modify the PG_ATTRIBUTE table. We've introduced new catalog
table with a set of internal support functions that keeps all table-toaster
relations:
postgres(at)postgres=# \d+ pg_toastrel;
Table "pg_catalog.pg_toastrel"
Column | Type | Collation | Nullable | Default | Storage |
Toaster | Compression | Stats target | Description
--------------+----------+-----------+----------+---------+---------+---------+-------------+--------------+-------------
oid | oid | | not null | | plain |
| | |
toasteroid | oid | | not null | | plain |
| | |
relid | oid | | not null | | plain |
| | |
toastentid | oid | | not null | | plain |
| | |
attnum | smallint | | not null | | plain |
| | |
version | smallint | | not null | | plain |
| | |
relname | name | | not null | | plain |
| | |
toastentname | name | | not null | | plain |
| | |
flag | "char" | | not null | | plain |
| | |
toastoptions | "char" | | not null | | plain |
| | |
Indexes:
"pg_toastrel_oid_index" PRIMARY KEY, btree (oid)
"pg_toastrel_name_index" UNIQUE CONSTRAINT, btree (toasteroid, relid,
version, attnum)
"pg_toastrel_rel_index" btree (relid, attnum)
"pg_toastrel_tsr_index" btree (toasteroid)
Access method: heap
(This is not final definition)

This approach allows us to keep all Toaster assignment history, as well as
correctly storing
Toasters assigned to different columns of the relation, and even have
separate TOAST
storage entities (these not necessary to be regular TOAST tables) for
different columns.

When the table with the TOASTable column is created - a new row is inserted
into
PG_TOASTREL with source table OID, Toaster OID, created TOAST entity OID,
column
(attribute) index. Special field "version" is used to keep history of
Toasters assigned to
the column - it is a counter which increases with each assignment, and the
biggest version
is the current Toaster for the column. All assigned Toasters are
automatically cached,
and when the value is TOASTed - first lookup is done in cache, and if there
is no cached
Toaster it is searched in PG_TOASTREL and inserted in cache.

2) Attribute "reltoastrelid" was replaced with calls of PG_TOASTREL support
functions.
This was done to allow each TOASTed column to be assigned with different
Toaster
and have its individual TOAST table.

3) DROP TABLE command was modified to remove corresponding records from the
PG_TOASTREL - to allow dropping toasters that are out of use.

4) DROP TOASTER command was introduced. This command allows to drop unused
Toasters - the ones that do not have records in PG_TOASTREL. If the Toaster
was
assigned to a column - it could not be dropped, because all data TOASTed
with it will
be lost.

The branch is still in development so I it is too early for patch but
here's link to the repo:
https://github.com/postgrespro/postgres/tree/toastapi_with_ctl

On Mon, Nov 7, 2022 at 1:35 PM Aleksander Alekseev <aleksander(at)timescale(dot)com>
wrote:

> Hi Nikita,
>
> > Pluggable TOAST is provided as an interface to allow developers to plug
> > in custom TOAST mechanics. It does not determines would it be universal
> > Toaster or one data type, but syntax for universal Toaster is out of
> scope
> > for this patchset.
>
> If I understand correctly, what is going to happen - the same
> interface TsrRoutine is going to be used for doing two things:
>
> 1) Implementing type-aware TOASTers as a special case for the default
> TOAST algorithm when EXTERNAL storage strategy is used.
> 2) Implementing universal TOASTers from scratch that have nothing to
> do with the default TOAST algorithm.
>
> Assuming this is the case, using the same interface for doing two very
> different things doesn't strike me as a great design decision. While
> working on v24 you may want to rethink this.
>
> Personally I believe that Pluggable TOASTers should support only case
> #2. If there is a need of reusing parts of the default TOASTer,
> corresponding pieces of code should be declared as non-static and
> called from the pluggable TOASTers directly.
>
> Alternatively we could have separate interfaces for case #1 and case
> #2 but this IMO becomes rather complicated.
>
> > I'm currently working on a revision of Pluggable TOAST that would make
> > dropping Toaster possible if there is no data TOASTed with it, along with
> > several other major changes. It will be available in this (I hope so) or
> the
> > following, if I won't make it in time, commitfest.
>
> Looking forward to v24!
>
> This is a major change so I hope there will be more feedback from
> other people on the mailing list.
>
> --
> Best regards,
> Aleksander Alekseev
>

--
Regards,

--
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/