Re: TOAST (was: BLOB)

Tom Lane wrote:

> wieck(at)debis(dot)com (Jan Wieck) writes:
> > To share some info with the other developers:
> > I just successfully stored the entire PG sources (~10M) in
> > one table. Stored are all 936 .[chyl] files.
>
> Cool! Sounds like you are making great progress.

At least I think so :-)

> > Anyway, TOAST is on it's way. And as soon as we go for 7.1,
> > I'll need alot of help to make all our existing types
> > toastable, add administrative utility commands and teach
> > pg_dump to deal with all that.
>
> Question: pg_dump shouldn't need to know about toasted tuples as
> such, should it? I'm guessing that what you are thinking of is
> adding administration capabilities to control whether particular
> columns are toast-able, and then pg_dump would need to know about
> saving and loading those control fields. But toasting a particular
> field value ought to be invisible to applications...

Answer: Right! The only change from a clients PoV is that
attribute values can be bigger now.

So it's only to teach pg_dump to correctly restore all the
possible tuning. For now there is an attstorage (char)
attribute telling in the TupleDesc, what can be done with a
particular value in a HeapTuple. The meanings are:

'p' don't touch, either DBA doesn't want or the type isn't
capable of dealing with it.
'e' never try to compress, if needed store external as is.
'x' try to compress and if needed store compression result
external.
'm' try to keep in main tuple as long as possible, if
needed handle like 'x'.

If a tuple doesn't fit into a block, the toaster performs 4
loops over the attributes, each one stops immediately when
the tuple fits.

1. Inline compress storage 'x' attributes.

2. Move storage 'x' and 'e' attributes to secondary
relation.

3. Inline compress storage 'm' attributes.

4. Move storage 'm' attributes to secondary relation.

My plan now is to add a default storage value to pg_type. At
CREATE TABLE time, a secondary relation plus it's index are
automatically created if attributes of the table are
toastable. This is questionable, because it'll cost 2 inodes
and 16K per table, but we can change it later and have
special admin commands to deal with it. The toaster doesn't
jump in if a relation doesn't have a secondary one. But it
will expand values from other relations in that case (INSERT
... SELECT).

With the pg_type entry telling what to do by default, we can
work on all our types one by one, and enable them at the time
all their type specific functions can handle it.

Finally, I'd like to give the secondary relations another
relkind, so they are ignored by vacuum, and then vacuum them
explicitly while the LOCK on the master table is still held.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck(at)debis(dot)com (Jan Wieck) #