TOAST (was: BLOB)

> Hello
>
> I would like to know
> - how to create under psql a BLOB field in sql language
> - how store a value (and not a file) in sql language
> - the maximum size of the BLOB field

For now (and 7.0), there is no way to have rows >8K in a
regular table.

But I'm actually making alot of progress in breaking it.
Seems we'll get rid of this restriction in 7.1.

To share some info with the other developers:

I just successfully stored the entire PG sources (~10M) in
one table. Stored are all 936 .[chyl] files. The tables
schema is:

sources (
s_path text,
s_data clob
)

clob is a user defined, very simple varsize datatype, I
created for testing. The entire source tree inserts in 23
seconds. If I cut off all sources at 7K and insert into a
text field it needs 8 seconds. Not bad, because the amount
of data shrinks <4M due to the cutoff.

Reading the data back shows similar results, 0.8 secs vs. 0.3
secs.

And the toaster already uses a heap+index to store values
external.

This all done with full toasting - i.e. try to compress, and
only if it still doesn't fit move already compressed
attribute into secondary relation. As a side effect, the 10M
of sources only need 3.4M main and 2.6M secondary heap.

I experienced some trouble with occational "Deadlock"
detections, when using multiple processes hammering on the
toaster. But I was able to reproduce the same error with a
plain 7.0 and "text" attributes (only far less frequent).
Seems there's something wrong in our deadlock detection
algorithm.

Another thing is, that ISTM that index's never shrink on
vacuum. One requirement of TOAST is, that the secondaries
tables index-OID is remembered in the external reference, so
the ref can be passed around in the entire backend and
whenever needed, it's real value can be fetched quickly by an
index scan. Thus, it's not possible to drop/recreate an index
on that right now. Seems we need a "REBUILD INDEX" utility or
the like.

Anyway, TOAST is on it's way. And as soon as we go for 7.1,
I'll need alot of help to make all our existing types
toastable, add administrative utility commands and teach
pg_dump to deal with all that.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck(at)debis(dot)com (Jan Wieck) #