Re: [HACKERS] LONG

>
> I thought about the huge size variable text type a little
> more. And I think I could get the following implementation
> to work reliable for our upcoming release.
>
> For any relation, having one or more LONG data type
> attributes, another relation (named pg_<something>) is
> created, accessible only to superusers (and internal access
> routines). All LONG data items are stored as a reference
> into that relation, split up automatically so the chunks fit
> into the installation specific tuple limit size. Items are
> added/updated/removed totally transparent.

Should we use large objects for this, and beef them up. Seems that
would be a good way. I have considered putting them in a hash
bucket/directory tree for faster access to lots of large objects.

There is a lot to say about storing long tuples outside the tables
because long tuples fill cache buffers and make short fields longer to
access.

>
> It would not be indexable (jesus no!) and using it in a WHERE
> clause will be expensive. But who ever uses a WHERE on a not
> indexable (possibly containing megabytes per item) data type
> is a silly fool who should get what he wanted, poor response
> times.

Good restriction.

> I'd like to name it LONG, like Oracle's 2G max. data type.
> Even if I intend to restrict the data size to some megabytes
> for now. All the data must still be processable in memory,
> and there might be multiple instances of one item in memory
> at the same time. So a real 2G datatype is impossible with
> this kind of approach. But isn't a 64MB #define'd limit
> enough for now? This would possibly still blow away many
> installations due to limited memory and/or swap space. And we
> can adjust that #define in 2001 (an address space odyssey),
> when 64bit hardware and plenty of GB real memory is the low
> end standard *1).
>
> I already thought that the 8K default BLKSIZE is a little out
> of date for today's hardware standards. Two weeks ago I
> bought a PC for my kids. It's a 433MHz Celeron, 64MB ram, 6GB
> disk - costs about $500 (exactly DM 999,-- at Media Markt).
> With the actual on disk cache <-> memory and cache <->
> surface transfer rates, the 8K size seems a little archaic to
> me.

We use 8K blocks because that is the base size for most file systems.
When we fsync an 8k buffer, the assumption is that that buffer is
written in a single write to the disk. Larger buffers would be spread
over the disk, making a single fsync() impossible to be atomic, I think.

Also, larger buffers take more cache space per buffer, makeing the
buffer cache more corse holding fewer buffers.

--
Bruce Momjian | http://www.op.net/~candle
maillist(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026