Re: Zstandard support for toast compression

On Tue, May 17, 2022 at 4:12 PM Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> I'm getting a bit of deja-vu here from when I was first trying to add
> TRUNCATE as a GRANT'able option and being told we didn't want to burn
> those precious bits.

Right, it's the same issue ... although in that case there are a lot
more bits available than we have here.

> But, fine, then I'd suggest to Michael that he work on actively solving
> the problem we've now got where we have such a limited number of bits,
> and then come back and add Zstd after that's done. I disagree that we
> should be pushing back so hard on adding Zstd in general, but if we are
> going to demand that we have a way to support more than these few
> compression options before ever adding any new ones (considering how
> long it's taken Zstd to get to the level it is now, we're talking
> about close to a *decade* from such a new algorithm showing up and
> getting to a similar level of adoption, and then apparently more because
> we don't feel it's 'ready' yet), then let's work towards that and not
> complain when it shows up that it's not needed yet (as I fear would
> happen ... and just leave us unable to make useful progress).

It's kind of ridiculous to talk about "pushing back so hard on adding
Zstd in general" when there's like 2 emails expressing only moderate
skepticism. I clearly said I wasn't 100% against it.

But I want to point out here that you haven't really offered any kind
of argument in favor of supporting Zstd. You basically seem to just be
arguing that it's dumb to worry about running out of bit space, and I
think that's just obviously false. PostgreSQL is full of things that
are hard to improve because nearly all of the bit space was gobbled up
early on, and there's not much left for future features. The heap
tuple header format is an excellent example of this. Surely if we were
designing that over again today we wouldn't have expended some of
those bits on the things we did.

I do understand that Zstandard is a good compression algorithm, and if
we had an extensibility mechanism here where one of the four possible
bit patterns then indicates that the next byte (or two or four) stores
the real algorithm type, then what about adding Zstandard that way
instead of consuming one of our four primary bit patterns? That way
we'd have this option for people who want it, but we'd have more
options for the future instead of fewer.

i.e. something like:

00 = PGLZ
01 = LZ4
10 = reserved for future emergencies
11 = extended header with additional type byte (1 of 256 possible
values reserved for Zstandard)

I wouldn't be worried about getting backed into a corner with that approach.

--
Robert Haas
EDB: http://www.enterprisedb.com