Re: Zstandard support for toast compression

On Tue, May 17, 2022 at 04:12:14PM -0400, Stephen Frost wrote:
> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>> I'm with Robert in that I do not see any urgency to add another method.

Okay.

>> The fact that Stephen is already questioning whether LZ4 should have
>> been added first is not making me any more eager to jump here.
>> Compression methods come, and they go, and we do not serve anyone's
>> interest by being early adopters.

FWIW, I don't really question the choice of LZ4 as an alternative to
pglz. One very easily outclasses the other, guess which one. Perhaps
we would have gone with zstd back in the day, but here we are, andx
this option is already very good in itself.

Zstandard may not be old enough to vote, being only 7, but its use is
already quite spread. So I would not be surprised if it remains
popular for many years. We'll see how it goes.

> But, fine, then I'd suggest to Michael that he work on actively solving
> the problem we've now got where we have such a limited number of bits,
> and then come back and add Zstd after that's done. I disagree that we
> should be pushing back so hard on adding Zstd in general, but if we are
> going to demand that we have a way to support more than these few
> compression options before ever adding any new ones (considering how
> long it's taken Zstd to get to the level it is now, we're talking
> about close to a *decade* from such a new algorithm showing up and
> getting to a similar level of adoption, and then apparently more because
> we don't feel it's 'ready' yet), then let's work towards that and not
> complain when it shows up that it's not needed yet (as I fear would
> happen ... and just leave us unable to make useful progress).

Saying that, I agree with the point to not set in stone the 4th bit
used in the toast compression header, and that is would be better to
use it for a more extensible design. Didn't the proposal to introduce
the custom compression mechanisms actually touch this area? The set
of macros we have currently for the toast values in a varlena are
already kind of hard to figure out. Making that harder to parse would
not be appealing, definitely.
--
Michael