Re: Compressed pluggable storage experiments

On 2019-Oct-10, Ildar Musin wrote:

> 1. Unlike FDW API, in pluggable storage API there are no routines like
> "begin modify table" and "end modify table" and there is no shared
> state between insert/update/delete calls.

Hmm. I think adding a begin/end to modifytable is a reasonable thing to
do (it'd be a no-op for heap and zheap I guess).

> 2. It looks like I cannot implement custom storage options. E.g. for
> compressed storage it makes sense to implement different compression
> methods (lz4, zstd etc.) and corresponding options (like compression
> level). But as i can see storage options (like fillfactor etc) are
> hardcoded and are not extensible. Possible solution is to use GUCs
> which would work but is not extremely convinient.

Yeah, the reloptions module is undergoing some changes. I expect that
there will be a way to extend reloptions from an extension, at the end
of that set of patches.

> 3. A bit surprising limitation that in order to use bitmap scan the
> maximum number of tuples per page must not exceed 291 due to
> MAX_TUPLES_PER_PAGE macro in tidbitmap.c which is calculated based on
> 8kb page size. In case of 1mb page this restriction feels really
> limiting.

I suppose this is a hardcoded limit that needs to be fixed by patching
core as we make table AM more pervasive.

> 4. In order to use WAL-logging each page must start with a standard 24
> byte PageHeaderData even if it is needless for storage itself. Not a
> big deal though. Another (acutally documented) WAL-related limitation
> is that only generic WAL can be used within extension. So unless
> inserts are made in bulks it's going to require a lot of disk space to
> accomodate logs and wide bandwith for replication.

Not sure what to suggest. Either you should ignore this problem, or
you should fix it.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services