Re: Add test module for Table Access Method

Hi all,

On Tue, Jan 16, 2024 at 10:40 AM Aleksander Alekseev <
aleksander(at)timescale(dot)com> wrote:

> Hi,
>
> > > I think a test module for a table AM will really help developers. Just
> > > to add to the above list - how about the table AM implementing a
> > > simple in-memory (columnar if possible) database storing tables
> > > in-memory and subsequently providing readers with the access to the
> > > tables?
> >
> > That's a good idea.
>
> Personally I would be careful with this idea.
>
> Practice shows that when you show the first incomplete, limited and
> buggy PoC it ends up being in the production environment the next day
> :) In other words sooner or later there will be users demanding a full
> in-memory columnar storage support from Postgres. I believe it would
> be a problem. Last time I checked TAM was not extremely good for
> implementing proper columnar storages, and there are lots of open
> questions when it comes to in-memory tables (e.g. what to do with
> foreign keys, inherited tables, etc).
>
> All in all I don't think we should provide something that can look /
> be interpreted as first-class alternative storage but in fact is not.
>

I tossed together a table access method for in-memory storage in column
format for experimental purposes over the holidays (I actually have a
row-based one as well, but that is in no shape to share at this point).
It's available under https://github.com/mkindahl/pg_arrow. The intention
was mostly to have something simple to play and experiment with. It is
loosely based on the Apache Arrow Columnar format, but the normal data
structures are not suitable for storing in shared memory so I have tweaked
it a little.

> > How about adding the same kind of coverage as dummy_index_am with a
> > couple of reloptions then? That can serve as a point of reference
> > when a table AM needs a few custom options. A second idea would be to
> > show how to use toast relations when implementing your new AM, where a
> > toast table could be created even in cases where we did not want one
> > with heap, when it comes to size limitations with char and/or varchar,
> > and that makes for a simpler needs_toast_table callback.
>
> Good ideas. Additionally we could provide a proxy TAM for a heap TAM
> which does nothing but logging used TAM methods, its arguments and
> return values. This would be a good example and also potentially can
> be used as a debugging tool.
>

We wrote a table access method for experimenting with and to be able to
trace what happens while executing various statements. It is available
under https://github.com/timescale/pg_traceam for anybody who is interested.

Best wishes,
Mats Kindahl

>
> --
> Best regards,
> Aleksander Alekseev
>
>
>