Re: Logical insert/update/delete WAL records for custom table AMs

On Fri, Nov 5, 2021 at 4:53 AM Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
>
> On Thu, 2021-11-04 at 14:33 +0530, Amit Kapila wrote:
> > Can't different tableAM's have different representations
> > for toast or may be some different concept like speculative
> > insertions?
>
> The decoding plugin should just use the common interfaces to toast, and
> if the table AM supports toast, everything should work fine.
>

I think that is true only if table AM uses same format as heap for
toast. It also seems to be relying heap tuple format.

> The only
> special thing it needs to do is check VARATT_IS_EXTERNAL_ONDISK(),
> because on-disk toast data can't be decoded (which is true for heap,
> too).
>
> I haven't looked as much into speculative insertions, but I don't think
> those are a problem either. Shouldn't they be handled before they make
> it into the change stream that the plugin sees?
>

They will be handled before the plugin sees them but I was talking
about what if the table AM has some other WAL similar to WAL of
speculative insertions that needs special handling.

> > Similarly, I remember that for zheap we didn't had
> > TransactionIds for subtransactions so we need to make some changes in
> > logical decoding to compensate for that. I guess similar stuff could
> > be required for another table AMs. Then a different table AM can have
> > a different tuple format which won't work for current change records
> > unless we convert it to heap tuple format before writing WAL for it.
>
> Columnar certainly has a different format. That makes me wonder whether
> ReorderBufferChange and/or ReorderBufferTupleBuf are too low-level, and
> we should have a higher-level representation of a change that is based
> on slots.
>
> Can you tell me more about the changes you made for zheap? I still
> don't understand why the decoding plugin would have to know what table
> AM the change came from.
>

I am not talking about decoding plugin but rather decoding itself,
basically, the work we do in reorderbuffer.c, decode.c, etc. The two
things I remember were tuple format and transaction ids as mentioned
in my previous email. I think we should try to find a solution for
tuple format as the current decoding code relies on it if we want
decoding to deal with another table AMs transparently.

--
With Regards,
Amit Kapila.