Re: Assertion failure while streaming toasted data

On Tue, May 25, 2021 at 1:59 PM Pavan Deolasee <pavan(dot)deolasee(at)gmail(dot)com> wrote:

>
> On Tue, May 25, 2021 at 1:49 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>>
>> On Tue, May 25, 2021 at 1:45 PM Pavan Deolasee <pavan(dot)deolasee(at)gmail(dot)com> wrote:
>> >
>> > I am not entirely sure if it works correctly. I'd tried something similar, but the downstream node using
>> > my output plugin gets NULL values for the toast columns. It's a bit hard to demonstrate that with the
>> > test_decoding plugin, but if you have some other mechanism to test that change with an actual downstream
>> > node receiving and applying changes, it will be useful to test with that.
>>
>> Okay, I will test that. Thanks.
>>
>
> I modified test_decoding to print the tuples (like in the non-streamed case) and included your proposed fix. PFA
>
> When the transaction is streamed, I see:
> ```
> + opening a streamed block for transaction
> + table public.toasted: INSERT: id[integer]:9001 other[text]:'bbb' data[text]:'ccc'
> + table public.toasted: INSERT: id[integer]:9002 other[text]:'ddd' data[text]:'eee'
> + table public.toasted: INSERT: id[integer]:9003 other[text]:'bar' data[text]:unchanged-toast-datum
> <snipped>
> ```
>
> For a non-streamed case, the `data[text]` column shows the actual data. That probably manifests into NULL data when downstream handles it.

Yes, I am able to reproduce this, basically, until we get the last
tuple of the multi insert we can not clear the toast data otherwise we
can never form a complete tuple. So the only possible fix I can think
of is to consider the multi-insert WAL without the final multi-insert
tuple as partial data then we will avoid streaming until we get the
complete WAL of one multi-insert. I am working on the patch to fix
this, I will share that in some time.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com