Re: Assertion failure while streaming toasted data

On Tue, May 25, 2021 at 2:57 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Tue, May 25, 2021 at 2:34 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
> > > When the transaction is streamed, I see:
> > > ```
> > > + opening a streamed block for transaction
> > > + table public.toasted: INSERT: id[integer]:9001 other[text]:'bbb' data[text]:'ccc'
> > > + table public.toasted: INSERT: id[integer]:9002 other[text]:'ddd' data[text]:'eee'
> > > + table public.toasted: INSERT: id[integer]:9003 other[text]:'bar' data[text]:unchanged-toast-datum
> > > <snipped>
> > > ```
> > >
> > > For a non-streamed case, the `data[text]` column shows the actual data. That probably manifests into NULL data when downstream handles it.
> >
> > Yes, I am able to reproduce this, basically, until we get the last
> > tuple of the multi insert we can not clear the toast data otherwise we
> > can never form a complete tuple. So the only possible fix I can think
> > of is to consider the multi-insert WAL without the final multi-insert
> > tuple as partial data then we will avoid streaming until we get the
> > complete WAL of one multi-insert.

Yeah, that sounds reasonable.

> > I am working on the patch to fix
> > this, I will share that in some time.
>
> The attached patch should fix the issue, now the output is like below
>

Your patch will fix the reported scenario but I don't like the way
multi_insert flag is used to detect incomplete tuple. One problem
could be that even when there are no toast inserts, it won't allow to
stream unless we get the last tuple of multi insert WAL. How about
changing the code such that when we are clearing the toast flag, we
additionally check 'clear_toast_afterwards' flag?

--
With Regards,
Amit Kapila.