Re: BitmapHeapScan streaming read user and prelim refactoring

From: Melanie Plageman <melanieplageman(at)gmail(dot)com>
To: Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com>
Cc: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>, Heikki Linnakangas <hlinnaka(at)iki(dot)fi>, Andres Freund <andres(at)anarazel(dot)de>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>, Nazir Bilal Yavuz <byavuz81(at)gmail(dot)com>
Subject: Re: BitmapHeapScan streaming read user and prelim refactoring
Date: 2024-03-31 15:45:51
Message-ID: 20240331154551.lty4mundyoyhtixw@liskov
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri, Mar 29, 2024 at 12:05:15PM +0100, Tomas Vondra wrote:
>
>
> On 3/29/24 02:12, Thomas Munro wrote:
> > On Fri, Mar 29, 2024 at 10:43 AM Tomas Vondra
> > <tomas(dot)vondra(at)enterprisedb(dot)com> wrote:
> >> I think there's some sort of bug, triggering this assert in heapam
> >>
> >> Assert(BufferGetBlockNumber(hscan->rs_cbuf) == tbmres->blockno);
> >
> > Thanks for the repro. I can't seem to reproduce it (still trying) but
> > I assume this is with Melanie's v11 patch set which had
> > v11-0016-v10-Read-Stream-API.patch.
> >
> > Would you mind removing that commit and instead applying the v13
> > stream_read.c patches[1]? v10 stream_read.c was a little confused
> > about random I/O combining, which I fixed with a small adjustment to
> > the conditions for the "if" statement right at the end of
> > read_stream_look_ahead(). Sorry about that. The fixed version, with
> > eic=4, with your test query using WHERE a < a, ends its scan with:
> >
>
> I'll give that a try. Unfortunately unfortunately the v11 still has the
> problem I reported about a week ago:
>
> ERROR: prefetch and main iterators are out of sync
>
> So I can't run the full benchmarks :-( but master vs. streaming read API
> should work, I think.

Odd, I didn't notice you reporting this ERROR popping up. Now that I
take a look, v11 (at least, maybe also v10) had this very sill mistake:

  if (scan->bm_parallel == NULL &&
      scan->rs_pf_bhs_iterator &&
      hscan->pfblockno > hscan->rs_base.blockno)
       elog(ERROR, "prefetch and main iterators are out of sync");

It errors out if the prefetch block is ahead of the current block --
which is the opposite of what we want. I've fixed this in attached v12.

This version also has v13 of the streaming read API. I noticed one
mistake in my bitmapheap scan streaming read user -- it freed the
streaming read object at the wrong time. I don't know if this was
causing any other issues, but it at least is fixed in this version.

- Melanie

Attachment Content-Type Size
v12-0001-BitmapHeapScan-begin-scan-after-bitmap-creation.patch text/x-diff 2.8 KB
v12-0002-BitmapHeapScan-set-can_skip_fetch-later.patch text/x-diff 2.2 KB
v12-0003-Push-BitmapHeapScan-skip-fetch-optimization-into.patch text/x-diff 15.0 KB
v12-0004-BitmapPrefetch-use-prefetch-block-recheck-for-sk.patch text/x-diff 2.2 KB
v12-0005-Update-BitmapAdjustPrefetchIterator-parameter-ty.patch text/x-diff 2.3 KB
v12-0006-table_scan_bitmap_next_block-returns-lossy-or-ex.patch text/x-diff 4.4 KB
v12-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local.patch text/x-diff 2.9 KB
v12-0008-Remove-table_scan_bitmap_next_tuple-parameter-tb.patch text/x-diff 4.1 KB
v12-0009-Make-table_scan_bitmap_next_block-async-friendly.patch text/x-diff 23.0 KB
v12-0010-Unify-parallel-and-serial-BitmapHeapScan-iterato.patch text/x-diff 16.0 KB
v12-0011-table_scan_bitmap_next_block-counts-lossy-and-ex.patch text/x-diff 5.2 KB
v12-0012-Hard-code-TBMIterateResult-offsets-array-size.patch text/x-diff 5.4 KB
v12-0013-Separate-TBM-Shared-Iterator-and-TBMIterateResul.patch text/x-diff 20.7 KB
v12-0014-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch text/x-diff 31.5 KB
v12-0015-Remove-table_scan_bitmap_next_block.patch text/x-diff 11.8 KB
v12-0016-v13-Streaming-Read-API.patch text/x-diff 70.5 KB
v12-0017-BitmapHeapScan-uses-read-stream-API.patch text/x-diff 26.5 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Bharath Rupireddy 2024-03-31 15:48:14 Re: New Table Access Methods for Multi and Single Inserts
Previous Message Devrim Gündüz 2024-03-31 15:40:20 Re: Security lessons from liblzma