Re: Inquiry on Generating Bitmaps from Filter Conditions in Index Scans

Thanks a lot! This is exactly what I'm asking. We've tried the CustomScanAPI at https://github.com/tensorchord/pgvecto.rs/pull/126, but met error with "variable not found in subplan target list". We're still investigating the root cause and thanks for your guidance!

Best
Jinjing Zhou
> From: "Matthias van de Meent"<boekewurm+postgres(at)gmail(dot)com>
> Date: Mon, Nov 20, 2023, 19:33
> Subject: Re: Inquiry on Generating Bitmaps from Filter Conditions in Index Scans
> To: "Jinjing Zhou"<allenzhou(at)tensorchord(dot)ai>
> Cc: "pgsql-hackers(at)lists(dot)postgresql(dot)org"<pgsql-hackers(at)lists(dot)postgresql(dot)org>
> On Mon, 20 Nov 2023 at 09:30, Jinjing Zhou <allenzhou(at)tensorchord(dot)ai> wrote:
> >
> > Hi hackers,
> >
> > I hope this message finds you well. I am reaching out to seek guidance on a specific aspect of PostgreSQL's index scanning functionality.
> >
> > I am currently working on a vector search extension for postgres, where I need to generate bitmaps based on filter conditions during an index scan. The goal is to optimize the query performance by efficiently identifying the rows that meet the given criteria.
> >
> > The query plan looks like this
> >
> > Index Scan using products_feature_idx on products (cost=0.00..27.24 rows=495 width=12)
> > Order By: (feature <-> '[0.5, 0.5, 0.5]'::vector)
> > Filter: ((price > '0.2'::double precision) AND (price <= '0.7'::double precision))
> >
> >
> > We have a custom index for the order by clause on the feature column. Now we want to utilize the index on other columns like price column. We want to access the bitmap of price column's filter condition in the feature column index. Is there any way I can achieve this goal?
>
> If you mean "I'd like to use bitmaps generated by combining filter
> results from index A, B, and C for (pre-)filtering the ordered index
> lookups in index D",
> then there is no current infrastructure to do this. Bitmap scans
> currently generate a data structure that is not indexable, and can
> thus not be used efficiently to push an index's generated bitmap into
> another bitmap's scans.
>
> There are efforts to improve the data structures we use for storing
> TIDs during vacuum [0] which could extend to the TID bitmap structure,
> but even then we'd need some significant effort to rewire Postgres'
> internals to push down the bitmap filters; and that is even under the
> assumption that pushing the bitmap down into the index AM is more
> efficient than doing the merges above the index AM and then re-sorting
> the data.
>
> So, in short, it's not currently available in community PostgreSQL.
> You could probably create a planner hook + custom executor node that
> does this, but it won't be able to use much of the features available
> inside PostgreSQL.
>
> Kind regards,
>
> Matthias van de Meent
>
> [0] https://www.postgresql.org/message-id/flat/CANWCAZbrZ58-w1W_3pg-0tOfbx8K41_n_03_0ndGV78hJWswBA%2540mail.gmail.com