Re: speeding up planning with partitions

On Thu, 14 Mar 2019 at 21:35, Imai, Yoshikazu
<imai(dot)yoshikazu(at)jp(dot)fujitsu(dot)com> wrote:
> 0007:
> * This changes some processes using "for loop" to using "while(bms_next_member())" which speeds up processing when we scan few partitions in one statement, but when we scan a lot of partitions in one statement, its performance will likely degraded. I measured the performance of both cases.
> I executed select statement to the table which has 4096 partitions.
>
> [scanning 1 partition]
> Without 0007 : 3,450 TPS
> With 0007 : 3,723 TPS
>
> [scanning 4096 partitions]
> Without 0007 : 10.8 TPS
> With 0007 : 10.5 TPS
>
> In the above result, performance degrades 3% in case of scanning 4096 partitions compared before and after applying 0007 patch. I think when scanning a lot of tables, executor time would be also longer, so the increasement of planner time would be relatively smaller than it. So we might not have to care this performance degradation.

I think it's better to focus on the fewer partitions case due to the
fact that execution initialisation time and actual execution are
likely to take much longer when more partitions are scanned. I did
some work on run-time pruning to tune it for this case. Tom did make
a similar argument in [1] and I explained my reasoning in [2].
bms_next_member has gotten a good performance boost since then and the
cases are not exactly the same since the old version the loop in
run-time pruning checked bms_is_member(), but the fact is, we did end
up tuning for the few partitions case in the end.

However, it would be good to see the performance results for
plan+execution time of say a table with 4k parts looking up a single
indexed value. You could have two columns, one that's the partition
key which allows the pruning to take place, and one that's not and
results in scanning all partitions. I'll be surprised if you even
notice the difference between with and without 0007 with the latter
case.

[1] https://www.postgresql.org/message-id/16107.1542307838%40sss.pgh.pa.us
[2] https://www.postgresql.org/message-id/CAKJS1f8ZnAW9VJNpJW16t5CtXSq3eAseyJXdumLaYb8DiTbhXA%40mail.gmail.com

--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services