Re: testing framework for MVCC & vacuum (freeze) & heap_page_prune etc.

"Andrey M. Borodin" <x4mmm(at)yandex-team(dot)ru> writes:

>> On 10 Dec 2024, at 08:31, Andy Fan <zhihuifan1213(at)163(dot)com> wrote:
>>
>> I want to know if we have some existing testing framework for
>> this area (design, code, licence etc).
>
> I think isolation tests [0] are what you are looking for. These tests are designed to test concurrent execution of various queries.
> More subtle race conditions require coordination of injection points
> [1] or are tested with stochastic tests [2].

Thanks, I always think the isolation tests under t/ directory is
valuable and it is a great way prove a bug is really fixed under the
known case. However it is not perfect because:

(a). All the cases are the known cases, kind of different from high
concurrently case which may find some bugs we are not aware of.
(b). every tap test needs to create a new instance and run the
test, it is hard to think this is a effective way for lots of tests.

So for testing some unknown bug, I am imaging the testing framework
should contains the following module at least.

(a). Workload. Produce kind of queries with different character, like
hot update, epq update and so on.
(b). Testing enginer with pgbench. run these queries with high
concurrey.
(c). error detection. error, core, wrong result and so on.

Some more modules like:
relkind module: the same column definitions with different
relkind, like (plain table, partition table etc.), so that the same
workload can hit different knowledge. different index definitions and so
on.

parallel module: different server configuration (debug_paralle_query
etc.).

Common GUCS module: different enable_xxx configuration, compare the
result under Read Repeatable isolation level.

In the past 2 ~ 0.7 years, we internally develop such testing framework and
found out some unkonwn bug in the community version. like [1] [2]. But
that project has stopped and I thought the project is kind of heavy in
implementation and test case organization, so I want to redesign the v2
and for the *MVCC & vacuum (freeze) & heap_page_prune* part only(which
is more related to my current work).

[1]
https://www.postgresql.org/message-id/tencent_A3CE810F59132D8E230475A5F0F7A08C8307%40qq.com
[2]
https://www.postgresql.org/message-id/202312121711.tzjyb5yeb3fa%40alvherre.pgsql

--
Best Regards
Andy Fan