Re: Issue with markers in isolation tester? Or not?

On Sun, Jan 19, 2025 at 09:55:32PM +0100, Michail Nikolaev wrote:
> > Stepping back, any variation in the total ordering of step-start and
> > step-complete events necessarily changes the expected output. Hence, we
> want
> > to freeze that ordering. We could do that directly, by having
> isolationtester
> > read a format that lists start and complete events. We could also have
> the
> > isolationtester write the same format, so you could draft a permutation
> > without blocking conditions and just save the order that isolationtester
> > discovers by experiment. I'd find that less mentally taxing than
> following
> > rules to reach a frozen ordering via blocking conditions.
>
> > How well would that meet your test's needs?
>
> It feels a bit strange to me to force isolationtest to output something in
> a specific order, especially since we don't actually care about the order
> in the test.
>
> But why do we need that output at all?

It's good to think of ways we can output differently for more stability.
However, ordered output helps the human studying what happened. Today's
output is well-optimized for that. Today's output resembles what a person
would see if running a collection of psql sessions. For example, if I'm
reviewing a patch, reviewing the expected spec output feels natural.

> I think a simpler solution would be
> to mark a step in a way that prevents isolationtester from outputting
> detailed information about the completion of that step—just an overview of
> the steps launched and completed at the end of the permutation.

My main concern about that is the difficulty of marking steps:

- Best: no markings
- Medium: easy rules for when you need a marking
- Worst: run on FreeBSD to determine marking requirements by experiment

I've not been able to identify an easy rule that avoids "run on FreeBSD" with
today's framework. (I have a non-easy list of four rules, but I'm not
confident even that list is complete.) I can't think of an easy rule for
"just an overview" markings, either. Even if there is one, not needing
markings is better, all else being equal. What other advantages may
compensate for this concern?

There's another, minor concern. The more "just an overview" markings in a
spec, the more reading its expected output wouldn't feel like reading psql
transcripts. I'd likely end up logging both today's ordered format and the
new format. We'd diff the new format only, and we'd write the old format for
human debugging.

> In my tests, I evaluate how [CREATE|RE] INDEX CONCURRENTLY interferes with
> UPSERTs. I’m not testing the exact ordering; I simply set the backends into
> a specific state (with different index information) and let them proceed
> one by one. I don’t care which one finishes first or last—I’m only
> interested in identifying errors in the output.

That makes sense.

Thanks,
nm