Re: WAL usage calculation patch

On Sun, Mar 29, 2020 at 1:26 PM Julien Rouhaud <rjuju123(at)gmail(dot)com> wrote:
>
> I'm not sure that I get your point. I'm assuming that you meant
> parallel-read-only queries, but surely buffer usage infrastructure for
> parallel query relies on the same approach as non-parallel one (each node
> computes the process-local pgBufferUsage diff) and sums all of that at the end
> of the parallel query execution. I also don't see how whether the query is
> read-only or not is relevant here as far as instrumentation is concerned,
> especially since read-only query can definitely do writes and increase the
> count of dirtied buffers, like a write query would. For instance a hint
> bit change can be done in a parallel query AFAIK, and this can generate WAL
> records in wal_log_hints is enabled, so that's probably one way to test it.
>

Yeah, that way we can test it. Can you try that?

> I now think that not adding support for WAL buffers in EXPLAIN output in the
> initial patch scope was a mistake, as this is probably the best way to test the
> WAL counters for parallel queries. This shouldn't be hard to add though, and I
> can work on it quickly if there's still a chance to get this feature included
> in pg13.
>

I am not sure we will add it in Explain or not (maybe we need inputs
from others in this regard), but if it helps in testing this part of
the patch, then it is a good idea to write a patch for it. You might
want to keep it separate from the main patch as we might not commit
it.

> > I think for now we should remove
> > that part of changes and rather think how to get that for parallel
> > operations that can write WAL. For ex. we might need to do something
> > similar to what this patch has done in begin_parallel_vacuum and
> > end_parallel_vacuum. Would you like to attempt that?
>
> Do you mean removing WAL buffers instrumentation from parallel query
> infrastructure?
>

Yes, I meant that but now I realize we need those and your proposed
way of testing it can help us in validating those changes.

> For parallel utility that can do writes it's probably better to keep the
> discussion in the other part of the thread.
>

Sure, I am fine with that but I am not sure if it is a good idea to
commit this patch without having a way to compute WAL utilization for
those commands.

I tried to think a little bit
> about that, but for now I don't have a better idea than adding something
> similar to intrumentation for utility command to have a general infrastructure,
> as building a workaround for specific utility looks like the wrong approach.
>

I don't know what exactly you have in mind as I don't see why it
should be too complex. Let's wait for a patch from Sawada-San on
buffer usage stuff and in the meantime, we can work on other parts of
this patch.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com