Re: parallel mode and parallel contexts

On Tue, Apr 21, 2015 at 11:38 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>
>
> After thinking about it a bit more, I realized that even if we settle
> on some solution to that problem, there's another issues: the
> wait-edges created by this system don't really have the same semantics
> as regular lock waits. Suppose A is waiting on a lock held by B and B
> is waiting for a lock held by A; that's a deadlock. But if A is
> waiting for B to write to a tuple queue and B is waiting for A to read
> from a tuple queue, that's not a deadlock if the queues in question
> are the same. If they are different queues, it might be a deadlock,
> but then again maybe not. It may be that A is prepared to accept B's
> message from one queue, and that upon fully receiving it, it will do
> some further work that will lead it to write a tuple into the other
> queue. If so, we're OK; if not, it's a deadlock.

I agree that the way deadlock detector works for locks, it might not
be same as it needs to work for communication buffers (tuple queues).
Here I think we also need to devise some different way to remove resources
from wait/resource queue, it might not be a good idea to do it similar to
locks
as locks are released at transaction end whereas this new resources
(communication buffers) don't operate at transaction boundary.

> I'm not sure
> whether you'll want to argue that that is an implausible scenario, but
> I'm not too sure it is. The worker could be saying "hey, I need some
> additional piece of your backend-local state in order to finish this
> computation", and the master could then provide it. I don't have any
> plans like that, but it's been suggested previously by others, so it's
> not an obviously nonsensical thing to want to do.
>

If such a thing is not possible today and also it seems that is a not a
good design to solve any problem, then why to spend too much effort
in trying to find ways to detect deadlocks for the same.

> A further difference in the semantics of these wait-edges is that if
> process A is awaiting AccessExclusiveLock on a resource held by B, C,
> and D at AccessShareLock, it needs to wait for all of those processes
> to release their locks before it can do anything else. On the other
> hand, if process A is awaiting tuples from B, C, and D, it just needs
> ONE of those processes to emit tuples in order to make progress. Now
> maybe that doesn't make any difference in practice, because even if
> two of those processes are making lively progress and A is receiving
> tuples from them and processing them like gangbusters, that's probably
> not going to help the third one get unstuck. If we adopt the approach
> of discounting that possibility, then as long as the parallel leader
> can generate tuples locally (that is, local_scan_done = false) we
> don't report the deadlock, but as soon as it can no longer do that
> (local_scan_done = true) then we do, even though we could still
> theoretically read more tuples from the non-stuck workers. So then
> you have to wonder why we're not solving problem #1, because the
> deadlock was just as certain before we generated the maximum possible
> number of tuples locally as it was afterwards.
>

I think the deadlock detection code should run if anytime we have to
wait for more than deadlock timeout. Now I think the important point
here is when to actually start waiting, as per current parallel_seqscan
patch we wait only after checking the tuples from all queues, we could
have done it other way (wait if we can't fetch from one queue) as well.
It seems both has pros and cons, so we can proceed assuming current
way is okay and we can consider to change it in future once we find some
reason for the same.

Having said above, I feel this is not the problem that we should try to
solve at this point unless there is any scenario where we could hit
deadlock due to communication buffers. I think such a solution would
be required for advanced form of parallelism (like intra-query parallelism).
By the way, why are we trying to solve this problem, is there any way
with which we can hit it for parallel sequential scan?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com