Re: Speed up Clog Access by increasing CLOG buffers

On Tue, Nov 17, 2015 at 6:30 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:

> On 17 November 2015 at 11:48, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
>> On Tue, Nov 17, 2015 at 5:04 PM, Simon Riggs <simon(at)2ndquadrant(dot)com>
>> wrote:
>>
>>> On 17 November 2015 at 11:27, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
>>> wrote:
>>>
>>> We are trying to speed up real cases, not just benchmarks.
>>>>>
>>>>> So +1 for the concept, patch is going in right direction though lets
>>>>> do the full press-up.
>>>>>
>>>>>
>>>> I have mentioned above the reason for not doing it for sub
>>>> transactions, if
>>>> you think it is viable to reserve space in shared memory for this
>>>> purpose, then
>>>> I can include the optimization for subtransactions as well.
>>>>
>>>
>>> The number of subxids is unbounded, so as you say, reserving shmem isn't
>>> viable.
>>>
>>> I'm interested in real world cases, so allocating 65 xids per process
>>> isn't needed, but we can say is that the optimization shouldn't break down
>>> abruptly in the presence of a small/reasonable number of subtransactions.
>>>
>>>
>> I think in that case what we can do is if the total number of
>> sub transactions is lesser than equal to 64 (we can find that by
>> overflowed flag in PGXact) , then apply this optimisation, else use
>> the existing flow to update the transaction status. I think for that we
>> don't even need to reserve any additional memory. Does that sound
>> sensible to you?
>>
>
> I understand you to mean that the leader should look backwards through the
> queue collecting xids while !(PGXACT->overflowed)
>
>
Yes, that is what the above idea is, but the problem with that is leader
won't be able to collect the subxids of member proc's (from each member
proc's XidCache) as it doesn't have information which of those subxid's
needs to be update as part of current transaction status update (for
subtransactions on different clog pages, we update the status of those
in multiple phases). I think it could only be possible to use the above
idea
if all the subtransactions are on same page, which we can identify in
function TransactionIdSetPageStatus(). Though it looks okay that we
can apply this optimization when number of subtransactions is lesser
than 65 and all exist on same page, still it would be better if we can
apply it generically for all cases when number of subtransactions is small
(say 32 or 64). Does this explanation clarify the problem with the above
idea to handle subtransactions?

No additional shmem is required
>
>
If we want to do it for all cases when number of subtransactions
are small then we need extra memory.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com