From: | Misa Simic <misa(dot)simic(at)gmail(dot)com> |
---|---|
To: | Steve Crawford <scrawford(at)pinpointresearch(dot)com> |
Cc: | "pgsql-performance(at)postgresql(dot)org" <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: Slow concurrent processing |
Date: | 2013-03-12 15:06:46 |
Message-ID: | CAH3i69=zLg69a+1aDzWEx3LgdoeHi8CEq_FwKKeAOG3eiH5mng@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-performance |
Thanks Steve
Well, the full story is too complex - but point was - whatever blackbox
does - it last 0.5 to 2secs per 1 processed record (maybe I was wrong but I
thought the reason why it takes the time how much it needs to actually do
the task -CPU/IO/memory whatever is not that important....) - so I really
don't see difference between: call web service, insert row in the table
(takes 3 secs) and sleep 3 seconds - insert result in the table...
if we do above task for two things sequential - it will last 6 secs...but
if we do it "concurentelly" - it should last 3 secs... (in theory :) )
I was guessed somewhere is lock - but wasn't clear where/why when there are
no updates - just inserts...
But I haven't know that during INSERT is done row lock on refferenced
tables as well - from FK columns...
So I guess now it is cause of the problem...
We will see how it goes with insert into unlogged tables with no FK...
Many thanks,
Misa
2013/3/12 Steve Crawford <scrawford(at)pinpointresearch(dot)com>
> On 03/11/2013 08:55 PM, Misa Simic wrote:
>
>> Hi all,
>>
>> We have one table with list of "records for processing"...
>>
>> We loop trough that table and call one long runing function:
>>
>> do_the_math_for_record(record_**id)...<snip>...
>>
>>
>> but - if replace do_the_math_and_save_results with pg_sleep(1); To
>> simulate long running function so processing each row - last 1 sec...
>>
>> Sequential processing last as expected 300 seconds!
>>
>> Concurrent processing last faster with higher number of concurrent
>> connections - about 30 seconds with 30 connections! (much faster - and
>> expected...)
>>
>> however, if we return our: do_the_math_and_save_results - we can't get
>> better results in concurrent processing...
>>
>
> Sleep will not have any significant impact on CPU, memory or disk use and
> thus is not a simulation of actual processing.
>
> All you have really shown us so far is a black box. Please provide an
> overview of your schemas and the type of processing you are attempting on
> them.
>
> Cheers,
> Steve
>
>
From | Date | Subject | |
---|---|---|---|
Next Message | Steve Crawford | 2013-03-12 15:54:57 | Re: Slow concurrent processing |
Previous Message | Steve Crawford | 2013-03-12 14:20:18 | Re: Slow concurrent processing |