From: | Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com> |
---|---|
To: | David Rowley <david(dot)rowley(at)2ndquadrant(dot)com> |
Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Amit Khandekar <amitdkhan(dot)pg(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, Ashutosh Bapat <ashutosh(dot)bapat(at)enterprisedb(dot)com>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: modeling parallel contention (was: Parallel Append implementation) |
Date: | 2017-05-08 02:00:26 |
Message-ID: | CAEepm=30bnfhoSt1W+kES+JE5GNE9Y7Mb6GwRvmBvXvS8uT+dg@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Mon, May 8, 2017 at 1:39 PM, David Rowley
<david(dot)rowley(at)2ndquadrant(dot)com> wrote:
> On 6 May 2017 at 13:44, Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com> wrote:
>> Experimentation required...
>
> Indeed. I do remember long discussions on this before Parallel seq
> scan went in, but I don't recall if anyone checked any OS kernels to
> see what they did.
>
> We really need a machine with good IO concurrency, and not too much
> RAM to test these things out. It could well be that for a suitability
> large enough table we'd want to scan a whole 1GB extent per worker.
I did a bunch of simple experiments this morning to try to observe RA
effects, using a couple of different EDB machines running Linux. I
wrote a simple program to read large files sequentially using lseek +
read, but rotate the reads over N file descriptors to simulate
parallel workers. I was surprised to find that I couldn't change
cache-cold read performance that way, up to very large numbers of N.
I did manage to break it by introducing some artificial disorder,
reversing/scrambling the read order of small groups of blocks, but
even that required groups over about 16 blocks before performance
started to drop (possibly related to the window size which I can't see
due to permissions right now). I've also learned that RAID cards
sometimes do read-ahead of their own, making matters more complicated.
I hope to report more when I figure out all the moving parts...
> I did post a patch to have heap_parallelscan_nextpage() use atomics
> instead of locking over in [1], but I think doing atomics there does
> not rule out also adding batching later. In fact, I think it
> structures things so batching would be easier than it is today.
+1
--
Thomas Munro
http://www.enterprisedb.com
From | Date | Subject | |
---|---|---|---|
Next Message | Amit Langote | 2017-05-08 02:05:52 | Re: pg_dump emits ALTER TABLE ONLY partitioned_table |
Previous Message | Amit Langote | 2017-05-08 01:44:28 | Re: Declarative partitioning - another take |