From: | Thomas Munro <thomas(dot)munro(at)gmail(dot)com> |
---|---|
To: | Pg Hackers <pgsql-hackers(at)postgresql(dot)org> |
Cc: | Masayuki Takahashi <masayuki038(at)gmail(dot)com> |
Subject: | Re: How to estimate the shared memory size required for parallel scan? |
Date: | 2019-05-02 08:06:19 |
Message-ID: | CA+hUKGKzn3mjCAp=TDrjiVSA1r5O=nhxXRmBX-yQMADjcRexvQ@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Mon, Aug 20, 2018 at 11:09 AM Thomas Munro
<thomas(dot)munro(at)enterprisedb(dot)com> wrote:
> 2. Teach your GetForeignPath function to do something like this:
> [blah blah blah]
I was pinged off-list by someone who is working on a parallel-aware
FDW, who asked if I still had the test code I mentioned up-thread.
While digging that out, I couldn't resist hacking it a bit more until
it gave the right answers, only sooner:
$ seq -f '%20.0f' 1 10000000 > numbers.csv
create extension file_fdw;
create server files foreign data wrapper file_fdw;
create foreign table numbers (n int) server files
options (filename '/path/to/numbers.csv', format 'csv');
explain select count(*) from numbers;
select count(*) from numbers;
Non-parallel: 2.6s
1 worker: 1.4s
2 workers: 0.9s
3 workers: 0.7s
Finally, I can do parallel hash joins between CSV files!
select count(*) from numbers n1 join numbers n2 using (n);
Non-parallel: 11.4s
1 worker: 6.6s
2 workers: 4.8s
3 workers: 4.1s
There are probably some weird fence-post or accounting bugs hiding in
this patch -- it has to count bytes carefully, and deal with some edge
cases around lines that span chunks. It's only a rough draft, but
might eventually serve as a useful example of a parallel-aware FDW.
--
Thomas Munro
https://enterprisedb.com
Attachment | Content-Type | Size |
---|---|---|
0001-Make-file_fdw-parallel-aware.patch | application/octet-stream | 14.3 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | Peter Eisentraut | 2019-05-02 08:44:44 | Re: Heap lock levels for REINDEX INDEX CONCURRENTLY not quite right? |
Previous Message | Heikki Linnakangas | 2019-05-02 07:37:29 | Re: Failure in contrib test _int on loach |