Parallel Foreign Scans - need advice

From: Korry Douglas <korry(at)me(dot)com>
To: PostgreSQL-development Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Parallel Foreign Scans - need advice
Date: 2019-05-15 16:55:33
Message-ID: 6ED3BCAA-C4A8-44E4-86EC-6309E41C6352@me.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi all, I’m working on an FDW that would benefit greatly from parallel foreign scan. I have implemented the callbacks described here:https://www.postgresql.org/docs/devel/fdw-callbacks.html#FDW-CALLBACKS-PARALLEL. and I see a big improvement in certain plans.

My problem is that I can’t seem to get a parallel foreign scan in a query that does not contain an aggregate.

For example:
SELECT count(*) FROM foreign table;
Gives me a parallel scan, but
SELECT * FROM foreign table;
Does not.

I’ve been fiddling with the costing GUCs, foreign scan row estimates, and foreign scan cost estimates - I can force the cost of a partial path to be much lower than a sequential foreign scan, but no luck.

Any troubleshooting advice?

A second related question - how can I find the actual number of workers chose for my ForeignScan? At the moment, I looking at ParallelContext->nworkers (inside of the InitializeDSMForeignScan() callback) because that seems to be the first callback function that might provide the worker count. I need the *actual* worker count in order to evenly distribute my workload. I can’t use the usual trick of having each worker grab the next available chunk (because I have to avoid seek operations on compressed data). In other words, it is of great advantage for each worker to read contiguous chunks of data - seeking to another part of the file is prohibitively expensive.

Thanks for all help.

— Korry

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Alvaro Herrera 2019-05-15 17:05:30 Re: New EXPLAIN option: ALL
Previous Message Daniel Verite 2019-05-15 16:41:32 RE: psql - add SHOW_ALL_RESULTS option