Re: Enhance file_fdw to report processed and skipped tuples in COPY progress

Hi,

On Thu, Oct 3, 2024 at 2:23 AM Fujii Masao <masao(dot)fujii(at)oss(dot)nttdata(dot)com> wrote:
>
> Hi,
>
> Currently, file_fdw updates several columns in the pg_stat_progress_copy view,
> like relid and bytes_processed, but it doesn't track tuples_processed or
> tuples_skipped. Monitoring these would be particularly useful when handling
> large data sets via file_fdw, as it helps track the progress of scan.
>
> The attached patch updates file_fdw to add support for reporting
> the number of tuples processed and skipped (due to on_error = 'ignore')
> in the pg_stat_progress_copy view. What are your thoughts?

While the patch works fine and looks good to me, in the first place,
it seems to me that the fact that file_fdw uses the COPY progress
itself doesn't work properly. For example, unlike COPY command,
queries could have multiple scans on one or more flie_fdw foreign
tables when joining tables. I found the discussion for that[1]: there
was a proposal of disabling COPY progress for file_fdw but the votes
are split. I think it would be better to consider if we really want to
support COPY progress for file_fdw before supporting more progress
information.

[1] https://www.postgresql.org/message-id/flat/20230119054703.GB13860%40telsasoft.com

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com