RFC: Giving bgworkers walsender-like grace during shutdown (for logical replication)

Hi folks

TL;DR: Anyone object to a new bgworker flag that exempts background workers
(such as logical apply workers) from the first round of fast shutdown
signals, and instead lets them to finish their currently in-progress txn
and exit?

This is a change proposal raised for comment before patch submission so
please consider it. Explanation of why I think we need it comes first, then
proposed implementation.

Rationale:

Currently a fast shutdown causes logical replication subscribers to abort
their currently in-progress transaction and terminate along with user
backends. This means they cannot finish receiving and flushing the
currently in-progress transaction, possibly wasting a very large amount of
work.

After restart the subscriber must reconnect, decode and reorder buffer from
the restart_lsn up to the current confirmed_flush_lsn, receive the whole
txn on the wire all over again, and apply the whole txn again locally. We
don't currently spool received txn change-streams to disk on the subscriber
and flush them so we can't repeat just the local apply part (see the
related thread "Logical archiving" for relevant discussion there). This can
create a lot of bloat, a lot of excess WAL, etc, if a big txn was in
progress at the time.

I'd like to add a bgworker flag that tells the postmaster to treat the
logical apply bgworker (or extension equivalents) somewhat like a walsender
for the purpose of fast shutdown. Instead of immediately terminating it
like user backends on fast shutdown, the bgworker should be sent a
ProcSignal warning that shutdown is pending and instructing it to finish
receiving and applying its current transaction, then exit gracefully.

It's not quite the same as the walsender, since there we try to flush
changes to downstreams up to the end of the last commit before shutting
down. That doesn't make sense on a subscriber because the upstream is
likely still generating txns. We just want to avoid wasting our effort on
any in-flight txn.

Any objections?

Proposed implementation:

* Add new bgworker flag like BGW_DELAYED_SHUTDOWN

* Define new ProcSignal PROCSIG_SHUTDOWN_REQUESTED. On fast shutdown send
this instead of a SIGTERM to bgworker backends flagged
BGW_DELAYED_SHUTDOWN. On smart shutdown send it to all backends when the
shutdown request arrives, since that could be handy for other uses too.

* Flagged bgworker is expected to finish its current txn and exit promptly.
Impose a grace period after which they get SIGTERM'd anyway. Also send a
SIGTERM if the postmaster receives a second fast shutdown request.

* Defer sending PROCSIG_WALSND_INIT_STOPPING to walsenders until all
BGW_DELAYED_SHUTDOWN flagged bgworkers have exited, so we can ensure that
cascaded downstreams receive any txns applied from the upstream.

This doesn't look likely to be particularly complicated to implement.

It might be better to use a flag in PGPROC rather than the bgworker struct,
in case we want to extend this to other backend types in future. Also to
make it easier for the postmaster to check the flag during shutdown. Could
just claim a bit from statusFlags for the purpose. Thoughts?