Re: BUG #16827: macOS interrupted syscall leads to a crash

Hi,

@Kimon, do you have an antivirus installed or an EndpointSecurity
Daemon running? I managed to get this crash only when my
EndpointSecurity Daemon was up and running.
Perhaps the problem is more generic (maybe when the import is very
large. My database is only 10GB) if you can replicate without the
EndpointSecurity framework running.

În vin., 22 ian. 2021 la 19:35, Andres Freund <andres(at)anarazel(dot)de> a scris:
>
> Hi,
>
> Tom, all, this seems like a serious problem likely to become more
> widespread. I don't really know how we can reasonably address this
> short-term, adding EINTR handling to all the places that don't yet have
> it (and ensuring that it stays that way) seems like it's a lot to backport.
>
> There's just been another report of this at
> https://postgr.es/m/16832-943d33fd58eb26e4%40postgresql.org
>
> On 2021-01-16 19:30:52 +0200, Ricardo Ungureanu wrote:
> > În vin., 15 ian. 2021 la 23:05, Andres Freund <andres(at)anarazel(dot)de> a scris:
> > >
> > > Hi,
> > >
> > > On 2021-01-15 14:00:03 +0000, PG Bug reporting form wrote:
> > > > I am using macOS 11.0 and trying to import a large dump into postgresql.
> > > > Under some circumstances, it crashes while importing.
> > > > I inspected the logs and found out a system call is interrupted (" LOG:
> > > > could not open file "pg_wal": Interrupted system call"). Apple has added a
> > > > new feature in macOS 11.0 to audit security events. I noticed that the
> > > > kernel, while waiting on a condition variable, if it receives an interrupt,
> > > > will just pass EINTR (error code 4) back to the usermode program. Your
> > > > function XLogFileInit does not treat such cases (just ENOENT is checked) and
> > > > decides to exit with an abort(). I have attached below the crash file
> > > > generated.
> > >
> > > Hm. It's fairly nasty to return EINTR from open() (except if open()ing a
> > > FIFO or such) - it should normally only happen when blocked. But I'm not
> > > sure it's *actually* violating any standards / promises made.
> >
> > There are two kinds of security events which Apple supports: AUTH and
> > NOTIFY. AUTH means that the system call is blocked (on that condition
> > variable I mentioned about), and the user mode daemon is asked about
> > the generated event: "postgres, pid 999, open() on file /path/file
> > with flags 0x400003" - something like that. The usermode can either
> > allow or deny the event by replying. If it decides to block this
> > system call, the return code seen by the target process (in this case,
> > postgres) is -1 (operation not permitted) .
> > On the other hand, NOTIFY events will only log the event, without
> > requesting a verdict (allow or deny).
> > In my scenario, usermode daemon responsible for auditing these events
> > is set to ALLOW everything. If I denied the system call I would see in
> > postgres log "operation not permitted (err 1)" insead of 'Interrupted
> > system call (err 4)").
>
> Does this happens in the default configuration of OSX now? Or is this
> something you have manually set up? If manual, why?

For me it happens in the default configuration when I start my
EndpointSecurity Daemon and just allow every action.
However, the bug might be more generic and could happen when the
system is overloaded. Let's see if Kimon has an antivirus installed or
any other software using EndpointSecurity Framework.

> Isn't forcing a back-forth context switch for a syscall as common as
> open(2) to that auth demon *terrible* for performance?
It is, but Apple does not seem to care at all. From my performance
testing, (without any event processing at all) I see an impact of
30-40%.

> > To sum up, the open() is blocked, waiting for a verdict from the
> > usermode, meanwhile an interrupt is triggered, msleep on the condition
> > variable returns EINTR and this is passed back to postgres as the
> > return code of open().
>
> > [1] https://developer.apple.com/documentation/endpointsecurity/es_event_type_t
>
> I don't understand how Apple is expecting this to not cause breakage
> left and right? Just about no software has EINTR handling for every
> syscall (as SA_RESTART is enough), so this seems like it'll lead to a
> long tail of bugs.
>
>
> > > > Apple has added a new feature in macOS 11.0 to audit security
> > > > events. I noticed that the kernel, while waiting on a condition
> > > > variable, if it receives an interrupt, will just pass EINTR (error
> > > > code 4) back to the usermode program.
> > >
> > > Does that also happen for close()? Because that can't reasonably be
> > > handled by userspace (userspace cannot retry because the fd could now
> > > point to something else in a threaded environment).
> >
> > Good point, however the close event is not supported as AUTH, only as
> > NOTIFY. Thus, this cannot happen on close().
> > Open() and other file system calls are both AUTH and NOTIFY (you can
> > choose which one to enable).
> > You can read more about this here[1]
>
> Well, that's at least something.
>
> Greetings,
>
> Andres Freund

Regards,
Ricardo